diff --git a/.serena/memories/critical-info.md b/.serena/memories/critical-info.md index 142d303310..4d9de4463e 100644 --- a/.serena/memories/critical-info.md +++ b/.serena/memories/critical-info.md @@ -12,6 +12,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo. - Commit only when explicitly asked. Commit/PR format + changelog: `mem:workflow/creating-commits`, `mem:workflow/creating-prs`. - You have access to the GitHub CLI `gh` or corresponding MCP tools. - Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool. +- Never run anything that destroys data without explicit permission, including `drop-devenv`, `docker compose down -v`, `docker volume rm ...`. The user's real work lives in the volumes of the shared infra. # Project modules @@ -29,8 +30,10 @@ This is a monorepo. Principles that apply to one module do *not* generally apply # Low-centrality project paths -- `docker/` contains devenv related code, not needed unless specifically instructed. - More info in docs/technical-guide if instructed to work on this. +- `docker/` contains devenv related code, not needed unless specifically instructed. + When working on devenv startup, compose layout, instance config (`defaults.env`), + tmux session lifecycle, MinIO provisioning, or anything in `manage.sh`'s + `*-devenv` commands, read `mem:devenv/core`. - `experiments/` contains standalone experimental HTML/JS/scripts; treat it as non-core unless the user explicitly asks about it. - `sample_media/` contains sample image/icon media and config used as fixtures/demo material; do not infer app behavior from it. diff --git a/.serena/memories/devenv/core.md b/.serena/memories/devenv/core.md new file mode 100644 index 0000000000..7487c2f2e7 --- /dev/null +++ b/.serena/memories/devenv/core.md @@ -0,0 +1,48 @@ +# Devenv startup and configuration + +Compose-based development environment under `docker/devenv/`, driven by `manage.sh`. + +## Source-of-truth layout + +- `docker/devenv/defaults.env`: single source of truth for devenv config. Loaded by `manage.sh`'s simple env-file parser and by `docker compose --env-file`. Holds `COMPOSE_PROJECT_NAME`, container names (`PENPOT_MAIN_CONTAINER_NAME`, `PENPOT_VALKEY_CONTAINER_NAME`, `PENPOT_VALKEY_HOSTNAME`), runtime config that is passed into the container env, every published host port, Serena host ports, tmux session/attach defaults. `manage.sh` aborts if the file is unreadable. +- `backend/scripts/_env`: backend-internal defaults only — `PENPOT_*_SHARED_KEY`, `PENPOT_SECRET_KEY`, `PENPOT_FLAGS`, deletion/upload sizes, `PENPOT_NITRATE_BACKEND_URI`, `JAVA_OPTS`, the `setup_minio` function. Never duplicates anything in `defaults.env`. +- `docker/devenv/docker-compose.infra.yml`: shared services — `postgres`, `minio`, `minio-setup`, `mailer`, `ldap`. Attached to external network `penpot_shared`. +- `docker/devenv/docker-compose.main.yml`: main devenv container plus `redis` (valkey). Same network. Pure `${VAR}` references (no inline `:-` defaults) — missing var = compose fails. + +## Invariants + +- Published ports are host-side only. Compose maps `${PENPOT_*_PORT}:` so parallel instances can offset host ports while container-local services keep their normal devenv ports. Do not pass host-side port offsets into processes that expect container-local ports. +- Volume keys in compose are literal (`user_data`, `valkey_data`). Docker prefixes them with `COMPOSE_PROJECT_NAME` to form the actual volume names. +- External network `penpot_shared` is created idempotently by `manage.sh ensure-devenv-network`; `drop-devenv` does **not** remove it. +- `PENPOT_SOURCE_PATH` is set by `manage.sh` to `$PWD` and bind-mounted as `/home/penpot/penpot`. Not in `defaults.env` because its value is dynamic. +- `CURRENT_USER_ID=$(id -u)` is exported by `manage.sh` and passed as `EXTERNAL_UID` so file ownership inside the container matches the host. +- `JAVA_OPTS` exported at the top of `manage.sh` (line ~28) is **shadowed inside the container** by `_env`, which reassigns it unconditionally to a much larger JVM config. The `-e JAVA_OPTS=$JAVA_OPTS` flag that `run-devenv-shell` / `run-devenv-isolated-shell` / `build` pass into `docker run`/`exec` only matters for processes that do not source `_env`. + +## MinIO provisioning split + +- Shared user/policy: provisioned once by the `minio-setup` one-shot service in the infra compose file. Alias-set loop bounded to 30 attempts. `main` depends on `service_completed_successfully`. +- Per-process bucket creation: `setup_minio()` in `_env`. Idempotent (`mc mb -p`). Short-circuits if `PENPOT_OBJECTS_STORAGE_BACKEND != s3`. + +## Tmux session lifecycle + +- `docker/devenv/files/start-tmux.sh` is idempotent at the session level. Reads `PENPOT_TMUX_SESSION` (default `penpot`) and `PENPOT_TMUX_ATTACH` (default `true`). If the session exists it attaches or exits depending on `PENPOT_TMUX_ATTACH`; otherwise runs `./scripts/setup` for frontend/exporter and creates the session with frontend-watch / storybook / exporter / backend / optional MCP / optional Serena windows. +- MCP and Serena windows are added only on session create (gated by `enable-mcp` in `PENPOT_FLAGS` and `SERENA_ENABLED=true`). `run-devenv-agentic` against an existing non-agentic session attaches without adding them — kill the session first to recreate. +- `manage.sh run-devenv`: ensures containers, then invokes start-tmux.sh interactively (attaches). +- `manage.sh attach-devenv`: pure attach — fails fast if devenv isn't running or session doesn't exist. Never starts containers. Takes no arguments. + +## Lifecycle commands + +`manage.sh` thin wrappers around `devenv-compose` (which adds `--env-file defaults.env` and both compose files): +- `start-devenv` / `create-devenv`: pull image if missing, ensure network, `up -d` / `create`. +- `stop-devenv`, `log-devenv`: as expected. +- `drop-devenv`: `down -v` (removes containers + named volumes) and prunes the devenv image. Preserves `penpot_shared`. +- `run-devenv-shell`: starts containers if needed, `docker exec -ti` as `penpot`. +- `run-devenv-isolated-shell` / `build`: one-shot `docker run` against `${COMPOSE_PROJECT_NAME}_user_data` volume + repo bind mount. Not driven by compose. + +## Exporter env + +`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env`, then `_env.local` if present. Backend-style env reaches the exporter via that chain. + +## MCP routing in parallel devenvs + +The normal MCP path is same-origin: frontend computes `/mcp/ws`, the plugin opens that WebSocket, and the instance-local nginx proxies it to the MCP server inside the same main container. This depends on fixed internal ports; per-instance overlays should only change the published host ports and `PENPOT_PUBLIC_URI`. diff --git a/.serena/memories/mcp/core.md b/.serena/memories/mcp/core.md index ba794b1d0f..772f7a3949 100644 --- a/.serena/memories/mcp/core.md +++ b/.serena/memories/mcp/core.md @@ -85,3 +85,11 @@ From the `mcp/` directory, run * `pnpm run build` to test the build of all packages * `pnpm run fmt` to apply the auto-formatter + +## Devenv plugin/server wiring + +In the normal Penpot devenv MCP path, the browser plugin does not discover or route through Postgres. The frontend provides the plugin extension API with `mcp.getServerUrl()`, currently derived from `frontend/src/app/config.cljs` as `penpotMcpServerURI` if set, otherwise `/mcp/ws`. The MCP plugin opens a direct WebSocket to that URL and appends the current MCP access token as a query parameter. + +The live plugin connection registry is in-memory inside each MCP server process (`PluginBridge.connectedClients` / `clientsByToken`). The database only stores MCP access tokens and profile props such as `mcp-enabled`; it does not manage which plugin is connected to which MCP server. + +For parallel devenvs, prefer same-origin MCP routing: each Penpot instance should expose `/mcp/ws` through its own nginx/Caddy path to the MCP server running inside the same main container. Keep container-internal ports fixed (MCP defaults `4401/4402/4403`, backend/exporter/frontend defaults, etc.) and only offset host-side published ports per instance. If internal ports are offset, hardcoded local proxy config such as `docker/devenv/files/nginx.conf` will misroute unless templated too. diff --git a/backend/scripts/_env b/backend/scripts/_env index 2a89244a81..63079bf34f 100644 --- a/backend/scripts/_env +++ b/backend/scripts/_env @@ -8,8 +8,9 @@ export PENPOT_SECRET_KEY=super-secret-devenv-key # DEPRECATED: only used for subscriptions export PENPOT_MANAGEMENT_API_KEY=super-secret-management-api-key -export PENPOT_HOST=devenv -export PENPOT_PUBLIC_URI=https://localhost:3449 +# Runtime config that varies per devenv instance (PENPOT_HOST, PENPOT_PUBLIC_URI, +# PENPOT_DATABASE_*, PENPOT_REDIS_URI, PENPOT_OBJECTS_STORAGE_*, AWS_*) is owned by +# docker/devenv/defaults.env and injected via the main service's env block. export PENPOT_FLAGS="\ $PENPOT_FLAGS \ @@ -60,12 +61,6 @@ export PENPOT_HTTP_SERVER_MAX_MULTIPART_BODY_SIZE=314572800 export PENPOT_USER_FEEDBACK_DESTINATION="support@example.com" -export AWS_ACCESS_KEY_ID=penpot-devenv -export AWS_SECRET_ACCESS_KEY=penpot-devenv -export PENPOT_OBJECTS_STORAGE_BACKEND=s3 -export PENPOT_OBJECTS_STORAGE_S3_ENDPOINT=http://minio:9000 -export PENPOT_OBJECTS_STORAGE_S3_BUCKET=penpot - export PENPOT_NITRATE_BACKEND_URI=http://localhost:3000/admin-console export JAVA_OPTS="\ @@ -84,14 +79,14 @@ export JAVA_OPTS="\ --enable-native-access=ALL-UNNAMED"; function setup_minio() { - # Initialize MINIO config - mc alias set penpot-s3/ http://minio:9000 minioadmin minioadmin -q - mc admin user add penpot-s3 penpot-devenv penpot-devenv -q - mc admin user info penpot-s3 penpot-devenv |grep -F -q "readwrite" - if [ "$?" = "1" ]; then - mc admin policy attach penpot-s3 readwrite --user=penpot-devenv -q + if [ "${PENPOT_OBJECTS_STORAGE_BACKEND}" != "s3" ]; then + return 0 fi - mc mb penpot-s3/penpot -p -q + + # Shared MinIO user/policy provisioning is handled by docker-compose.infra.yml. + # Per process startup only ensures that the configured bucket exists. + mc alias set penpot-s3/ "${PENPOT_OBJECTS_STORAGE_S3_ENDPOINT}" minioadmin minioadmin -q + mc mb "penpot-s3/${PENPOT_OBJECTS_STORAGE_S3_BUCKET}" -p -q } diff --git a/docker/devenv/defaults.env b/docker/devenv/defaults.env new file mode 100644 index 0000000000..eb0e01a6cf --- /dev/null +++ b/docker/devenv/defaults.env @@ -0,0 +1,56 @@ +# Single source of truth for instance-specific devenv configuration. +# Loaded by docker compose via --env-file (see manage.sh). +# +# Stage 2 adds per-instance overlay files (e.g. instances/ws1.env) loaded +# after this one; variables not overridden fall back to the values here. +# +# This file is consumed by compose only. Backend runtime defaults that +# compose does not care about live in backend/scripts/_env. + +# Compose project name (replaces the -p flag). +COMPOSE_PROJECT_NAME=penpotdev + +# Container names and volume names. Volumes are pinned by explicit name +# (rather than relying on COMPOSE_PROJECT_NAME prefixing) so the physical +# volumes can survive a future project rename without a data-migration step. +PENPOT_MAIN_CONTAINER_NAME=penpot-devenv-main +PENPOT_VALKEY_CONTAINER_NAME=penpot-devenv-valkey +PENPOT_VALKEY_HOSTNAME=penpot-devenv-valkey +PENPOT_POSTGRES_DATA_VOLUME=penpotdev_postgres_data_pg16 +PENPOT_MINIO_DATA_VOLUME=penpotdev_minio_data +PENPOT_USER_DATA_VOLUME=penpotdev_user_data +PENPOT_VALKEY_DATA_VOLUME=penpotdev_valkey_data + +# Backend runtime config (passed to the container env block). +PENPOT_HOST=devenv +PENPOT_PUBLIC_URI=https://localhost:3449 +PENPOT_DATABASE_URI=postgresql://postgres/penpot +PENPOT_DATABASE_USERNAME=penpot +PENPOT_DATABASE_PASSWORD=penpot +PENPOT_DATABASE_MAX_POOL_SIZE=60 +PENPOT_REDIS_URI=redis://redis/0 + +# Object storage (MinIO user/policy are provisioned by the infra compose file). +PENPOT_OBJECTS_STORAGE_BACKEND=s3 +PENPOT_OBJECTS_STORAGE_S3_ENDPOINT=http://minio:9000 +PENPOT_OBJECTS_STORAGE_S3_BUCKET=penpot +AWS_ACCESS_KEY_ID=penpot-devenv +AWS_SECRET_ACCESS_KEY=penpot-devenv + +# Published host ports. Only ports that need to be reachable from outside the +# container are exposed; everything else (frontend dev server, backend API, +# storybook, exporter, REPLs, plugins, MCP inspector/websocket, aux) is +# accessed in-process or through the same-origin Caddy/nginx proxy at +# PENPOT_PUBLIC_HTTP_PORT. Container-internal ports remain fixed; per-instance +# overlays may offset these host-side values. +PENPOT_PUBLIC_HTTP_PORT=3449 +PENPOT_MCP_SERVER_PORT=4401 +PENPOT_MCP_REPL_PORT=4403 + +# Serena (agentic devenv). Internal ports are fixed by Serena itself. +SERENA_EXTERNAL_PORT=14281 +SERENA_DASHBOARD_EXTERNAL_PORT=14282 + +# Tmux session inside the main container. +PENPOT_TMUX_SESSION=penpot +PENPOT_TMUX_ATTACH=true diff --git a/docker/devenv/docker-compose.infra.yml b/docker/devenv/docker-compose.infra.yml new file mode 100644 index 0000000000..5bf979c4f8 --- /dev/null +++ b/docker/devenv/docker-compose.infra.yml @@ -0,0 +1,99 @@ +networks: + default: + name: penpot_shared + external: true + +volumes: + postgres_data_pg16: + name: ${PENPOT_POSTGRES_DATA_VOLUME} + minio_data: + name: ${PENPOT_MINIO_DATA_VOLUME} + +services: + minio: + image: "minio/minio:RELEASE.2025-04-03T14-56-28Z" + command: minio server /mnt/data --console-address ":9001" + + volumes: + - "minio_data:/mnt/data" + + environment: + - MINIO_ROOT_USER=minioadmin + - MINIO_ROOT_PASSWORD=minioadmin + + networks: + default: + aliases: + - minio + + minio-setup: + image: "minio/mc:latest" + depends_on: + - minio + entrypoint: ["/bin/sh", "-c"] + command: + - | + attempts=0 + until mc alias set penpot-s3 http://minio:9000 minioadmin minioadmin -q; do + attempts=$$((attempts + 1)) + if [ "$$attempts" -ge 30 ]; then + echo "minio-setup: gave up waiting for MinIO after $$attempts attempts" >&2 + exit 1 + fi + sleep 1 + done + mc admin user info penpot-s3 penpot-devenv >/dev/null 2>&1 || mc admin user add penpot-s3 penpot-devenv penpot-devenv -q + mc admin policy attach penpot-s3 readwrite --user=penpot-devenv -q + networks: + default: + + postgres: + image: postgres:16.8 + command: postgres -c config_file=/etc/postgresql.conf + restart: always + stop_signal: SIGINT + environment: + - POSTGRES_INITDB_ARGS=--data-checksums + - POSTGRES_DB=penpot + - POSTGRES_USER=penpot + - POSTGRES_PASSWORD=penpot + volumes: + - ./files/postgresql.conf:/etc/postgresql.conf:z + - ./files/postgresql_init.sql:/docker-entrypoint-initdb.d/init.sql:z + - postgres_data_pg16:/var/lib/postgresql/data + networks: + default: + aliases: + - postgres + + mailer: + image: sj26/mailcatcher:latest + restart: always + expose: + - '1025' + ports: + - "1080:1080" + + networks: + default: + aliases: + - mailer + + # https://github.com/rroemhild/docker-test-openldap + ldap: + image: rroemhild/test-openldap:2.1 + expose: + - '10389' + - '10636' + ports: + - "10389:10389" + - "10636:10636" + ulimits: + nofile: + soft: 1024 + hard: 1024 + + networks: + default: + aliases: + - ldap diff --git a/docker/devenv/docker-compose.main.yml b/docker/devenv/docker-compose.main.yml new file mode 100644 index 0000000000..e5375b1819 --- /dev/null +++ b/docker/devenv/docker-compose.main.yml @@ -0,0 +1,110 @@ +networks: + default: + name: penpot_shared + external: true + +volumes: + user_data: + name: ${PENPOT_USER_DATA_VOLUME} + valkey_data: + name: ${PENPOT_VALKEY_DATA_VOLUME} + +services: + main: + privileged: true + image: "penpotapp/devenv:latest" + build: + context: "." + container_name: "${PENPOT_MAIN_CONTAINER_NAME}" + stop_signal: SIGINT + + depends_on: + postgres: + condition: service_started + redis: + condition: service_started + minio-setup: + condition: service_completed_successfully + + volumes: + - "user_data:/home/penpot/" + - "${PENPOT_SOURCE_PATH}:/home/penpot/penpot:z" + + ports: + # Host ports are instance-specific; container ports stay fixed. + - ${PENPOT_PUBLIC_HTTP_PORT}:3449 + - ${PENPOT_PUBLIC_HTTP_PORT}:3449/udp + + # MCP + - ${PENPOT_MCP_SERVER_PORT}:4401 + - ${PENPOT_MCP_REPL_PORT}:4403 + + # Serena MCP server (agentic mode only). Internal ports fixed by Serena. + - ${SERENA_EXTERNAL_PORT}:14281 + - ${SERENA_DASHBOARD_EXTERNAL_PORT}:24282 + + environment: + - EXTERNAL_UID=${CURRENT_USER_ID} + + # SMTP setup (shared infra service; identical across instances) + - PENPOT_SMTP_ENABLED=true + - PENPOT_SMTP_DEFAULT_FROM=no-reply@example.com + - PENPOT_SMTP_DEFAULT_REPLY_TO=no-reply@example.com + - PENPOT_SMTP_HOST=mailer + - PENPOT_SMTP_PORT=1025 + - PENPOT_SMTP_USERNAME= + - PENPOT_SMTP_PASSWORD= + - PENPOT_SMTP_SSL=false + - PENPOT_SMTP_TLS=false + + # LDAP setup (shared infra service; identical across instances) + - PENPOT_LDAP_HOST=ldap + - PENPOT_LDAP_PORT=10389 + - PENPOT_LDAP_SSL=false + - PENPOT_LDAP_STARTTLS=false + - PENPOT_LDAP_BASE_DN=ou=people,dc=planetexpress,dc=com + - PENPOT_LDAP_BIND_DN=cn=admin,dc=planetexpress,dc=com + - PENPOT_LDAP_BIND_PASSWORD=GoodNewsEveryone + - PENPOT_LDAP_ATTRS_USERNAME=uid + - PENPOT_LDAP_ATTRS_EMAIL=mail + - PENPOT_LDAP_ATTRS_FULLNAME=cn + - PENPOT_LDAP_ATTRS_PHOTO=jpegPhoto + + # Per-instance runtime config. Defaults live in defaults.env. + - PENPOT_HOST=${PENPOT_HOST} + - PENPOT_PUBLIC_URI=${PENPOT_PUBLIC_URI} + - PENPOT_DATABASE_URI=${PENPOT_DATABASE_URI} + - PENPOT_DATABASE_USERNAME=${PENPOT_DATABASE_USERNAME} + - PENPOT_DATABASE_PASSWORD=${PENPOT_DATABASE_PASSWORD} + - PENPOT_DATABASE_MAX_POOL_SIZE=${PENPOT_DATABASE_MAX_POOL_SIZE} + - PENPOT_REDIS_URI=${PENPOT_REDIS_URI} + - PENPOT_OBJECTS_STORAGE_BACKEND=${PENPOT_OBJECTS_STORAGE_BACKEND} + - PENPOT_OBJECTS_STORAGE_S3_ENDPOINT=${PENPOT_OBJECTS_STORAGE_S3_ENDPOINT} + - PENPOT_OBJECTS_STORAGE_S3_BUCKET=${PENPOT_OBJECTS_STORAGE_S3_BUCKET} + - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} + - PENPOT_TMUX_SESSION=${PENPOT_TMUX_SESSION} + - PENPOT_TMUX_ATTACH=${PENPOT_TMUX_ATTACH} + + # Agentic devenv: set to a commit/tag to update Serena on startup, + # leave empty to skip update and use the version baked into the image. + - SERENA_UPDATE_VERSION=1.5.0 + + networks: + default: + aliases: + - main + + redis: + image: valkey/valkey:8.1 + hostname: "${PENPOT_VALKEY_HOSTNAME}" + container_name: "${PENPOT_VALKEY_CONTAINER_NAME}" + restart: always + command: valkey-server --save 120 1 --loglevel warning + volumes: + - "valkey_data:/data" + + networks: + default: + aliases: + - redis diff --git a/docker/devenv/docker-compose.yaml b/docker/devenv/docker-compose.yaml deleted file mode 100644 index 6963b18d8a..0000000000 --- a/docker/devenv/docker-compose.yaml +++ /dev/null @@ -1,180 +0,0 @@ -networks: - default: - driver: bridge - ipam: - config: - - subnet: 172.177.9.0/24 - -volumes: - postgres_data_pg16: - user_data: - minio_data: - valkey_data: - -services: - main: - privileged: true - image: "penpotapp/devenv:latest" - build: - context: "." - container_name: "penpot-devenv-main" - stop_signal: SIGINT - - depends_on: - - postgres - - redis - # - keycloak - - volumes: - - "user_data:/home/penpot/" - - "${PWD}:/home/penpot/penpot:z" - - ports: - - 3447:3447 - - 3448:3448 - - 3449:3449 - - 3449:3449/udp - - 3450:3450 - - 6006:6006 - - 6060:6060 - - 6061:6061 - - 6062:6062 - - 6063:6063 - - 6064:6064 - - 9000:9000 - - 9001:9001 - - 9090:9090 - - 9091:9091 - - # MCP - - 4400:4400 - - 4401:4401 - - 4402:4402 - - 4403:4403 - - # Plugins - - 4200:4200 - - 4201:4201 - - 4202:4202 - - # Serena MCP server (agentic mode only) - - ${SERENA_EXTERNAL_PORT:-14281}:14281 - - ${SERENA_DASHBOARD_EXTERNAL_PORT:-14282}:24282 - - environment: - - EXTERNAL_UID=${CURRENT_USER_ID} - # SMTP setup - - PENPOT_SMTP_ENABLED=true - - PENPOT_SMTP_DEFAULT_FROM=no-reply@example.com - - PENPOT_SMTP_DEFAULT_REPLY_TO=no-reply@example.com - - PENPOT_SMTP_HOST=mailer - - PENPOT_SMTP_PORT=1025 - - PENPOT_SMTP_USERNAME= - - PENPOT_SMTP_PASSWORD= - - PENPOT_SMTP_SSL=false - - PENPOT_SMTP_TLS=false - - # LDAP setup - - PENPOT_LDAP_HOST=ldap - - PENPOT_LDAP_PORT=10389 - - PENPOT_LDAP_SSL=false - - PENPOT_LDAP_STARTTLS=false - - PENPOT_LDAP_BASE_DN=ou=people,dc=planetexpress,dc=com - - PENPOT_LDAP_BIND_DN=cn=admin,dc=planetexpress,dc=com - - PENPOT_LDAP_BIND_PASSWORD=GoodNewsEveryone - - PENPOT_LDAP_ATTRS_USERNAME=uid - - PENPOT_LDAP_ATTRS_EMAIL=mail - - PENPOT_LDAP_ATTRS_FULLNAME=cn - - PENPOT_LDAP_ATTRS_PHOTO=jpegPhoto - - # agentic devenv - # Serena update: set to a commit/tag to update Serena on startup, leave empty to skip update and use the version in the image - - SERENA_UPDATE_VERSION=1.5.0 - - networks: - default: - aliases: - - main - - minio: - image: "minio/minio:RELEASE.2025-04-03T14-56-28Z" - command: minio server /mnt/data --console-address ":9001" - - volumes: - - "minio_data:/mnt/data" - - environment: - - MINIO_ROOT_USER=minioadmin - - MINIO_ROOT_PASSWORD=minioadmin - - networks: - default: - aliases: - - minio - - postgres: - image: postgres:16.8 - command: postgres -c config_file=/etc/postgresql.conf - restart: always - stop_signal: SIGINT - environment: - - POSTGRES_INITDB_ARGS=--data-checksums - - POSTGRES_DB=penpot - - POSTGRES_USER=penpot - - POSTGRES_PASSWORD=penpot - volumes: - - ./files/postgresql.conf:/etc/postgresql.conf:z - - ./files/postgresql_init.sql:/docker-entrypoint-initdb.d/init.sql:z - - postgres_data_pg16:/var/lib/postgresql/data - networks: - default: - aliases: - - postgres - - redis: - image: valkey/valkey:8.1 - hostname: "penpot-devenv-valkey" - container_name: "penpot-devenv-valkey" - restart: always - command: valkey-server --save 120 1 --loglevel warning - volumes: - - "valkey_data:/data" - - networks: - default: - aliases: - - redis - - mailer: - image: sj26/mailcatcher:latest - restart: always - expose: - - '1025' - ports: - - "1080:1080" - - networks: - default: - aliases: - - mailer - - - # https://github.com/rroemhild/docker-test-openldap - ldap: - image: rroemhild/test-openldap:2.1 - expose: - - '10389' - - '10636' - ports: - - "10389:10389" - - "10636:10636" - ulimits: - nofile: - soft: 1024 - hard: 1024 - - networks: - default: - aliases: - - ldap - diff --git a/docker/devenv/files/start-tmux.sh b/docker/devenv/files/start-tmux.sh index ae61f32c5c..835290941b 100755 --- a/docker/devenv/files/start-tmux.sh +++ b/docker/devenv/files/start-tmux.sh @@ -6,6 +6,23 @@ cd ~; source ~/.bashrc +PENPOT_TMUX_SESSION="${PENPOT_TMUX_SESSION:-penpot}" +PENPOT_TMUX_ATTACH="${PENPOT_TMUX_ATTACH:-true}" + +function attach_or_exit() { + if [ "$PENPOT_TMUX_ATTACH" = "true" ]; then + exec tmux -2 attach-session -t "$PENPOT_TMUX_SESSION" + fi + + echo "[start-tmux.sh] tmux session '$PENPOT_TMUX_SESSION' is running detached" + exit 0 +} + +if tmux has-session -t "$PENPOT_TMUX_SESSION" 2>/dev/null; then + echo "[start-tmux.sh] Reusing existing tmux session '$PENPOT_TMUX_SESSION'" + attach_or_exit +fi + echo "[start-tmux.sh] Installing node dependencies" pushd ~/penpot/frontend/ ./scripts/setup; @@ -14,32 +31,32 @@ pushd ~/penpot/exporter/ ./scripts/setup; popd -tmux -2 new-session -d -s penpot +tmux -2 new-session -d -s "$PENPOT_TMUX_SESSION" -tmux rename-window -t penpot:0 'frontend watch' -tmux select-window -t penpot:0 -tmux send-keys -t penpot 'cd penpot/frontend' enter C-l -tmux send-keys -t penpot './scripts/watch app' enter +tmux rename-window -t "$PENPOT_TMUX_SESSION:0" 'frontend watch' +tmux select-window -t "$PENPOT_TMUX_SESSION:0" +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/frontend' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/watch app' enter -tmux new-window -t penpot:1 -n 'frontend storybook' -tmux select-window -t penpot:1 -tmux send-keys -t penpot 'cd penpot/frontend' enter C-l -tmux send-keys -t penpot './scripts/watch storybook' enter +tmux new-window -t "$PENPOT_TMUX_SESSION:1" -n 'frontend storybook' +tmux select-window -t "$PENPOT_TMUX_SESSION:1" +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/frontend' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/watch storybook' enter -tmux new-window -t penpot:2 -n 'exporter' -tmux select-window -t penpot:2 -tmux send-keys -t penpot 'cd penpot/exporter' enter C-l -tmux send-keys -t penpot 'rm -f target/app.js*' enter C-l -tmux send-keys -t penpot './scripts/watch' enter +tmux new-window -t "$PENPOT_TMUX_SESSION:2" -n 'exporter' +tmux select-window -t "$PENPOT_TMUX_SESSION:2" +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/exporter' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'rm -f target/app.js*' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/watch' enter -tmux split-window -v -tmux send-keys -t penpot 'cd penpot/exporter' enter C-l -tmux send-keys -t penpot './scripts/wait-and-start.sh' enter +tmux split-window -v -t "$PENPOT_TMUX_SESSION" +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/exporter' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/wait-and-start.sh' enter -tmux new-window -t penpot:3 -n 'backend' -tmux select-window -t penpot:3 -tmux send-keys -t penpot 'cd penpot/backend' enter C-l -tmux send-keys -t penpot './scripts/start-dev' enter +tmux new-window -t "$PENPOT_TMUX_SESSION:3" -n 'backend' +tmux select-window -t "$PENPOT_TMUX_SESSION:3" +tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/backend' enter C-l +tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/start-dev' enter if echo "$PENPOT_FLAGS" | grep -q "enable-mcp"; then pushd ~/penpot/mcp/ @@ -47,10 +64,10 @@ if echo "$PENPOT_FLAGS" | grep -q "enable-mcp"; then pnpm run build; popd - tmux new-window -t penpot:4 -n 'mcp' - tmux select-window -t penpot:4 - tmux send-keys -t penpot 'cd penpot/mcp' enter C-l - tmux send-keys -t penpot './scripts/start-mcp-devenv' enter + tmux new-window -t "$PENPOT_TMUX_SESSION:4" -n 'mcp' + tmux select-window -t "$PENPOT_TMUX_SESSION:4" + tmux send-keys -t "$PENPOT_TMUX_SESSION" 'cd penpot/mcp' enter C-l + tmux send-keys -t "$PENPOT_TMUX_SESSION" './scripts/start-mcp-devenv' enter fi if [ "${SERENA_ENABLED:-false}" = "true" ]; then @@ -58,9 +75,9 @@ if [ "${SERENA_ENABLED:-false}" = "true" ]; then # update Serena (use sudo since the initial Serena installation is global; see Dockerfile) sudo -E uv tool install -p 3.13 serena-agent@${SERENA_UPDATE_VERSION} --prerelease=allow fi - tmux new-window -t penpot:5 -n 'serena' - tmux select-window -t penpot:5 - tmux send-keys -t penpot "serena start-mcp-server --transport streamable-http --port 14281 --project penpot --context ${SERENA_CONTEXT} --host 0.0.0.0" enter + tmux new-window -t "$PENPOT_TMUX_SESSION:5" -n 'serena' + tmux select-window -t "$PENPOT_TMUX_SESSION:5" + tmux send-keys -t "$PENPOT_TMUX_SESSION" "serena start-mcp-server --transport streamable-http --port 14281 --project penpot --context ${SERENA_CONTEXT} --host 0.0.0.0" enter fi -tmux -2 attach-session -t penpot +attach_or_exit diff --git a/docs/technical-guide/developer/agentic-devenv.md b/docs/technical-guide/developer/agentic-devenv.md index 19940ccc00..5ae0e3c05f 100644 --- a/docs/technical-guide/developer/agentic-devenv.md +++ b/docs/technical-guide/developer/agentic-devenv.md @@ -73,6 +73,18 @@ var penpotFlags = "enable-mcp"; ./manage.sh run-devenv-agentic ``` +> **Note:** the MCP and Serena tmux windows are only added when the tmux +> session is created, not when an existing session is reattached. If you have +> already run `./manage.sh run-devenv` (non-agentic) in the current devenv +> container, the agentic command will just attach you to that session without +> starting MCP or Serena. To switch to agentic mode, kill the existing session +> first and rerun: +> +> ```bash +> docker exec penpot-devenv-main sudo -u penpot tmux kill-session -t penpot +> ./manage.sh run-devenv-agentic +> ``` + ## Opening Penpot with Remote Debugging & MCP Enabled **Enable Remote Debugging in Your Browser.** diff --git a/docs/technical-guide/developer/devenv.md b/docs/technical-guide/developer/devenv.md index 0443466e03..337253ed99 100644 --- a/docs/technical-guide/developer/devenv.md +++ b/docs/technical-guide/developer/devenv.md @@ -48,10 +48,45 @@ manage.sh script: ./manage.sh build-devenv-local # builds the local devenv docker image (called by run-devenv automatically when needed) ./manage.sh start-devenv # starts background running containers ./manage.sh run-devenv # enters to new tmux session inside of one of the running containers +./manage.sh attach-devenv # re-attaches to the tmux session inside an already-running devenv container ./manage.sh stop-devenv # stops background running containers ./manage.sh drop-devenv # removes all the containers, volumes and networks used by the devenv ``` +### Upgrading from a pre-split devenv + +The devenv compose configuration was split into two files, +`docker/devenv/docker-compose.infra.yml` (Postgres, MinIO, mailer, LDAP) and +`docker/devenv/docker-compose.main.yml` (the main devenv container and its +Valkey), joined by an external Docker network called `penpot_shared`. +Per-instance defaults live in `docker/devenv/defaults.env`. + +If you had the devenv running on the previous single-file compose setup, the +first `./manage.sh start-devenv` after pulling the new code may fail with +`minio-setup: gave up waiting for MinIO after 30 attempts`. The cause is that +docker compose silently reuses the old infra containers (`penpotdev-postgres-1`, +`penpotdev-minio-1`, `penpotdev-mailer-1`, `penpotdev-ldap-1`), which are still +attached to the auto-generated `penpotdev_default` network from the old project +layout. The freshly created `minio-setup` joins the new `penpot_shared` +network instead, so it cannot resolve the `minio` hostname. + +Migration steps (data in the named volumes is preserved): + +```bash +./manage.sh stop-devenv +docker rm penpotdev-postgres-1 penpotdev-minio-1 penpotdev-minio-setup-1 \ + penpotdev-mailer-1 penpotdev-ldap-1 \ + penpot-devenv-main penpot-devenv-valkey 2>/dev/null +docker network rm penpotdev_default 2>/dev/null +./manage.sh start-devenv +``` + +After this one-time cleanup the new compose files create fresh containers on +`penpot_shared` and `start-devenv` works normally. The named volumes +(`penpotdev_postgres_data_pg16`, `penpotdev_minio_data`, +`penpotdev_user_data`, `penpotdev_valkey_data`) are not touched, so existing +Penpot state survives. + Having the container running and tmux opened inside the container, you are free to execute commands and open as many shells as you want. diff --git a/exporter/scripts/run b/exporter/scripts/run index 4bb272fc6a..a7f8836f74 100755 --- a/exporter/scripts/run +++ b/exporter/scripts/run @@ -3,4 +3,8 @@ SCRIPT_DIR=$(dirname $0); source $SCRIPT_DIR/../../backend/scripts/_env; +if [ -f $SCRIPT_DIR/../../backend/scripts/_env.local ]; then + source $SCRIPT_DIR/../../backend/scripts/_env.local; +fi + exec node target/app.js diff --git a/exporter/scripts/wait-and-start.sh b/exporter/scripts/wait-and-start.sh index f9638eb06d..53d1bc73f0 100755 --- a/exporter/scripts/wait-and-start.sh +++ b/exporter/scripts/wait-and-start.sh @@ -3,6 +3,10 @@ SCRIPT_DIR=$(dirname $0); source $SCRIPT_DIR/../../backend/scripts/_env; +if [ -f $SCRIPT_DIR/../../backend/scripts/_env.local ]; then + source $SCRIPT_DIR/../../backend/scripts/_env.local; +fi + bb -i '(babashka.wait/wait-for-port "localhost" 9630)'; bb -i '(babashka.wait/wait-for-path "target/app.js")'; sleep 2; diff --git a/manage.sh b/manage.sh index 332febdfe8..d631d271d7 100755 --- a/manage.sh +++ b/manage.sh @@ -2,7 +2,31 @@ export ORGANIZATION="penpotapp"; export DEVENV_IMGNAME="$ORGANIZATION/devenv"; -export DEVENV_PNAME="penpotdev"; +export DEVENV_NETWORK="penpot_shared"; +export DEVENV_DEFAULTS_FILE="docker/devenv/defaults.env"; + +# Load instance configuration (project name, container names, ports, runtime +# config). Single source of truth for the devenv; consumed by both docker +# compose (via --env-file) and the shell logic below. Hard dependency — abort +# loudly if it's missing or unreadable. +# +# Host-shell env wins over file values: a value already set in the parent +# environment is preserved. This matches docker compose's own precedence rule +# for --env-file (so substitution-time and shell-time agree). +if [ ! -r "$DEVENV_DEFAULTS_FILE" ]; then + echo "manage.sh: cannot read $DEVENV_DEFAULTS_FILE" >&2 + exit 1 +fi +while IFS='=' read -r __key __value; do + [[ -z "$__key" || "$__key" =~ ^[[:space:]]*# ]] && continue + if [ -z "${!__key+x}" ]; then + export "$__key=$__value" + fi +done < "$DEVENV_DEFAULTS_FILE" +unset __key __value + +# Source path for the workspace bind mount; consumed by docker-compose.main.yml. +export PENPOT_SOURCE_PATH="${PENPOT_SOURCE_PATH:-$PWD}" export CURRENT_USER_ID=$(id -u); export CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD); @@ -17,6 +41,43 @@ export JAVA_OPTS=${JAVA_OPTS:-"-Xmx1000m -Xms50m"}; set -e +# ---------------------------------------------------------------------------- +# Function map +# +# Utility helpers +# print-current-version, setup-buildx, put-license-file +# +# Devenv image lifecycle +# build-devenv, pull-devenv, pull-devenv-if-not-exists +# +# Devenv compose plumbing (used by every *-devenv command below) +# ensure-devenv-network create the external 'penpot_shared' network +# devenv-compose wrap 'docker compose' with --env-file + both files +# devenv-main-container resolve the 'main' container id via compose ps +# devenv-main-running true if 'main' is up +# +# Devenv lifecycle (operate on the whole compose project) +# start-devenv, create-devenv, stop-devenv, drop-devenv, log-devenv +# +# Devenv interactive entry points (all operate on the running 'main' container) +# run-devenv-tmux starts 'main' if needed and execs start-tmux.sh +# interactively (this is what 'run-devenv' resolves to) +# run-devenv-agentic same as run-devenv-tmux but enables MCP + Serena +# attach-devenv pure attach to the existing tmux session; fails +# fast if the devenv or session is missing +# run-devenv-shell starts 'main' if needed and execs a bash shell +# run-devenv-isolated-shell one-shot 'docker run' (NOT compose) against the +# project user_data volume and the current PWD; used +# for ad-hoc operations that should not touch a +# running devenv +# +# Production build pipeline +# build one-shot 'docker run' that invokes a per-module +# build script inside the devenv image +# build--bundle project a module's build output into ./bundles/ +# build--docker-image package a bundle into a release docker image +# ---------------------------------------------------------------------------- + ARCH=$(uname -m) if [[ "$ARCH" == "x86_64" || "$ARCH" == "amd64" || "$ARCH" == "i386" || "$ARCH" == "i686" ]]; then @@ -80,31 +141,54 @@ function pull-devenv-if-not-exists { fi } +function ensure-devenv-network { + docker network inspect "$DEVENV_NETWORK" >/dev/null 2>&1 || docker network create "$DEVENV_NETWORK" >/dev/null +} + +function devenv-compose { + docker compose \ + --env-file "$DEVENV_DEFAULTS_FILE" \ + -f docker/devenv/docker-compose.infra.yml \ + -f docker/devenv/docker-compose.main.yml \ + "$@" +} + +function devenv-main-container { + devenv-compose ps -q main +} + +function devenv-main-running { + local container=$(devenv-main-container) + [[ -n "$container" ]] && [[ "$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)" = "true" ]] +} + function start-devenv { pull-devenv-if-not-exists $@; + ensure-devenv-network; - docker compose -p $DEVENV_PNAME -f docker/devenv/docker-compose.yaml up -d; + devenv-compose up -d; } function create-devenv { pull-devenv-if-not-exists $@; + ensure-devenv-network; - docker compose -p $DEVENV_PNAME -f docker/devenv/docker-compose.yaml create; + devenv-compose create; } function stop-devenv { - docker compose -p $DEVENV_PNAME -f docker/devenv/docker-compose.yaml stop -t 2; + devenv-compose stop -t 2; } function drop-devenv { - docker compose -p $DEVENV_PNAME -f docker/devenv/docker-compose.yaml down -t 2 -v; + devenv-compose down -t 2 -v; echo "Clean old development image $DEVENV_IMGNAME..." docker images $DEVENV_IMGNAME -q | awk '{print $3}' | xargs --no-run-if-empty docker rmi } function log-devenv { - docker compose -p $DEVENV_PNAME -f docker/devenv/docker-compose.yaml logs -f --tail=50 + devenv-compose logs -f --tail=50 } function run-devenv-tmux { @@ -117,39 +201,38 @@ function run-devenv-tmux { -e*) extra_env_args+=(-e "${1#-e}"); shift;; *) - shift;; + echo "run-devenv: unknown argument '$1'" >&2 + return 1;; esac done - if [[ ! $(docker ps -f "name=penpot-devenv-main" -q) ]]; then + if ! devenv-main-running; then start-devenv echo "Waiting for containers fully start (5s)..." sleep 5; fi + local container=$(devenv-main-container) docker exec -ti \ "${extra_env_args[@]}" \ - penpot-devenv-main sudo -EH -u penpot PENPOT_PLUGIN_DEV=$PENPOT_PLUGIN_DEV /home/start-tmux.sh + "$container" sudo -EH -u penpot PENPOT_PLUGIN_DEV=$PENPOT_PLUGIN_DEV /home/start-tmux.sh } function run-devenv-agentic { local serena_context="desktop-app" - local serena_external_port="14281" - local serena_dashboard_external_port="14282" while [[ $# -gt 0 ]]; do case "$1" in --serena-context) serena_context="$2"; shift 2;; *) - shift;; + echo "run-devenv-agentic: unknown argument '$1'" >&2 + return 1;; esac done - if [[ ! $(docker ps -f "name=penpot-devenv-main" -q) ]]; then - SERENA_EXTERNAL_PORT="$serena_external_port" \ - SERENA_DASHBOARD_EXTERNAL_PORT="$serena_dashboard_external_port" \ + if ! devenv-main-running; then start-devenv echo "Waiting for containers fully start (5s)..." sleep 5; @@ -162,19 +245,39 @@ function run-devenv-agentic { } function run-devenv-shell { - if [[ ! $(docker ps -f "name=penpot-devenv-main" -q) ]]; then + if ! devenv-main-running; then start-devenv fi + local container=$(devenv-main-container) docker exec -ti \ -e JAVA_OPTS="$JAVA_OPTS" \ -e EXTERNAL_UID=$CURRENT_USER_ID \ - penpot-devenv-main sudo -EH -u penpot $@ + "$container" sudo -EH -u penpot $@ +} + +function attach-devenv { + if ! devenv-main-running; then + echo "devenv is not running." >&2 + echo "Start it first with './manage.sh run-devenv' (or './manage.sh start-devenv' for containers only)." >&2 + return 1 + fi + + local session="${PENPOT_TMUX_SESSION:-penpot}" + local container=$(devenv-main-container) + + if ! docker exec "$container" sudo -EH -u penpot tmux has-session -t "$session" 2>/dev/null; then + echo "No tmux session '$session' inside the devenv container." >&2 + echo "Start it with './manage.sh run-devenv'." >&2 + return 1 + fi + + docker exec -ti "$container" sudo -EH -u penpot tmux attach -t "$session" } function run-devenv-isolated-shell { - docker volume create ${DEVENV_PNAME}_user_data; + docker volume create ${PENPOT_USER_DATA_VOLUME}; docker run -ti --rm \ - --mount source=${DEVENV_PNAME}_user_data,type=volume,target=/home/penpot/ \ + --mount source=${PENPOT_USER_DATA_VOLUME},type=volume,target=/home/penpot/ \ --mount source=`pwd`,type=bind,target=/home/penpot/penpot \ -e EXTERNAL_UID=$CURRENT_USER_ID \ -e BUILD_STORYBOOK=$BUILD_STORYBOOK \ @@ -217,9 +320,9 @@ function build { local script=${2:-build} pull-devenv-if-not-exists; - docker volume create ${DEVENV_PNAME}_user_data; + docker volume create ${PENPOT_USER_DATA_VOLUME}; docker run -t --rm \ - --mount source=${DEVENV_PNAME}_user_data,type=volume,target=/home/penpot/ \ + --mount source=${PENPOT_USER_DATA_VOLUME},type=volume,target=/home/penpot/ \ --mount source=`pwd`,type=bind,target=/home/penpot/penpot \ -e EXTERNAL_UID=$CURRENT_USER_ID \ -e BUILD_STORYBOOK=$BUILD_STORYBOOK \ @@ -405,6 +508,7 @@ function usage { echo " Optional -e flags are forwarded to 'docker exec' (e.g. -e MY_VAR=value)." echo "- run-devenv-agentic Like run-devenv but with additional processes for agentic development enabled." echo " Options: --serena-context CONTEXT (default: desktop-app)" + echo "- attach-devenv Attaches to the tmux session inside the running devenv container." echo "- run-devenv-shell Attaches to the running devenv container and starts a bash shell." echo "- isolated-shell Starts a bash shell in a new devenv container." echo "- log-devenv Show logs of the running devenv docker compose service." @@ -455,6 +559,9 @@ case $1 in run-devenv-agentic) run-devenv-agentic ${@:2} ;; + attach-devenv) + attach-devenv ${@:2} + ;; run-devenv-shell) run-devenv-shell ${@:2} ;;