148 Commits

Author SHA1 Message Date
Dr. Dominik Jain
6031f6c318
⬆️ Upgrade MCP SDK to v2, removing SSE support and HTTP session management (#11841)
* ⬆️ Upgrade MCP SDK to v2 and remove HTTP sessions

MCP's per-request protocol removes the need to retain HTTP sessions.
Use the v2 handler to manage each request's transport and lifecycle,
so requests can reach any server instance without session affinity
or the workaround that adopts sessions through private SDK fields.

Keep legacy SSE support and the shared plugin and Redis bridges.
Remove the shared expiry checker, including legacy SSE idle expiry;
SSE connections now remain until disconnection or server shutdown.
Verify stateless requests, token isolation, and legacy compatibility.

Resolves #11827

AI-assisted-by: gpt-6-astra

* 🔥 Remove legacy MCP SSE support

Use Streamable HTTP as the sole MCP client transport so the server no
longer needs a separate SSE connection registry or lifecycle.

Remove /sse and /messages, their nginx routes, and the server-legacy
dependency. Legacy SSE clients must switch to /mcp; older Streamable
HTTP clients remain supported. Document the migration and verify that
the removed endpoints return 404.

Resolves #11846

AI-assisted-by: gpt-6-astra
2026-09-23 15:07:19 +02:00
Andrey Antukh
adb0fe01d2 🔧 Drop loopback bind on the stub_status endpoint
A compose port mapping delivers traffic to the container address,
never to loopback, so `listen 127.0.0.1:8082` made
`ports: <host>:8082` fail from the host. Both configs (image
template and devenv) now use `listen 8082`, which binds every
interface and matches the implicit bind of the public
`listen 8080 default_server`.

Rewrite both block comments to state the new bind and who decides
access from outside the host. The scrape URI stays on
127.0.0.1:8082: it still reaches the socket.

AI-assisted-by: mimo-v2.6-flash-free
2026-09-23 12:44:37 +00:00
Andrey Antukh
89e91ba372
Add observability improvements (#11854)
* 🐳 Add upstream diagnostics to nginx access log

Enrich every access-log line with the internal journey of the request:
the status the backend answered (us), the time spent connecting to it
(uct), the time spent waiting for its answer (urt) and the internal
address that served the request (ua).

A plain 502 line used to say nothing about where the request died. With
this format, the tail of the line classifies the failure: connection
rejected, backend accepted and hung (uct + urt under 1s), or backend
stuck until read timeout. This was the missing witness in the Sep 20
incident, where nginx received connection resets with zero timeouts and
zero rejections.

Applied both to the production image template and the devenv config.
With proxy_pass on variables there is no upstream keepalive, so uct
measures one real TCP connection per request.

Parsing the new fields (us, uct, urt, ua) on the log shipper is left to
ops, so they can be filtered in Loki.

AI-assisted-by: glm-5.3-flash

* 🐳 Add stub_status endpoint for nginx metrics

Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing
/stub_status next to every other location of the public server. Ops can
run the official nginx-prometheus-exporter as a sidecar against
http://127.0.0.1:8082/stub_status and get nginx_connections_active,
accepted vs handled, reading/writing/waiting and request rates in
Prometheus.

Binding it to localhost and its own server keeps it unreachable from
outside the host and out of the public surface, and access_log off
avoids polluting Loki with one line per Prometheus scrape. The base
image already ships stub_status compiled in, so no image rebuild is
needed.

Applied both to the production image template and the devenv config.

AI-assisted-by: glm-5.3-flash

*  Expose http server gate metrics (worker and connector)

The backend already measured dispatch latency but nothing reported the
state of the "house door": the xnio worker queue and threads, and the
monitor-level listener counters. This was the exact blind spot of the
Sep 20 incident, where the server kept answering health checks while it
accepted connections and dropped them without response.

Add a periodic metrics sampler that lives and dies with the http
server (single daemon thread, 15s interval, each sample guarded so an
unexpected error does not cancel subsequent runs) and publishes:

- worker (xnio MXBean gauges): penpot_http_worker_queue_size,
  busy_threads, pool_size and max_pool_size. Negative samples are
  discarded: the MXBean transiently reports -1 on the busy thread
  count (verified live), and a stale negative would read as zero.
- listener (Undertow connector statistics, enabled via the new
  :server/statistics yetti option): penpot_http_connector_active*
  _connections gauge and requests_total / errors_total counters.
  Undertow exposes absolute totals, so the sampler keeps a watermark
  atom and publishes deltas, skipping (and moving forward past) a
  counter reset.

The connector-level part depends on yetti v11.11, which now accepts
a :server/statistics server option (patch authored and released
upstream; before it, ListenerInfo#getConnectorStatistics always
returned nil).

New tests cover the samplers with fake MXBean/collector statistics
against real prometheus collectors, including the negative-sample
filter, the delta/watermark logic and the sampler lifecycle.

AI-assisted-by: glm-5.3-flash

* 🐛 Include jdk.management in the backend runtime JRE

The production image builds a trimmed JRE with jlink and omitted
jdk.management. Without that module the OS MXBean is
sun.management.BaseOperatingSystemImpl, which has no
getProcessCpuTime, getOpenFileDescriptorCount nor
getMaxFileDescriptorCount. The prometheus client StandardExports
reads those getters reflectively and collect() swallows the
NoSuchMethodException, so process_open_fds, process_max_fds and
process_cpu_seconds_total silently disappeared from /metrics while
the other process_* families kept flowing.

Verified against Prometheus: the app job only ever exposed
process_start_time_seconds, process_virtual_memory_bytes and
process_resident_memory_bytes; the fd and cpu families were absent.
Reproduced locally by running the backend metrics registry on a JRE
built with the same jlink module list (false/false/false) and on one
with jdk.management added (true/true/true).

Add the module to --add-modules and pin the metric contract with
backend-tests.metrics-test.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Build the http metrics sampler on promesa.exec

Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory
with promesa.exec primitives: px/scheduled-executor with a daemon
thread factory, and a px/schedule chain that reschedules the next
sample when the current one finishes.

Beyond fitting the existing periodic-task pattern (worker/cron,
rpc/rlimit), the chained schedule makes the docstring promise real:
with scheduleAtFixedRate an exception escaping the runnable cancelled
the following executions, while the reschedule now happens in a
finally block.

The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in
promesa 12.0.0) to cancel the pending sample, keeping the previous
halt semantics.

The lifecycle test moves to the promesa predicates and a new test
covers the error-resilience promise: the first sample runs, throws,
and the next one is still scheduled.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Tighten the http metrics samplers

The samplers are leaf functions: they receive what they need and
publish it. Drop the internal nil guards (if there is no metrics
instance or no mxbean there is nothing to call them for) and move the
checks to the boundary, where the optional data is resolved:
sample-http-metrics now short-circuits with some-> and when-let.

Write the four worker gauges as four static operations instead of a
vector of pairs walked by doseq: the set is fixed, so the collection
only adds an allocation and hides each operation.

Drop the ! suffix from the sample-*-metrics family: ! marks a function
whose contract is to mutate state, while these report, and the mutation
happens in the mtx/run! they call. The constant true return, which only
existed so the removed guard tests could assert it, goes away too.

Tests follow the move: the internal-guard tests are replaced by one
boundary test (a nil server publishes nothing).

AI-assisted-by: deepseek-v4.1-flash

* 📚 Add the function design rules memory

Document the rules that came out of the http metrics sampler review:
preconditions are checked at the boundary instead of re-checked in the
core, optional-by-design data is guarded where the optionality is born,
a fixed set of operations is written statically, ! marks mutation and
not reporting, and production code is not shaped for tests.

Also state in the memory maintenance guide that memories must not use
manual line wrapping.

Linked from critical-info so it is read when designing a solution or
an API, not only when touching the samplers.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Unwrap the critical-info memory lines

The memory maintenance guide forbids manual line wrapping, so rewrite
critical-info with one line per bullet and paragraph. A stray `*` at
the start of one continuation line is dropped.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop the redundant guard in the http server halt

create-metrics-sampler always returns the scheduler, so the sampler is
always present when integrant calls halt-key!; the nil check was dead
code, same as the yt/stop! call next to it.

AI-assisted-by: deepseek-v4.1-flash

*  Add srepl helper to delete profiles by email

Add `delete-profiles-by-email!` to app.srepl.main. It accepts a
single email, a comma separated list of emails or a coll of emails,
resolves each profile, logs it to audit and enqueues the
delete-object task. The deleted-at is backdated with the configured
deletion-delay so profiles and their owned teams are purged on the
next gc pass.

Extract the per-email deletion logic into a private fn and reuse it
from `delete-profiles-in-bulk!`. Add tests for the new
`parse-emails` helper.

AI-assisted-by: glm-5.3-flash
2026-09-23 12:50:18 +02:00
Andrey Antukh
117c8db0bb Merge remote-tracking branch 'origin/staging' into develop 2026-09-22 10:24:30 +02:00
Andrey Antukh
d68531b783
⬆️ Update devenv dependencies (#11790)
* ⬆️ Update devenv dependencies

Update Node.js, OpenCode, clj-kondo, Babashka, Pixi, GitHub CLI, uv,
and Serena to their current stable releases.

AI-assisted-by: gpt-5.6-sol

* ⬆️ Update devenv to Java 27

Use Zulu JDK 27 in the development image for compatibility testing.
Update the official checksums for both supported architectures.

AI-assisted-by: gpt-5.6-sol

* 🐳 Replace MinIO with RustFS in devenv

Run RustFS as the development S3 service and wait for its health check.
Install a pinned AWS CLI with checksums and use it to create the bucket
idempotently from each backend entry point.

Keep the old MinIO volume untouched and use a new RustFS volume.

AI-assisted-by: gpt-5.6-sol

* 🐳 Replace MailCatcher with persistent Mailpit

Run Mailpit as the devenv SMTP sink while preserving mailer:1025 and the
localhost:1080 UI.

Store its SQLite inbox in a named volume and wait for the readiness
endpoint before starting runtime containers. Bind the web UI to loopback so
development emails stay local.

AI-assisted-by: gpt-5.6-sol

* ⬆️ Update Node.js to 24.21.0

Align the host NVM version with the Node.js version used by devenv.

AI-assisted-by: gpt-5.6-sol

* ⬆️ Update devenv to PostgreSQL 18.6

Run PostgreSQL 18 with its versioned volume layout and a TCP readiness
check that ignores the temporary initialization server.

Install the matching client, create penpot_nexus, and preserve the old
PostgreSQL 16 volume for rollback or logical migration.

AI-assisted-by: gpt-5.6-sol

* 🐳 Expose RustFS ports in devenv

Publish the RustFS S3 API and management console on localhost port 9000
and 9001.

Keep both bindings on loopback so object storage is not exposed to the local
network.

AI-assisted-by: gpt-5.6-sol

* 🐳 Install standalone pnpm in devenv

Install pnpm 12.5.0 from architecture-specific release archives and
verify their published checksums.

Remove the Corepack setup while allowing pnpm to honor the project
packageManager pins.

AI-assisted-by: gpt-5.6-sol

* 🔥 Remove corepack, use system pnpm everywhere

Corepack is gone from Node 25+, so every `corepack enable` call
fails. pnpm now ships as a system binary (devenv, CI runners and
Docker images install it directly) and auto-downloads the version
pinned in `packageManager` on mismatch.

Scripts, workflows and Dockerfiles call `pnpm` straight away; the
three deploy workflows use a single `pnpm/setup@v2` step; and the
new `scripts/sync-pnpm-version` stamps all 35 `packageManager`
fields from the system pnpm, replacing the `corepack use` sweep.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix exporter watch missing render-wasm build step

The exporter watch compiled CLJS requiring the generated
src/app/wasm/shared.js, which only render-wasm/build export
produces. Without it shadow-cljs failed with a cryptic missing
./shared.js dependency. Run build:wasm before watching, as
the frontend watch:app and exporter scripts/build already do.

AI-assisted-by: muse-spark-1.3-contributor

* 🔧 Add opencode V2 support and adapt plugins

Register the penpot tools for both opencode V1 (server())
and V2 (setup() with JSON Schema inputs) from a single
dependency-free plugin file, sharing the psql and
paren-repair runners between both paths.

Install the opencode2 binary side-by-side with V1 in the
devenv image and document the dual registration in the
paren-repair and psql memories.

AI-assisted-by: muse-spark-1.3-contributor

* ⬆️ Update pnpm and opencode
2026-09-22 10:22:31 +02:00
elhombretecla
a91d81c695 🎉 Add link preview metadata for shared links 2026-09-15 17:10:54 +02:00
David Barragán Merino
328fa0559a 🐳 Migrate imagemagick and devenv images to DHI
Move docker/imagemagick/Dockerfile and docker/devenv/Dockerfile from
ubuntu:26.04 to Docker Hardened Images (Debian 13 / trixie).
imagemagick gets a true non-dev runtime with its shared libraries
vendored via ldd; devenv keeps the -dev tag as its final image since
it's an interactive development container, not a production
artifact.
2026-07-31 19:04:33 +02:00
Andrey Antukh
80c84a3331
🐛 Strip Authorization header when proxying asset redirects to S3 (#10777)
When nginx follows a backend 307 redirect to a presigned S3 URL, it was
forwarding the client's Authorization header to S3. Production S3 rejects
this because it sees two auth mechanisms (presigned URL signature +
Authorization header). MinIO in devenv is more lenient and ignores the
extra header.

Fix: add proxy_set_header Authorization "" in the @handle_redirect block.

Fixes #10776

AI-assisted-by: mimo-v2.5-pro
2026-07-22 14:00:35 +02:00
Andrey Antukh
50c0299e7f ⬆️ Update devenv dockerfile 2026-07-03 09:49:31 +02:00
Andrey Antukh
f457c68355 📎 Backport devenv improvements 2026-06-05 11:44:20 +02:00
Andrey Antukh
676e7f8646 🔧 Update devenv postgresql config 2026-05-27 11:35:21 +02:00
Andrey Antukh
cd882f9ebd 🔧 Add minor changes to devenv config 2026-05-11 12:39:03 +02:00
Francis Santiago
e9588f3939
🐳 Reuse shared Nginx security headers (#9473)
Signed-off-by: Francis Santiago <francis.santiago@kaleidos.net>
2026-05-08 14:11:09 +02:00
Andrey Antukh
c0989d4261 Merge remote-tracking branch 'origin/main' into main-staging 2026-04-28 10:08:51 +02:00
Andrey Antukh
4e1968bbab 📎 Add updated version of github cli to devenv 2026-04-28 00:01:46 +02:00
Andrey Antukh
3a39676969 Backport MCP from staging (part 1) 2026-04-20 19:37:02 +02:00
Andrey Antukh
6db3c6cf89
🐛 Fix regression on subpath support (#8793) 2026-03-26 15:43:30 +01:00
Andrey Antukh
920fbe34ad 🐛 Fix invalid deps passed to http management routes service 2026-02-10 13:46:29 +01:00
Andrey Antukh
5ec345162a Add mcp plugin into the frontend bundle 2026-02-09 17:38:40 +01:00
Andrey Antukh
1656fefdc9 Merge remote-tracking branch 'origin/staging-render' into develop 2026-02-04 16:23:46 +01:00
Andrey Antukh
c14ccc18b8 Import mcp from develop 2026-02-04 16:21:19 +01:00
Andrey Antukh
61d7dd3167 Update devenv with mcp required dependencies 2026-02-04 12:22:36 +01:00
Andrey Antukh
e8fd4698c9 🔧 Update caddy configuration 2026-01-27 15:10:53 +01:00
Pablo Alba
d5abc52dac
🎉 Add first integration with nitrate (#7803)
* 🐛 Display missing selected tokens set info (#8098)

* 🐛 Display missing selected tokens set info

*  Add integration tests to verify current active set

* 🎉 Integration with nitrate platform

* 🐛 Fix nitrate get-teams returns deleted teams

*  Add nitrate to tmux devenv

*  Add retry and validation to nitrate module

*  Add photoUrl to profile on nitrate authenticate

*  Move nitrate url to an env variable

* ♻️ Change Nitrate organization-id schema to text

* ♻️ Cleanup unused imports

* 🔧 Add control-center to nginx

*  Add create org link

* 🔧 Fix nginx entrypoint

* 🐛 Fix control-center proxy pass

* 🎉 Add nitrate licence check

* Revert " Add nitrate to tmux devenv"

This reverts commit dc6f6c458995dac55cab7be365ced0972760a058.

*  Add feature flag check

* 🐛 Rename licences for licenses

*  MR changes

*  MR changes 2

* 📎 Add the ability to have local config on start backend

* 📎 Add FIXME comment

---------

Co-authored-by: Xaviju <xavier.julian@kaleidos.net>
Co-authored-by: Juanfran <juanfran.ag@gmail.com>
Co-authored-by: Yamila Moreno <yamila.moreno@kaleidos.net>
Co-authored-by: Marina López <marina.lopez.yap@gmail.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-01-27 10:04:53 +01:00
Andrey Antukh
6470db8d5f Remove mention of yarn on several files 2026-01-22 13:55:41 +01:00
Andrey Antukh
3ecf509f3b 🔧 Add missing gif files handling on defaut nginx config 2026-01-19 13:32:01 +01:00
Andrey Antukh
b7cd315872 🐛 Fix wasm-playground on devenv 2026-01-08 13:48:09 +01:00
Andrey Antukh
36732a4bd3
Make the devenv runtine initialization yarn independent (#8023) 2026-01-07 11:21:58 +01:00
Andrey Antukh
52b8560b70 Merge branch 'staging-render' into develop 2025-12-30 15:30:56 +01:00
Andrey Antukh
824ca1bbca 🔧 Make devenv init yarn indpendent 2025-12-30 15:28:19 +01:00
Pablo Alba
35fb376a78
Add proxypass to caddyfile on devenv (#7985) 2025-12-22 09:21:22 +01:00
Andrey Antukh
eb1eeb4750 Merge remote-tracking branch 'origin/staging-render' into niwinz-develop-merge 2025-12-10 13:53:15 +01:00
Andrey Antukh
f111cbb2a4 Add better cache config on devenv nginx 2025-12-09 19:38:30 +01:00
Andrey Antukh
d04fdb5fbd
Make the dist bundle use consistent and cache-aware uris (#7911) 2025-12-09 08:05:28 +01:00
Andrey Antukh
eeaf28bb25 📎 Disable caddy logging 2025-12-02 13:27:09 +01:00
Andrey Antukh
6b8091bb90 Make devenv https and http2 capable (#7871)
Making it more similar on how it runs on production
environments and improves large amount of files loading
thanks to http2.
2025-12-02 10:49:37 +01:00
Andrey Antukh
7fe20b65dc 🔧 Add more cache efficient configuration for devenv nginx 2025-11-27 17:59:12 +01:00
Andrey Antukh
3ec001de44
🔧 Add nitrate url to devenv nginx (#7800) 2025-11-21 12:30:49 +01:00
Andrey Antukh
683db071d6 Merge remote-tracking branch 'origin/staging' into develop 2025-10-06 12:09:57 +02:00
Andrey Antukh
b72704e54b Set explicit no-buffering for sse responses 2025-10-06 09:19:48 +02:00
Andrey Antukh
e8434c3370 📎 Update devenv tmux script to start exporter using yarn 2025-10-01 10:59:41 +02:00
Belén Albeza
cb8aae4d5f 🔧 Drop the -R in chown (dockerfile mac) 2025-07-08 15:45:34 +02:00
Andrey Antukh
2d4fc3e05f ♻️ Refactor devenv build mechanism
This introduces multistage build process for devenv making
different dependencies build depend on its own (per example, when
jvm version is changed, only the jvm stage is rebuild)

This commit also introduces imagemagick 7.x custom build
in the same way as we have on public docker images, so on
devenv we use the same version.
2025-06-18 09:46:15 +02:00
Andrey Antukh
74430aad21 📎 Add missing emsd loading to devenv bashrc 2025-06-09 15:57:31 +02:00
Andrey Antukh
9584e1b02d 📎 Add missing playwright chromium install to devenv 2025-06-06 15:10:52 +02:00
Andrey Antukh
1909189ce0 Use different approach for setup cargo home 2025-06-02 22:29:39 +02:00
Andrey Antukh
46f330fef3 Move several logic from init to entrypoint on devenv
For make commands consistent independently if they are executed
inside devenv or from manage.sh
2025-06-02 19:10:48 +02:00
Andrey Antukh
f067c86b02 🔥 Remove unnecesary env vars from bashrc (devenv) 2025-06-02 19:10:20 +02:00
Andrey Antukh
d8913ab18b
Add minor changes to devenv for avoid repeated deps download (#6600)
*  Add minor changes to devenv for avoid repeated dependency download

*  Add minor changes to devenv for integrate payments service

*  Remove playwright deps install from circleci config

*  Move cargo_home to userspace on devenv start

*  Improve cache management on CI

*  Improve cargo installation

*  Add missing playwright install cmd on CI

*  Install cargo-watch on devenv

---------

Co-authored-by: David Barragán Merino <david.barragan@kaleidos.net>
2025-06-01 09:16:28 +02:00
Alejandro Alonso
2aaa2f3033
🐛 Fix template import (#6299) 2025-04-15 10:39:22 +02:00