mirror of
https://github.com/penpot/penpot.git
synced 2026-09-23 20:36:15 +00:00
5b3844c37a
* 🐳 Add upstream diagnostics to nginx access log Enrich every access-log line with the internal journey of the request: the status the backend answered (us), the time spent connecting to it (uct), the time spent waiting for its answer (urt) and the internal address that served the request (ua). A plain 502 line used to say nothing about where the request died. With this format, the tail of the line classifies the failure: connection rejected, backend accepted and hung (uct + urt under 1s), or backend stuck until read timeout. This was the missing witness in the Sep 20 incident, where nginx received connection resets with zero timeouts and zero rejections. Applied both to the production image template and the devenv config. With proxy_pass on variables there is no upstream keepalive, so uct measures one real TCP connection per request. Parsing the new fields (us, uct, urt, ua) on the log shipper is left to ops, so they can be filtered in Loki. AI-assisted-by: glm-5.3-flash * 🐳 Add stub_status endpoint for nginx metrics Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing /stub_status next to every other location of the public server. Ops can run the official nginx-prometheus-exporter as a sidecar against http://127.0.0.1:8082/stub_status and get nginx_connections_active, accepted vs handled, reading/writing/waiting and request rates in Prometheus. Binding it to localhost and its own server keeps it unreachable from outside the host and out of the public surface, and access_log off avoids polluting Loki with one line per Prometheus scrape. The base image already ships stub_status compiled in, so no image rebuild is needed. Applied both to the production image template and the devenv config. AI-assisted-by: glm-5.3-flash * ✨ Expose http server gate metrics (worker and connector) The backend already measured dispatch latency but nothing reported the state of the "house door": the xnio worker queue and threads, and the monitor-level listener counters. This was the exact blind spot of the Sep 20 incident, where the server kept answering health checks while it accepted connections and dropped them without response. Add a periodic metrics sampler that lives and dies with the http server (single daemon thread, 15s interval, each sample guarded so an unexpected error does not cancel subsequent runs) and publishes: - worker (xnio MXBean gauges): penpot_http_worker_queue_size, busy_threads, pool_size and max_pool_size. Negative samples are discarded: the MXBean transiently reports -1 on the busy thread count (verified live), and a stale negative would read as zero. - listener (Undertow connector statistics, enabled via the new :server/statistics yetti option): penpot_http_connector_active* _connections gauge and requests_total / errors_total counters. Undertow exposes absolute totals, so the sampler keeps a watermark atom and publishes deltas, skipping (and moving forward past) a counter reset. The connector-level part depends on yetti v11.11, which now accepts a :server/statistics server option (patch authored and released upstream; before it, ListenerInfo#getConnectorStatistics always returned nil). New tests cover the samplers with fake MXBean/collector statistics against real prometheus collectors, including the negative-sample filter, the delta/watermark logic and the sampler lifecycle. AI-assisted-by: glm-5.3-flash * 🐛 Include jdk.management in the backend runtime JRE The production image builds a trimmed JRE with jlink and omitted jdk.management. Without that module the OS MXBean is sun.management.BaseOperatingSystemImpl, which has no getProcessCpuTime, getOpenFileDescriptorCount nor getMaxFileDescriptorCount. The prometheus client StandardExports reads those getters reflectively and collect() swallows the NoSuchMethodException, so process_open_fds, process_max_fds and process_cpu_seconds_total silently disappeared from /metrics while the other process_* families kept flowing. Verified against Prometheus: the app job only ever exposed process_start_time_seconds, process_virtual_memory_bytes and process_resident_memory_bytes; the fd and cpu families were absent. Reproduced locally by running the backend metrics registry on a JRE built with the same jlink module list (false/false/false) and on one with jdk.management added (true/true/true). Add the module to --add-modules and pin the metric contract with backend-tests.metrics-test. AI-assisted-by: deepseek-v4.1-flash * ♻️ Build the http metrics sampler on promesa.exec Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory with promesa.exec primitives: px/scheduled-executor with a daemon thread factory, and a px/schedule chain that reschedules the next sample when the current one finishes. Beyond fitting the existing periodic-task pattern (worker/cron, rpc/rlimit), the chained schedule makes the docstring promise real: with scheduleAtFixedRate an exception escaping the runnable cancelled the following executions, while the reschedule now happens in a finally block. The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in promesa 12.0.0) to cancel the pending sample, keeping the previous halt semantics. The lifecycle test moves to the promesa predicates and a new test covers the error-resilience promise: the first sample runs, throws, and the next one is still scheduled. AI-assisted-by: deepseek-v4.1-flash * ♻️ Tighten the http metrics samplers The samplers are leaf functions: they receive what they need and publish it. Drop the internal nil guards (if there is no metrics instance or no mxbean there is nothing to call them for) and move the checks to the boundary, where the optional data is resolved: sample-http-metrics now short-circuits with some-> and when-let. Write the four worker gauges as four static operations instead of a vector of pairs walked by doseq: the set is fixed, so the collection only adds an allocation and hides each operation. Drop the ! suffix from the sample-*-metrics family: ! marks a function whose contract is to mutate state, while these report, and the mutation happens in the mtx/run! they call. The constant true return, which only existed so the removed guard tests could assert it, goes away too. Tests follow the move: the internal-guard tests are replaced by one boundary test (a nil server publishes nothing). AI-assisted-by: deepseek-v4.1-flash * 📚 Add the function design rules memory Document the rules that came out of the http metrics sampler review: preconditions are checked at the boundary instead of re-checked in the core, optional-by-design data is guarded where the optionality is born, a fixed set of operations is written statically, ! marks mutation and not reporting, and production code is not shaped for tests. Also state in the memory maintenance guide that memories must not use manual line wrapping. Linked from critical-info so it is read when designing a solution or an API, not only when touching the samplers. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap the critical-info memory lines The memory maintenance guide forbids manual line wrapping, so rewrite critical-info with one line per bullet and paragraph. A stray `*` at the start of one continuation line is dropped. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the redundant guard in the http server halt create-metrics-sampler always returns the scheduler, so the sampler is always present when integrant calls halt-key!; the nil check was dead code, same as the yt/stop! call next to it. AI-assisted-by: deepseek-v4.1-flash * ✨ Add srepl helper to delete profiles by email Add `delete-profiles-by-email!` to app.srepl.main. It accepts a single email, a comma separated list of emails or a coll of emails, resolves each profile, logs it to audit and enqueues the delete-object task. The deleted-at is backdated with the configured deletion-delay so profiles and their owned teams are purged on the next gc pass. Extract the per-email deletion logic into a private fn and reuse it from `delete-profiles-in-bulk!`. Add tests for the new `parse-emails` helper. AI-assisted-by: glm-5.3-flash