mirror of
https://github.com/penpot/penpot.git
synced 2026-11-06 10:00:38 +00:00
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Alejandro Alonso
|
a8e8a54a2b | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Alejandro Alonso
|
00dec9143d
|
🎉 Add SREPL helpers to update emails and clear bounce reports (#12668)
Support currently fixes wrong profile emails and false bounce marks with raw SQL. Add update-profile-email! (rejects collisions) and clear-email-complaint-reports! (deletes global and profile reports and unmutes the matching profile) so these ops run through the REPL. Closes #12663 |
||
|
Andrey Antukh
|
dfc55a7192 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
a53481c3b7
|
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids Backend audit events now carry the data a consumer needs to rebuild state without joining production tables: - prepare-rpc-event adds :team-id when the event carries a project or file id; commands that already loaded the team set it themselves. - add-profile-to-team! emits add-team-member on every real insert. - create-project and create-file emit the new row id. - clone-template, import-binfile and duplicate-project emit one event per created entity. - both binfile versions return {:file-ids [...]} from import-files!. Closes #12219 AI-assisted-by: deepseek-v4.1-flash * ✨ Emit delete and hard-delete audit events with entity data The event stream must let Nexus rebuild local state without joining production tables. This completes the enrichment with the delete paths: - delete-team keeps returning nil but now carries the deleted row (id, name, timestamps) in the audit props, like delete-project. - create-team adds created-at to its audit props. - delete-profile carries id, email, fullname and deleted-at; its props used to be empty because every param is namespaced. - objects-gc emits hard-delete-team/project/file per purged batch, with the deleted ids, only when the audit log is on. - delete-team-member needs no change: the delete is physical and the params already carry team-id and member-id. The nitrate push-audit-events tests now drop the membership events emitted by their own profile setup before counting. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Announce cascade deletes and missing ids in audit events The :delete-object worker task marked teams, projects and files without leaving any event, so an event-sourced projection kept them alive for weeks. It now emits one cascade-deleted event per execution with every affected id, in the same transaction, only with :audit-log on. Rows purged by objects-gc and snapshot cascades stay silent on purpose. Also: create-team-with-invitations carries the new team id, and update-team-member-role names the demoted member on owner transfers. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile delete in audit events The SREPL delete-profile! emitted no profile-id and empty props, so the row could not be attributed by any consumer. It now carries the profile id in both fields. delete-team! named the wrong cause in its context. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile restore in audit events The SREPL restore-profile! emitted no profile-id, so the consumer keyed on the zero profile instead of the restored one. It now carries the profile id like a normal profile event, and no consumer special case is needed. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Stop attributing SREPL events to the subject profile profile-id names the actor, and SREPL calls have none: setting the subject id blamed the user for an admin action and would have leaked those rows into PostHog as user activity. The subject id stays in props, where the consumer resolves it, and the event profile stays zero. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Expose file analysis data for the nexus pipeline The nexus file-security pipeline needs file content and scheduling state from the backend. This adds a read-only management method, `get-file-for-analysis`, returning the semantic projection of one file (pages, frames, texts, links, interactions; ids as strings; capped with a truncated flag), plus identity, revn, timestamps and share state. No prior projection code existed, so the method ships its own pure `file->projection`. Audit events now carry what the `:local` consumer schedules from: revn and modified-at on update-file, name and timestamps on every create-file emitter, deleted-at on delete-file, and new create/delete-share-link events with file and link ids. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename deletion audit events to action names Event names are actions, not past tense: `cascade-deleted` becomes `delete-object-with-cascade` with the same shape and transaction. The three `hard-delete-*` events merge into `hard-delete-objects`: one event per purged chunk carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`), because each GC section purges in its own chunk transactions and a single event could never share them all. Every submit stays inside its delete transaction, `triggered-by` stays annotated, and silent paths stay silent. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Read worker parallelism knobs with plain config keys The runner wiring looked up `:app.main/worker-*-parallelism` (namespaced) while the schema and the environment provide plain keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and `PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers always ran with the fallback. Read the keys the config holds. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix binfile audit test with a real file row `submit-file-created-event` reads the light columns back from the row, so the mocked importer must return an existing file id: a phantom id emits no event and the test saw zero `create-file` events. Create the file first and use its id. The audit-log memory now tells both bulk-event helpers apart. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix audit event fidelity from review findings The review of the audit enrichment found data-fidelity gaps; fix them while the branch is fresh. Membership events now belong to the actor: the flows where the owner adds somebody else pass an explicit actor id, the role is derived from the flags (the table has no role column), and the event carries the request context like its siblings. Also: team-id resolution skips read events, duplicate-project emits the project name and timestamps, delete-profile is idempotent again, the GC tests assert physical purge, and get-file-for-analysis leaves an audit trace. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Nest cascade ids under :cascade in delete event The cascade event carried every affected id flat next to the object identity; nest them under `:cascade` so the payload reads as header plus affected set. The event is an action like its siblings, not a command. Drop the `:audit-log` flag re-check: `submit` already fans out per flag. AI-assisted-by: muse-spark-1.3-contributor * 📎 Update copyright name on file header The company name changed; a single header in this branch still carried the old one. Align it with the current form. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix nil event-id on webhook trigger audit rows process-event generated the row id inside append-audit-entry, so the :process-webhook-event task never received it and the trigger row stored a nil :event-id. Assign the id up-front and share it between the stored row and the worker params; the telemetry shadow row keeps its own id to avoid a PK collision. Add regression tests covering the handoff and the end-to-end trigger row. AI-assisted-by: muse-spark-1.3-contributor |
||
|
Andrey Antukh
|
0c0336da42 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
3bbc1c3079 |
♻️ Make cascade helpers stream rows and enqueue tasks
The four `process-deleted-*-cascade` helpers of `app.srepl.main` loaded every soft-deleted row of a table into memory (`db/exec!`) and ran `:delete-object` inline (`wrk/invoke!`), so they could not be used against production: `file` alone has 2.1M deleted rows. They now share a single `submit-deleted-cascade!` that reads the rows with a server-side cursor (`db/plan`, in a read-only transaction) and submits one task per row, so the work goes through the workers like any other task. The row id goes as `label` with `dedupe` on, and the helpers accept `:limit`, `:ids` and `:dry-run?` to run bounded batches. Public names and their zero-arity form are unchanged. Tests cover the task params, the row's own date (not `now`), the `limit`/`ids`/`dry-run?` options and that each helper reads its own table. AI-assisted-by: deepseek-v4.1-flash |
||
|
Andrey Antukh
|
5b3844c37a |
✨ Add observability improvements (#11854)
* 🐳 Add upstream diagnostics to nginx access log Enrich every access-log line with the internal journey of the request: the status the backend answered (us), the time spent connecting to it (uct), the time spent waiting for its answer (urt) and the internal address that served the request (ua). A plain 502 line used to say nothing about where the request died. With this format, the tail of the line classifies the failure: connection rejected, backend accepted and hung (uct + urt under 1s), or backend stuck until read timeout. This was the missing witness in the Sep 20 incident, where nginx received connection resets with zero timeouts and zero rejections. Applied both to the production image template and the devenv config. With proxy_pass on variables there is no upstream keepalive, so uct measures one real TCP connection per request. Parsing the new fields (us, uct, urt, ua) on the log shipper is left to ops, so they can be filtered in Loki. AI-assisted-by: glm-5.3-flash * 🐳 Add stub_status endpoint for nginx metrics Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing /stub_status next to every other location of the public server. Ops can run the official nginx-prometheus-exporter as a sidecar against http://127.0.0.1:8082/stub_status and get nginx_connections_active, accepted vs handled, reading/writing/waiting and request rates in Prometheus. Binding it to localhost and its own server keeps it unreachable from outside the host and out of the public surface, and access_log off avoids polluting Loki with one line per Prometheus scrape. The base image already ships stub_status compiled in, so no image rebuild is needed. Applied both to the production image template and the devenv config. AI-assisted-by: glm-5.3-flash * ✨ Expose http server gate metrics (worker and connector) The backend already measured dispatch latency but nothing reported the state of the "house door": the xnio worker queue and threads, and the monitor-level listener counters. This was the exact blind spot of the Sep 20 incident, where the server kept answering health checks while it accepted connections and dropped them without response. Add a periodic metrics sampler that lives and dies with the http server (single daemon thread, 15s interval, each sample guarded so an unexpected error does not cancel subsequent runs) and publishes: - worker (xnio MXBean gauges): penpot_http_worker_queue_size, busy_threads, pool_size and max_pool_size. Negative samples are discarded: the MXBean transiently reports -1 on the busy thread count (verified live), and a stale negative would read as zero. - listener (Undertow connector statistics, enabled via the new :server/statistics yetti option): penpot_http_connector_active* _connections gauge and requests_total / errors_total counters. Undertow exposes absolute totals, so the sampler keeps a watermark atom and publishes deltas, skipping (and moving forward past) a counter reset. The connector-level part depends on yetti v11.11, which now accepts a :server/statistics server option (patch authored and released upstream; before it, ListenerInfo#getConnectorStatistics always returned nil). New tests cover the samplers with fake MXBean/collector statistics against real prometheus collectors, including the negative-sample filter, the delta/watermark logic and the sampler lifecycle. AI-assisted-by: glm-5.3-flash * 🐛 Include jdk.management in the backend runtime JRE The production image builds a trimmed JRE with jlink and omitted jdk.management. Without that module the OS MXBean is sun.management.BaseOperatingSystemImpl, which has no getProcessCpuTime, getOpenFileDescriptorCount nor getMaxFileDescriptorCount. The prometheus client StandardExports reads those getters reflectively and collect() swallows the NoSuchMethodException, so process_open_fds, process_max_fds and process_cpu_seconds_total silently disappeared from /metrics while the other process_* families kept flowing. Verified against Prometheus: the app job only ever exposed process_start_time_seconds, process_virtual_memory_bytes and process_resident_memory_bytes; the fd and cpu families were absent. Reproduced locally by running the backend metrics registry on a JRE built with the same jlink module list (false/false/false) and on one with jdk.management added (true/true/true). Add the module to --add-modules and pin the metric contract with backend-tests.metrics-test. AI-assisted-by: deepseek-v4.1-flash * ♻️ Build the http metrics sampler on promesa.exec Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory with promesa.exec primitives: px/scheduled-executor with a daemon thread factory, and a px/schedule chain that reschedules the next sample when the current one finishes. Beyond fitting the existing periodic-task pattern (worker/cron, rpc/rlimit), the chained schedule makes the docstring promise real: with scheduleAtFixedRate an exception escaping the runnable cancelled the following executions, while the reschedule now happens in a finally block. The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in promesa 12.0.0) to cancel the pending sample, keeping the previous halt semantics. The lifecycle test moves to the promesa predicates and a new test covers the error-resilience promise: the first sample runs, throws, and the next one is still scheduled. AI-assisted-by: deepseek-v4.1-flash * ♻️ Tighten the http metrics samplers The samplers are leaf functions: they receive what they need and publish it. Drop the internal nil guards (if there is no metrics instance or no mxbean there is nothing to call them for) and move the checks to the boundary, where the optional data is resolved: sample-http-metrics now short-circuits with some-> and when-let. Write the four worker gauges as four static operations instead of a vector of pairs walked by doseq: the set is fixed, so the collection only adds an allocation and hides each operation. Drop the ! suffix from the sample-*-metrics family: ! marks a function whose contract is to mutate state, while these report, and the mutation happens in the mtx/run! they call. The constant true return, which only existed so the removed guard tests could assert it, goes away too. Tests follow the move: the internal-guard tests are replaced by one boundary test (a nil server publishes nothing). AI-assisted-by: deepseek-v4.1-flash * 📚 Add the function design rules memory Document the rules that came out of the http metrics sampler review: preconditions are checked at the boundary instead of re-checked in the core, optional-by-design data is guarded where the optionality is born, a fixed set of operations is written statically, ! marks mutation and not reporting, and production code is not shaped for tests. Also state in the memory maintenance guide that memories must not use manual line wrapping. Linked from critical-info so it is read when designing a solution or an API, not only when touching the samplers. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap the critical-info memory lines The memory maintenance guide forbids manual line wrapping, so rewrite critical-info with one line per bullet and paragraph. A stray `*` at the start of one continuation line is dropped. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the redundant guard in the http server halt create-metrics-sampler always returns the scheduler, so the sampler is always present when integrant calls halt-key!; the nil check was dead code, same as the yt/stop! call next to it. AI-assisted-by: deepseek-v4.1-flash * ✨ Add srepl helper to delete profiles by email Add `delete-profiles-by-email!` to app.srepl.main. It accepts a single email, a comma separated list of emails or a coll of emails, resolves each profile, logs it to audit and enqueues the delete-object task. The deleted-at is backdated with the configured deletion-delay so profiles and their owned teams are purged on the next gc pass. Extract the per-email deletion logic into a private fn and reuse it from `delete-profiles-in-bulk!`. Add tests for the new `parse-emails` helper. AI-assisted-by: glm-5.3-flash |
||
|
Andrey Antukh
|
89e91ba372
|
✨ Add observability improvements (#11854)
* 🐳 Add upstream diagnostics to nginx access log Enrich every access-log line with the internal journey of the request: the status the backend answered (us), the time spent connecting to it (uct), the time spent waiting for its answer (urt) and the internal address that served the request (ua). A plain 502 line used to say nothing about where the request died. With this format, the tail of the line classifies the failure: connection rejected, backend accepted and hung (uct + urt under 1s), or backend stuck until read timeout. This was the missing witness in the Sep 20 incident, where nginx received connection resets with zero timeouts and zero rejections. Applied both to the production image template and the devenv config. With proxy_pass on variables there is no upstream keepalive, so uct measures one real TCP connection per request. Parsing the new fields (us, uct, urt, ua) on the log shipper is left to ops, so they can be filtered in Loki. AI-assisted-by: glm-5.3-flash * 🐳 Add stub_status endpoint for nginx metrics Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing /stub_status next to every other location of the public server. Ops can run the official nginx-prometheus-exporter as a sidecar against http://127.0.0.1:8082/stub_status and get nginx_connections_active, accepted vs handled, reading/writing/waiting and request rates in Prometheus. Binding it to localhost and its own server keeps it unreachable from outside the host and out of the public surface, and access_log off avoids polluting Loki with one line per Prometheus scrape. The base image already ships stub_status compiled in, so no image rebuild is needed. Applied both to the production image template and the devenv config. AI-assisted-by: glm-5.3-flash * ✨ Expose http server gate metrics (worker and connector) The backend already measured dispatch latency but nothing reported the state of the "house door": the xnio worker queue and threads, and the monitor-level listener counters. This was the exact blind spot of the Sep 20 incident, where the server kept answering health checks while it accepted connections and dropped them without response. Add a periodic metrics sampler that lives and dies with the http server (single daemon thread, 15s interval, each sample guarded so an unexpected error does not cancel subsequent runs) and publishes: - worker (xnio MXBean gauges): penpot_http_worker_queue_size, busy_threads, pool_size and max_pool_size. Negative samples are discarded: the MXBean transiently reports -1 on the busy thread count (verified live), and a stale negative would read as zero. - listener (Undertow connector statistics, enabled via the new :server/statistics yetti option): penpot_http_connector_active* _connections gauge and requests_total / errors_total counters. Undertow exposes absolute totals, so the sampler keeps a watermark atom and publishes deltas, skipping (and moving forward past) a counter reset. The connector-level part depends on yetti v11.11, which now accepts a :server/statistics server option (patch authored and released upstream; before it, ListenerInfo#getConnectorStatistics always returned nil). New tests cover the samplers with fake MXBean/collector statistics against real prometheus collectors, including the negative-sample filter, the delta/watermark logic and the sampler lifecycle. AI-assisted-by: glm-5.3-flash * 🐛 Include jdk.management in the backend runtime JRE The production image builds a trimmed JRE with jlink and omitted jdk.management. Without that module the OS MXBean is sun.management.BaseOperatingSystemImpl, which has no getProcessCpuTime, getOpenFileDescriptorCount nor getMaxFileDescriptorCount. The prometheus client StandardExports reads those getters reflectively and collect() swallows the NoSuchMethodException, so process_open_fds, process_max_fds and process_cpu_seconds_total silently disappeared from /metrics while the other process_* families kept flowing. Verified against Prometheus: the app job only ever exposed process_start_time_seconds, process_virtual_memory_bytes and process_resident_memory_bytes; the fd and cpu families were absent. Reproduced locally by running the backend metrics registry on a JRE built with the same jlink module list (false/false/false) and on one with jdk.management added (true/true/true). Add the module to --add-modules and pin the metric contract with backend-tests.metrics-test. AI-assisted-by: deepseek-v4.1-flash * ♻️ Build the http metrics sampler on promesa.exec Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory with promesa.exec primitives: px/scheduled-executor with a daemon thread factory, and a px/schedule chain that reschedules the next sample when the current one finishes. Beyond fitting the existing periodic-task pattern (worker/cron, rpc/rlimit), the chained schedule makes the docstring promise real: with scheduleAtFixedRate an exception escaping the runnable cancelled the following executions, while the reschedule now happens in a finally block. The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in promesa 12.0.0) to cancel the pending sample, keeping the previous halt semantics. The lifecycle test moves to the promesa predicates and a new test covers the error-resilience promise: the first sample runs, throws, and the next one is still scheduled. AI-assisted-by: deepseek-v4.1-flash * ♻️ Tighten the http metrics samplers The samplers are leaf functions: they receive what they need and publish it. Drop the internal nil guards (if there is no metrics instance or no mxbean there is nothing to call them for) and move the checks to the boundary, where the optional data is resolved: sample-http-metrics now short-circuits with some-> and when-let. Write the four worker gauges as four static operations instead of a vector of pairs walked by doseq: the set is fixed, so the collection only adds an allocation and hides each operation. Drop the ! suffix from the sample-*-metrics family: ! marks a function whose contract is to mutate state, while these report, and the mutation happens in the mtx/run! they call. The constant true return, which only existed so the removed guard tests could assert it, goes away too. Tests follow the move: the internal-guard tests are replaced by one boundary test (a nil server publishes nothing). AI-assisted-by: deepseek-v4.1-flash * 📚 Add the function design rules memory Document the rules that came out of the http metrics sampler review: preconditions are checked at the boundary instead of re-checked in the core, optional-by-design data is guarded where the optionality is born, a fixed set of operations is written statically, ! marks mutation and not reporting, and production code is not shaped for tests. Also state in the memory maintenance guide that memories must not use manual line wrapping. Linked from critical-info so it is read when designing a solution or an API, not only when touching the samplers. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap the critical-info memory lines The memory maintenance guide forbids manual line wrapping, so rewrite critical-info with one line per bullet and paragraph. A stray `*` at the start of one continuation line is dropped. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the redundant guard in the http server halt create-metrics-sampler always returns the scheduler, so the sampler is always present when integrant calls halt-key!; the nil check was dead code, same as the yt/stop! call next to it. AI-assisted-by: deepseek-v4.1-flash * ✨ Add srepl helper to delete profiles by email Add `delete-profiles-by-email!` to app.srepl.main. It accepts a single email, a comma separated list of emails or a coll of emails, resolves each profile, logs it to audit and enqueues the delete-object task. The deleted-at is backdated with the configured deletion-delay so profiles and their owned teams are purged on the next gc pass. Extract the per-email deletion logic into a private fn and reuse it from `delete-profiles-in-bulk!`. Add tests for the new `parse-emails` helper. AI-assisted-by: glm-5.3-flash |