103 Commits

Author	SHA1	Message	Date
Andrey Antukh	452f38cf5d	✨ Add Prometheus metrics for storage operations (#11700) ... * ✨ Add storage operation metrics for S3 and buckets Expose Prometheus metrics for the object storage subsystem. The S3 backend now attaches an AWS SDK MetricPublisher that counts API calls, retries and latency per operation and target. The storage layer counts logical operations and deduplication outcomes per Penpot bucket, and the assets handlers count served requests per route. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Fix storage metrics labels, errors and test gaps Address the review findings on the storage metrics commit. Label reads with the object's own backend, count failed asset serving as errors without swallowing them, and cover the failed S3 call, S3 asset path and permission-denied branches with tests. Also share the label helper and reuse the metrics test helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics and fill test gaps Address the second-round review findings on storage metrics. Unknown backends now fail explicitly and count as errors, exists stays paired with its dedup outcome, and the thumbnail, missing storage, expired reads, unknown buckets and write failure paths are covered by tests. Label coercion goes through the shared metrics helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics accuracy and coverage Address the full-branch review findings on storage metrics. Touch and delete emit only on changed rows, reads emit after the backend fetch, unknown backends fail explicitly, and tempfile mismatches count as unauthorized. Publisher nil policy, pairing rules and attempt semantics are documented and covered by tests. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Address full-branch review findings on storage metrics Touch and delete resolve labels from the row, reads stay paired, failures are covered by tests, and logging, ranges and docs are tightened. Includes the label helper unit tests and the retries wording clarification. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ⚡ Label touch and del metrics from UPDATE RETURNING The storage metrics change resolved metric labels for touch-object! and del-object! with an extra SELECT per id-based call. Since app.main instruments storage unconditionally, every GC collector and binfile import paid that extra round trip: deleting a team with 10k media objects doubled the storage_object statements exactly on the paths that already process the most rows. touch-object! and del-object! now take only the object id (UUID) and read the labels from the updated row itself via RETURNING id, backend, metadata: one statement, no pre-read, and labels that always match the row actually mutated. del-object! additionally guards on deleted_at IS NULL, so a repeated delete returns false and emits no metric. Also from the review of the full branch: extract the duplicated serve/emit/rethrow block in app.http.assets into one helper; give penpot_storage_s3_timing explicit histogram buckets up to 60s (the default cap at 7.5s hid the slow S3 calls the metric exists for); drop the unused ::target-id config key from the S3 backend and hardcode the :default target label until per-bucket routing lands. AI-assisted-by: glm-5.3-flash * ✨ Harden storage metric recording and definitions The metric definition schema is now closed and declares every key the collectors read: buckets, quantiles, max-age and reg. A typo such as a misspelled ::mdef/buckets used to compile and silently fall back to the default histogram buckets; it now fails at startup. The asset result-label fallback coerced an absent status to 500, so a future serve path without a status would have counted successes as errors. The mapping is now explicit and documented: served below 400, unauthorized for 401/403, not-found for 404, and error for everything else, including an absent status. The never-fail try/catch around metric recording existed four times with drift. One app.metrics/run-safe! helper replaces them: it no-ops on a nil metrics instance and logs the first failure per hint at warn level, then at debug, so a broken setup surfaces once without flooding the log. The S3 publisher keeps its outer try/catch: it is the SDK MetricPublisher contract boundary. AI-assisted-by: glm-5.3-flash * ✨ Make metrics mandatory and run! safe by default Recording a metric must never change the behavior of the operation being measured, so `run!` now catches recording failures itself: the first failure per metric id logs at warn, later ones at debug. This replaces the `run-safe!` helper, whose four copies had drifted, and applies the guarantee to every emit site instead of only storage. The metrics instance precondition is a plain assert, and the collector lookup stays outside the recording guard, so a missing instance fails hard even when asserts are disabled. Metrics is therefore no longer optional: the storage, s3-backend and db-pool schemas require `::mtx/metrics`, and the assets handler cfg always carries it. `wrap-publisher` no longer returns nil for a nil instance, and the db pool wires the prometheus tracker unconditionally. AI-assisted-by: deepseek-v4.1-flash	2026-09-23 15:25:21 +02:00
Andrey Antukh	89e91ba372	✨ Add observability improvements (#11854) ... * 🐳 Add upstream diagnostics to nginx access log Enrich every access-log line with the internal journey of the request: the status the backend answered (us), the time spent connecting to it (uct), the time spent waiting for its answer (urt) and the internal address that served the request (ua). A plain 502 line used to say nothing about where the request died. With this format, the tail of the line classifies the failure: connection rejected, backend accepted and hung (uct + urt under 1s), or backend stuck until read timeout. This was the missing witness in the Sep 20 incident, where nginx received connection resets with zero timeouts and zero rejections. Applied both to the production image template and the devenv config. With proxy_pass on variables there is no upstream keepalive, so uct measures one real TCP connection per request. Parsing the new fields (us, uct, urt, ua) on the log shipper is left to ops, so they can be filtered in Loki. AI-assisted-by: glm-5.3-flash * 🐳 Add stub_status endpoint for nginx metrics Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing /stub_status next to every other location of the public server. Ops can run the official nginx-prometheus-exporter as a sidecar against http://127.0.0.1:8082/stub_status and get nginx_connections_active, accepted vs handled, reading/writing/waiting and request rates in Prometheus. Binding it to localhost and its own server keeps it unreachable from outside the host and out of the public surface, and access_log off avoids polluting Loki with one line per Prometheus scrape. The base image already ships stub_status compiled in, so no image rebuild is needed. Applied both to the production image template and the devenv config. AI-assisted-by: glm-5.3-flash * ✨ Expose http server gate metrics (worker and connector) The backend already measured dispatch latency but nothing reported the state of the "house door": the xnio worker queue and threads, and the monitor-level listener counters. This was the exact blind spot of the Sep 20 incident, where the server kept answering health checks while it accepted connections and dropped them without response. Add a periodic metrics sampler that lives and dies with the http server (single daemon thread, 15s interval, each sample guarded so an unexpected error does not cancel subsequent runs) and publishes: - worker (xnio MXBean gauges): penpot_http_worker_queue_size, busy_threads, pool_size and max_pool_size. Negative samples are discarded: the MXBean transiently reports -1 on the busy thread count (verified live), and a stale negative would read as zero. - listener (Undertow connector statistics, enabled via the new :server/statistics yetti option): penpot_http_connector_active* _connections gauge and requests_total / errors_total counters. Undertow exposes absolute totals, so the sampler keeps a watermark atom and publishes deltas, skipping (and moving forward past) a counter reset. The connector-level part depends on yetti v11.11, which now accepts a :server/statistics server option (patch authored and released upstream; before it, ListenerInfo#getConnectorStatistics always returned nil). New tests cover the samplers with fake MXBean/collector statistics against real prometheus collectors, including the negative-sample filter, the delta/watermark logic and the sampler lifecycle. AI-assisted-by: glm-5.3-flash * 🐛 Include jdk.management in the backend runtime JRE The production image builds a trimmed JRE with jlink and omitted jdk.management. Without that module the OS MXBean is sun.management.BaseOperatingSystemImpl, which has no getProcessCpuTime, getOpenFileDescriptorCount nor getMaxFileDescriptorCount. The prometheus client StandardExports reads those getters reflectively and collect() swallows the NoSuchMethodException, so process_open_fds, process_max_fds and process_cpu_seconds_total silently disappeared from /metrics while the other process_* families kept flowing. Verified against Prometheus: the app job only ever exposed process_start_time_seconds, process_virtual_memory_bytes and process_resident_memory_bytes; the fd and cpu families were absent. Reproduced locally by running the backend metrics registry on a JRE built with the same jlink module list (false/false/false) and on one with jdk.management added (true/true/true). Add the module to --add-modules and pin the metric contract with backend-tests.metrics-test. AI-assisted-by: deepseek-v4.1-flash * ♻️ Build the http metrics sampler on promesa.exec Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory with promesa.exec primitives: px/scheduled-executor with a daemon thread factory, and a px/schedule chain that reschedules the next sample when the current one finishes. Beyond fitting the existing periodic-task pattern (worker/cron, rpc/rlimit), the chained schedule makes the docstring promise real: with scheduleAtFixedRate an exception escaping the runnable cancelled the following executions, while the reschedule now happens in a finally block. The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in promesa 12.0.0) to cancel the pending sample, keeping the previous halt semantics. The lifecycle test moves to the promesa predicates and a new test covers the error-resilience promise: the first sample runs, throws, and the next one is still scheduled. AI-assisted-by: deepseek-v4.1-flash * ♻️ Tighten the http metrics samplers The samplers are leaf functions: they receive what they need and publish it. Drop the internal nil guards (if there is no metrics instance or no mxbean there is nothing to call them for) and move the checks to the boundary, where the optional data is resolved: sample-http-metrics now short-circuits with some-> and when-let. Write the four worker gauges as four static operations instead of a vector of pairs walked by doseq: the set is fixed, so the collection only adds an allocation and hides each operation. Drop the ! suffix from the sample-*-metrics family: ! marks a function whose contract is to mutate state, while these report, and the mutation happens in the mtx/run! they call. The constant true return, which only existed so the removed guard tests could assert it, goes away too. Tests follow the move: the internal-guard tests are replaced by one boundary test (a nil server publishes nothing). AI-assisted-by: deepseek-v4.1-flash * 📚 Add the function design rules memory Document the rules that came out of the http metrics sampler review: preconditions are checked at the boundary instead of re-checked in the core, optional-by-design data is guarded where the optionality is born, a fixed set of operations is written statically, ! marks mutation and not reporting, and production code is not shaped for tests. Also state in the memory maintenance guide that memories must not use manual line wrapping. Linked from critical-info so it is read when designing a solution or an API, not only when touching the samplers. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap the critical-info memory lines The memory maintenance guide forbids manual line wrapping, so rewrite critical-info with one line per bullet and paragraph. A stray `*` at the start of one continuation line is dropped. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the redundant guard in the http server halt create-metrics-sampler always returns the scheduler, so the sampler is always present when integrant calls halt-key!; the nil check was dead code, same as the yt/stop! call next to it. AI-assisted-by: deepseek-v4.1-flash * ✨ Add srepl helper to delete profiles by email Add `delete-profiles-by-email!` to app.srepl.main. It accepts a single email, a comma separated list of emails or a coll of emails, resolves each profile, logs it to audit and enqueues the delete-object task. The deleted-at is backdated with the configured deletion-delay so profiles and their owned teams are purged on the next gc pass. Extract the per-email deletion logic into a private fn and reuse it from `delete-profiles-in-bulk!`. Add tests for the new `parse-emails` helper. AI-assisted-by: glm-5.3-flash	2026-09-23 12:50:18 +02:00
Andrey Antukh	1c7a73ec16	✨ Add account lockout after failed login attempts (#11402) ... * ✨ Add account lockout after failed login attempts Implement per-account brute-force protection using a Redis-backed failed-login counter. After 5 failed attempts within 15 minutes, the account is temporarily locked out and all login attempts (including with the correct password) are rejected with a 429 response. Closes #11397 AI-assisted-by: longcat-2.0 * 🐛 Bind LDAP session to directory-verified profile The account-lockout change added a shortcut that preferred the profile matching the typed email over the one returned by the LDAP directory. These can differ with aliases, UPNs, or multi-valued mail attributes, letting a user with valid LDAP credentials bind a session to another Penpot account. Keep the typed-email profile only for lockout checks. After LDAP succeeds, resolve the session profile from the directory identity as before and clear failed attempts on that profile. AI-assisted-by: deepseek-v4.1-flash	2026-09-23 12:31:50 +02:00
alonso.torres	e62546a03a	✨ Wait previous fail state before retry	2026-09-23 12:05:24 +02:00
Andrey Antukh	34b24a9d9d	✨ Retry transient saves with backoff and reconnect notice ... Classify save failures as transient or terminal (`transient-error?` over the repo retryable types plus `:invalid-save-response`). Transient failures keep the head commit queued under a new `:retrying` status and resend it with backoff (2s/8s/20s, then terminal): stamp rotation reuses the same `:commit-id`, the in-flight guard prevents double-sends, and episode tokens silence stale timers. One tagged reconnect notice per episode (hidden on save and on terminal failure, silent recovery) plus a `:retrying` save-indicator state; the browser `online` event and new edits resume the episode. Terminal failures keep the exact `:error` path. Covers tasks 4, 6 and 7 with 31 persistence tests; updates the persistence memory. Relates to #11724 AI-assisted-by: muse-spark-1.3-contributor	2026-09-23 12:05:24 +02:00
Andrey Antukh	95e551697f	🐛 Report environment failures as compact audit events ... Connectivity and gateway failures (network, offline, 502/503 and nitrate configuration) are not application defects, but offline fell through to :default and 502/503 rendered exception-page, so they reached the internal error reports and alerts with the full payload (stack plus the last events). They are now classified as environment failures and reported as audit-only handled-exception events. generate-report accepts an explicit :format, as keyword arguments or as a trailing map. :compact keeps the context header plus type, code and uri, and skips the stack, the ex-data dump (which may contain request headers) and the last-events list. flash derives the payload format from the cause, so environment failures get a compact report; the audit event name stays the canonical one requested by the caller (handled-exception/unhandled-exception) because external tooling filters on those names. Environment fingerprints drop the stack frame, so grouping does not depend on the internal call site. submit-report now requires an exception cause: a report without one is ignored instead of using a separate fallback fingerprint, so a single fingerprint format governs every report. :offline gets its own handler and both connectivity handlers show the new errors.connection-error message instead of the generic toast. Closes #11743 AI-assisted-by: deepseek-v4.1-flash	2026-09-23 12:05:24 +02:00
Andrey Antukh	ee651b86d8	🐛 Bound error report amplification with a dedup governor ... Add a report governor in app.main.errors: each report carries a fingerprint, the first occurrence is always emitted, and repeats within 2 minutes are counted and included in the next emitted report as :occurrences. The fingerprint cache is bounded by evicting the oldest entry. flash reserves the report before generating it, so suppressed occurrences do not build a report. static.cljs now passes the cause so the exception page gets a full fingerprint. Closes #11726 AI-assisted-by: deepseek-v4.1-flash	2026-09-23 12:05:24 +02:00
Andrey Antukh	2f679eaa0e	💥 Remove client-provided id from creation RPC commands (#11784) ... The seven creation commands no longer accept an optional client id: create-file, create-project, create-team, create-team-with-invitations, upload-file-media-object, create-file-media-object-from-url and assemble-file-media-object. The server always generates the identifier; a sent id is ignored. Malli maps are open and the RPC layer never strips unknown params, so the handlers that would still honor an id (create-file, create-project) now drop it explicitly. Internal callers that pass remapped ids (project duplicate, binfile import) keep working. Closes #11783 AI-assisted-by: muse-spark-1.3-contributor	2026-09-22 15:51:50 +02:00
Andrey Antukh	2041473cc4	🐛 Add regression test for viewer zoom url loop (#11821) ... Lock in the fix from 31b73460c3 (#11803) with a regression test for the exact reported scenario: loading the viewer with a URL that already contains `zoom=fill`. At 2.18.0-RC5 `update-zoom-querystring` navigated without any comparison, so the load sequence bundle-fetched → zoom-to-fill → update-zoom-querystring → nav → navigated re-ran forever and crashed the page with React error #185 ("maximum update depth exceeded"). The guard added in 31b73460c3 breaks the cycle; the new test asserts that a bundle fetch against a `zoom=fill` route emits no navigation events. Also updates the dashboard/viewer frontend memory to document the guard and the loop it prevents. AI-assisted-by: glm-5.3-flash	2026-09-22 14:58:32 +02:00
Andrey Antukh	bc3cb4bddf	🌐 Complete Catalan translations in frontend (#11741) ... * 🌐 Complete Catalan translations in frontend Complete the Catalan (ca.po) locale to 100% coverage against en.po, using es.po as support reference. Adds the 1439 missing entries across workspace, dashboard, labels, shortcuts, subscription, errors, modals and onboarding, keeping vosaltres treatment and IEC/Termcat terminology consistent with the existing strings. Normalizes placeholders and plural forms, drops the 14 stale obsolete entries and canonicalizes the file with the repo translations script. Closes #11739 AI-assisted-by: muse-spark-1.3-contributor * 📚 Add frontend translations memory with Catalan criteria Record the PO workflow, the sync fuzzy-flag gotcha and the Catalan glossary and tone agreed upon while completing ca.po, and link the new memory from the frontend core routing. AI-assisted-by: muse-spark-1.3-contributor * 🔧 Add gettext to devenv image Provide msgfmt and msgattrib in the dev environment for checking PO translation files. AI-assisted-by: muse-spark-1.3-contributor * 🌐 Fix Catalan translations and add PO checker Review of the missing-whitespace pattern found ~90 glued words across 75 entries, plus 4 lost plural forms and 2 placeholder mismatches verified against tr call sites. All fixed in ca.po. Adds frontend/scripts/check-translations.js (vocabulary-free PO QA: glued words, punctuation, placeholders, plurals) with --self-test, wired as pnpm run check-translations and documented in mem:frontend/translations. AI-assisted-by: muse-spark-1.3-contributor * 🌐 Multi-locale PO checker with word catalogs Split the checker engine from its word lists: ca/es catalogs now live in scripts/check-translations/words.<locale>.txt and all messages are in English. Adds an es seed (calibrated to zero errors) and fixes 7 typos it found in es.po. Universal checks (placeholders, plurals, punctuation) run without a catalog. AI-assisted-by: muse-spark-1.3-contributor * 🌐 Merge PO checker into translations.js Fold check-translations.js into translations.js as a check subcommand reusing its locale helpers; word lists stay in scripts/check-translations/words.<locale>.txt. Also fixes the getopts stopEarly bug that made -l useless after the command (sync -l ca synced every locale), drops dead lodash import and code, unifies help and exit codes. Removes the check-translations package alias; use translations.js check -l <locale> with explicit -l. AI-assisted-by: muse-spark-1.3-contributor * 🌐 Keep unused placeholders out of the gate Reverts the %s-stripping on unused auth.terms-privacy-agreement: the links mirror its markdown sibling and a reactivation may need them. Placeholder mismatches on #, unused keys now warn instead of failing, and the rule is recorded in mem:frontend/translations. AI-assisted-by: muse-spark-1.3-contributor	2026-09-22 14:10:01 +02:00
Andrey Antukh	2255266d45	✨ Enable closed schemas for RPC methods (#11136) ... * ✨ Enable closed schemas for RPC methods * 🐛 Fix duplicate make-dummy-request test helper definition The branch added a variadic DummyRequest/make-dummy-request pair but left the pre-existing single-arg definition in place. Because it was loaded last, zero-arg (make-dummy-request) calls added by prepare-rpc-params and rpc-nitrate-test threw ArityException, which broke 384 tests and caused 14 downstream assertion failures. Remove the stale duplicate so the variadic definition is the only one, and drop the now-unused yrq alias and duplicate yres alias. AI-assisted-by: deepseek-v4.1-flash * ✨ Add focused tests for make-dummy-request helper Pin the call contract of make-dummy-request, which the suite uses in three styles: no arguments, a single options map, and keyword arguments. The helper's redefinition shadowing in 8ca95adb98 was only caught by a full-suite run with hundreds of unrelated errors; these tests fail locally in a focused --focus run. Cover the zero-arg defaults, map and keyword overrides, the :body-bytes -> ByteArrayInputStream wrapping, :body-stream precedence, and cookie readback. Also clarify the docstring to list all supported call styles. AI-assisted-by: deepseek-v4.1-flash * 🚑 Prevent RPC client params from overriding auth context Strip qualified keys from decoded request params before merging them with the server-built auth context, so transit bodies can no longer override ::profile-id, ::auth-type or ::token-perms. Adds a regression test proving the override and the fix. AI-assisted-by: muse-spark-1.3-contributor * 📚 Merge backend subtleties memories under generic name Rename rpc-db-worker-subtleties to subtleties and fold in http-storage-filedata-subtleties, so the name no longer enumerates topics. Update all mem: references accordingly. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add realistic tests for RPC auth override Cover the transit wire vector and the real wrapped :get-profile method with two database profiles, proving a session cannot read another profile by smuggling :app.rpc/profile-id in the body. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add e2e test for RPC auth context override Parametrize rpcPost with contentType, accept and query so e2e can send hand-written transit bodies without new dependencies. The new test proves a transit-smuggled :app.rpc/profile-id no longer overrides the session in get-profile. Also fix the demo email assertion in auth-flow to the current uuid format. AI-assisted-by: muse-spark-1.3-contributor	2026-09-22 14:03:02 +02:00
Andrey Antukh	117c8db0bb	Merge remote-tracking branch 'origin/staging' into develop	2026-09-22 10:24:30 +02:00
Andrey Antukh	5c22f5bfb7	⚡ Build the frontend bundle once for all E2E suites (#11792) ... * ⚡ Build the frontend bundle once for all E2E suites Merge tests-integration, tests-composable-suite and tests-plugin-api-suite into one "CI: E2E" workflow. Each of the three ran its own full frontend/scripts/build on every PR, so one PR paid the build three times. The new build-bundle job restores actions/cache key frontend-bundle-<sha>, runs frontend/scripts/build only on a miss and saves the key before the job ends. The integration shards, the composable suite and the mocked Plugin API suite now all need build-bundle and restore the same key with fail-on-cache-miss, so none of them builds. A workflow re-run of the same SHA reuses the cached bundle instead of rebuilding it. Triggers become the union of the previous paths (frontend, common, render-wasm, plugins): the bundle embeds the built plugins, so a plugins change runs the whole set. workflow_dispatch keeps running the integration job only, as before. Job names are kept identical on purpose: they are the GitHub check contexts and branch protection may match them by name. Docs: new mem:frontend/e2e-ci-workflow records the build-once contract, referenced from mem:frontend/core and mem:frontend/testing; the composable memory and both suite READMEs are updated. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix mocked plugin suites crashing without frontend deps The mocked CI drivers shelled out to frontend/scripts/e2e-server.js, which imports express from frontend/node_modules. CI jobs install only plugins/ deps, so the import failed with ERR_MODULE_NOT_FOUND and the run timed out waiting for localhost:3000. Serve the prebuilt bundle with a zero-dependency static server built into each driver (ci/static-server.ts, kept in sync in both suites) plus node:test coverage for it. AI-assisted-by: muse-spark-1.3-contributor	2026-09-22 10:23:15 +02:00
Andrey Antukh	d68531b783	⬆️ Update devenv dependencies (#11790) ... * ⬆️ Update devenv dependencies Update Node.js, OpenCode, clj-kondo, Babashka, Pixi, GitHub CLI, uv, and Serena to their current stable releases. AI-assisted-by: gpt-5.6-sol * ⬆️ Update devenv to Java 27 Use Zulu JDK 27 in the development image for compatibility testing. Update the official checksums for both supported architectures. AI-assisted-by: gpt-5.6-sol * 🐳 Replace MinIO with RustFS in devenv Run RustFS as the development S3 service and wait for its health check. Install a pinned AWS CLI with checksums and use it to create the bucket idempotently from each backend entry point. Keep the old MinIO volume untouched and use a new RustFS volume. AI-assisted-by: gpt-5.6-sol * 🐳 Replace MailCatcher with persistent Mailpit Run Mailpit as the devenv SMTP sink while preserving mailer:1025 and the localhost:1080 UI. Store its SQLite inbox in a named volume and wait for the readiness endpoint before starting runtime containers. Bind the web UI to loopback so development emails stay local. AI-assisted-by: gpt-5.6-sol * ⬆️ Update Node.js to 24.21.0 Align the host NVM version with the Node.js version used by devenv. AI-assisted-by: gpt-5.6-sol * ⬆️ Update devenv to PostgreSQL 18.6 Run PostgreSQL 18 with its versioned volume layout and a TCP readiness check that ignores the temporary initialization server. Install the matching client, create penpot_nexus, and preserve the old PostgreSQL 16 volume for rollback or logical migration. AI-assisted-by: gpt-5.6-sol * 🐳 Expose RustFS ports in devenv Publish the RustFS S3 API and management console on localhost port 9000 and 9001. Keep both bindings on loopback so object storage is not exposed to the local network. AI-assisted-by: gpt-5.6-sol * 🐳 Install standalone pnpm in devenv Install pnpm 12.5.0 from architecture-specific release archives and verify their published checksums. Remove the Corepack setup while allowing pnpm to honor the project packageManager pins. AI-assisted-by: gpt-5.6-sol * 🔥 Remove corepack, use system pnpm everywhere Corepack is gone from Node 25+, so every `corepack enable` call fails. pnpm now ships as a system binary (devenv, CI runners and Docker images install it directly) and auto-downloads the version pinned in `packageManager` on mismatch. Scripts, workflows and Dockerfiles call `pnpm` straight away; the three deploy workflows use a single `pnpm/setup@v2` step; and the new `scripts/sync-pnpm-version` stamps all 35 `packageManager` fields from the system pnpm, replacing the `corepack use` sweep. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix exporter watch missing render-wasm build step The exporter watch compiled CLJS requiring the generated src/app/wasm/shared.js, which only render-wasm/build export produces. Without it shadow-cljs failed with a cryptic missing ./shared.js dependency. Run build:wasm before watching, as the frontend watch:app and exporter scripts/build already do. AI-assisted-by: muse-spark-1.3-contributor * 🔧 Add opencode V2 support and adapt plugins Register the penpot tools for both opencode V1 (server()) and V2 (setup() with JSON Schema inputs) from a single dependency-free plugin file, sharing the psql and paren-repair runners between both paths. Install the opencode2 binary side-by-side with V1 in the devenv image and document the dual registration in the paren-repair and psql memories. AI-assisted-by: muse-spark-1.3-contributor * ⬆️ Update pnpm and opencode	2026-09-22 10:22:31 +02:00
Andrey Antukh	fe89e9e52c	Merge remote-tracking branch 'origin/staging' into develop	2026-09-17 20:23:41 +02:00
Andrey Antukh	5b3e36489c	📚 Document int?/integer? predicate coverage in Clojure memory ... Review assumed int? was 32-bit; it covers Long/Integer/Short/Byte. Note it in mem:clojure/idioms so the mistake is not repeated. AI-assisted-by: muse-spark-1.3-contributor	2026-09-17 20:23:22 +02:00
Andrey Antukh	30e52af22e	📎 Backport creating-issue serena memories from develop	2026-09-16 19:48:50 +02:00
Andrey Antukh	df383be6b2	Merge remote-tracking branch 'origin/staging' into develop	2026-09-15 19:59:54 +02:00
Andrey Antukh	8ff99f0766	📚 Document how to add issues as sub-issues ... Add the verified REST procedure for linking an issue as a sub-issue of an umbrella/EPIC: get the REST id, POST to the parent's sub_issues endpoint with a typed -F field, and verify both directions. Route it from the create-issue skill. AI-assisted-by: deepseek-v4.1-flash	2026-09-15 19:56:48 +02:00
Andrey Antukh	91ecfefa31	📚 Add EPIC issue type to creating-issues memory ... The repository defines an EPIC issue type, but the memory only listed Bug, Enhancement, Feature, Task, Question and Docs. Add the EPIC row with its type id and its mapping entry so umbrella tracking issues are typed consistently. AI-assisted-by: deepseek-v4.1-flash	2026-09-15 16:12:02 +00:00
Andrey Antukh	ad7e035b63	✨ Add indirection for upload-chunks storage via new table (#11651) ... * ✨ Add upload_session_chunk indirection for chunked uploads Chunks now live in the upload_session_chunk table with non-deleting foreign keys to storage_object and upload_session, instead of tempfile objects with session metadata. Reads go through a JOIN, so chunk state never scans storage_object. Uploads validate the live session, reject duplicate indexes, and store objects in the new upload-session bucket without extra metadata. Assemble removes mappings and marks the session consumed; objects-gc procedurally purges consumed and stalled sessions, touching referenced objects first. Touched-gc and deleted-gc handle the new bucket, and upload-session-gc is removed. Closes #11644 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix quota, give-up and coverage for session chunks Exclude consumed sessions from the sessions-per-profile quota so finished uploads free their slot at once. Remove chunk mappings before the gc-deleted give-up delete to respect the NO ACTION keys. Catch java.sql.SQLException for duplicate chunks. Cover the profile-owned session purge and the UNIQUE race backstop with tests. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Align chunked upload tests with upload_session_chunk Drop the duplicate-index tests written against metadata-backed chunks; the UNIQUE mapping makes those cases unrepresentable and the new tests cover them. Rewrite the rejected-duplicate tests to expect :validation/:chunk-already-exists and assert against the upload_session_chunk table, and scope the chunk-too-large "nothing stored" check to the mapping table. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Use NO ACTION DEFERRABLE session FKs in single migration Fold the profile FK change into 0154 so the feature ships one migration. All three upload session FKs use ON DELETE NO ACTION DEFERRABLE: identical to RESTRICT in normal operation, but deferrable for tooling that relies on SET CONSTRAINTS ALL DEFERRED. Extend the RESTRICT test to the direct profile delete. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Reserve chunk slot before writing blob in upload-chunk Make object_id nullable and insert the mapping with NULL inside the session-locking transaction, then write the blob outside it and link it with a conditional update. A failed write removes the mapping and reraises; a mid-flight death leaves a NULL row and the client starts a new session. AI-assisted-by: muse-spark-1.3-contributor * 🔥 Remove redundant session_id index on upload_session_chunk The UNIQUE(session_id, chunk_index) btree already serves session_id-only lookups and the session FK check through its leftmost column, so the standalone index only taxed the per-chunk INSERT path. Verified with EXPLAIN on an equivalent table shape. AI-assisted-by: muse-spark-1.3-contributor * ⚡ Merge chunk touch and delete into single RETURNING query Replace the SELECT-then-DELETE round-trip in delete-upload-sessions! with DELETE ... RETURNING object_id, touching each returned object. Same semantics, one less query per purged session. Follows the RETURNING pattern already used in file-gc. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Let objects-gc own chunk mapping deletion Assemble-chunks now only marks the session as consumed; the chunk mappings stay until objects-gc purges them (touching the chunk objects first), leaving a single procedural deletion path for consumed, stalled and profile-purge sessions. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix font-deletion GC expectations for chunk objects Update final storage-gc-touched counts to include the two chunk objects touched by objects-gc when purging consumed upload sessions (8/5/5 instead of 6/3/3). AI-assisted-by: muse-spark-1.3-contributor * 🐛 Release chunk reservation when the link UPDATE fails Review feedback on #11651: the link UPDATE in upload-chunk could leave a NULL reservation behind, blocking retries of the same index with :chunk-already-exists. Remove the reservation when the link fails so the client can retry in the same session; the orphaned blob stays touched for touched-gc. Also realign the process-bucket! cond branches in gc-touched. Tests: chunked-upload-link-failure-releases-slot and chunked-upload-null-reservation-blocks-retry. AI-assisted-by: muse-spark-1.3-contributor	2026-09-15 17:26:43 +02:00
Andrey Antukh	b660ea9d53	✨ Route the app by query string with screen key ... Move SPA routing out of the URL fragment into the normal query string. The screen travels in a reserved `screen` key holding the route name (`?screen=workspace&team-id=…`); every other param keeps its name. `rt/nav` and `rt/resolve` keep their signatures. This deletes the fragment-mirroring URL surgery, simplifies link-preview (the server sees everything) and nginx (single path, no SPA fallback rules needed), and migrates OIDC redirects, email links, e2e helpers and plugin test utils to the new format. Legacy `#/…` URLs translate client-side for one Penpot version (`legacy-routes`, marked TODO(next-version)); non-SPA paths are untouched. AI-assisted-by: muse-spark-1.3-contributor	2026-09-15 17:10:54 +02:00
Andrey Antukh	39ca4c264c	🔥 Remove onboarding A/B test and welcome file creation (#11707) ... Drop the onboarding-03 experiment consulted through external-feature-flag, keeping the false-branch behavior: registration never requests a welcome file and the workspace never shows the onboarding modals. Remove the now-unused welcome-file machinery on the backend (RPC wiring, welcome_file namespace, welcome-file-id prop and the post-login redirect). Keep the external-feature-flag helper as the seam for future experiments and note it in mem:frontend/core. Closes #11705 AI-assisted-by: muse-spark-1.3-contributor	2026-09-15 14:14:32 +02:00
Andrey Antukh	bda8459d89	Merge remote-tracking branch 'origin/staging' into develop	2026-09-11 12:57:51 +02:00
Andrey Antukh	09736aa4c9	✨ Enforce commit body line wrapping ... Add a body line-length validator to scripts/check-commit. It fails when a body line exceeds 76 characters, exempting trailers, URLs, and unbreakable tokens. The 76 limit leaves room for git log's four-space indent in an 80-column terminal. Align the subject limit with the documented 70 characters; the checker allowed 90 before. Document the rule as a hard, verifiable requirement in AGENTS.md, CONTRIBUTING.md, the create-commit skill, and the workflow memory, and point at scripts/check-commit. Add tests for the validator and the subject length rule. AI-assisted-by: deepseek-flash	2026-09-11 08:10:49 +00:00
Andrey Antukh	37dab75e1a	Merge remote-tracking branch 'origin/staging' into develop	2026-09-10 20:29:44 +02:00
Andrey Antukh	eca1d81692	🔧 Remove legacy pnpm build key and clarify updating doc ... Drop the ignored-since-pnpm-11 onlyBuiltDependencies entry from render-wasm/pnpm-workspace.yaml, keeping allowBuilds as the single source of build approvals. Clarify the updating-pnpm gotcha so it no longer claims pnpm writes ignoredBuiltDependencies. AI-assisted-by: muse-spark-1.3-contributor	2026-09-09 11:45:00 +02:00
Andrey Antukh	831953c41e	Merge remote-tracking branch 'origin/staging' into develop	2026-09-09 11:02:06 +02:00
Andrey Antukh	8a3540336b	📚 Add local-ci skill and scripts/ci memory ... Teach agents to verify their changes with ./scripts/ci: module list, task selection flags, log locations under .ci-logs/, and per-module workflows (lint-only pass, --fix, --paren-repair, common/ consumer checks). Register the skill in the skills README, point AGENTS.md at the new memory, and add the script to the critical-info dev scripts. AI-assisted-by: omen-alpha	2026-09-09 11:01:49 +02:00
Andrey Antukh	c1bd3cb9f0	Merge remote-tracking branch 'origin/staging' into develop	2026-09-09 10:31:41 +02:00
Andrey Antukh	5c474939ac	🔧 Pin all pnpm workspaces to one shared pnpm store ... Set storeDir in every pnpm-workspace.yaml: `.pnpm-store` at the repo root and `../.pnpm-store` in the ten module workspaces, so all of them resolve to <repo>/.pnpm-store. pnpm resolves the value against the workspace root, and nested workspaces do not inherit settings, which had left the root workspace and the modules on two different stores. Add scripts/clean-node-modules: removes every workspace node_modules in one pass (ignores external/ and .opencode/), keeps the shared store unless --store removes it too. Verified: every workspace resolves the same store path; reinstalls after a full clean reuse the cache with zero downloads; frozen-lockfile installs pass in all 11 workspaces with no lockfile changes; the frontend storybook suite stays green. AI-assisted-by: omen-alpha	2026-09-09 08:04:11 +02:00
Andrey Antukh	f68c266380	♻️ Fold composable-test-suite into the plugins workspace ... Drop the nested pnpm-workspace.yaml and pnpm-lock.yaml from plugins/apps/composable-test-suite. It stays a plain member of the plugins workspace (apps/** glob); its dependencies already resolve through plugins/pnpm-lock.yaml, so no lockfile change is needed. Update the pnpm update procedure memory to the single-workspace-per- module layout, keep the invariant that members carry no nested pnpm-workspace.yaml, and drop the now-stale 12-workspace count from critical-info. AI-assisted-by: omen-alpha	2026-09-09 08:04:11 +02:00
Andrey Antukh	e4d1816117	⬆️ Update pnpm to 12.3.4 across all workspaces ... Run `corepack use pnpm@next-12` (resolved to 12.3.4) on every directory with a package.json: the repo root, the 11 module workspaces, and all submodules. Every packageManager field now carries the same pinned version and hash; the root and backend move off 11.20.0. Fix the composable-test-suite workspace config (esbuild allowBuilds placeholder left by pnpm 12) so its install passes, and add the missing packageManager fields to frontend/packages/ui and mcp/packages/plugin, since corepack only updates existing fields. Document the canonical update procedure in .serena/memories/workflow/updating-pnpm.md. AI-assisted-by: omen-alpha	2026-09-09 08:04:11 +02:00
Andrey Antukh	3b49ff532f	Merge remote-tracking branch 'origin/staging' into develop	2026-09-08 21:41:43 +02:00
Andrey Antukh	fbfef42145	♻️ Move skills and plans to .agents for cross-tool use ... .agents is the shared home that opencode, Claude Code (through the .claude/skills symlink) and Codex all read, so the skills and the saved plans now live there instead of .opencode: - .opencode/skills moved to .agents/skills (24 files, no content changes). - .opencode/plans moved to .agents/plans; the .gitignore entry follows, so plans stay untracked. - .claude/skills symlink retargeted to ../.agents/skills. - planner, make-a-plan and review-plan updated to the new plans path; new .agents/README.md documents every skill with when-to-use examples and links to each SKILL.md. - workflow/creating-issues memory: create-issue path updated. opencode discovers .agents/skills natively, so .opencode needs no reciprocal link. AI-assisted-by: omen-alpha	2026-09-08 19:40:38 +00:00
Andrey Antukh	f695553469	Merge remote-tracking branch 'origin/staging' into develop	2026-09-08 21:15:58 +02:00
Andrey Antukh	99e6d4f1ad	📚 Forbid hand-editing CHANGES.md in agent guides ... Adds the hard rule to AGENTS.md and to mem:critical-info: CHANGES.md is generated from GitHub milestones during the release process; it must be updated only via the update-changelog skill flow or on explicit user request. AI-assisted-by: omen-alpha	2026-09-08 16:39:35 +00:00
Andrey Antukh	c52778d6f6	Merge remote-tracking branch 'origin/staging' into develop	2026-09-07 11:28:06 +02:00
Andrey Antukh	5c10ea5bd6	📚 Document branch naming convention ... The creating-prs memory described a branch format (issue/... with a <type>/<short-description> fallback) that does not match actual repo practice, where issue-driven branches are issue-NNNN. Replace it with a Branch Naming section: issue-NNNN as primary, descriptive name without slashes as fallback. Add the matching public convention to CONTRIBUTING.md under Pull Requests, which previously defined no branch naming at all. AI-assisted-by: omen-alpha	2026-09-07 08:41:54 +00:00
Alejandro Alonso	c5897bc50a	⚡ Paint plain text directly onto Current (#11355) ... * ♻️ Share text layout paragraphs across modifier clones Store Skia paragraphs in Rc so TextContentLayout::clone keeps the cached layout for rotate/pan modifiers. Add layout.clear() and treat needs_update as paragraphs-empty only. * ⚡ Reuse cached Skia paragraphs when painting text Add try_paint_from_layout_cache to paint from TextContent.layout when versions match, skipping ParagraphBuilder rebuild and layout on each frame. Wire into the layered text path for plain fills without strokes or effects. * ⚡ Paint plain text directly onto Current Extend can_render_directly for stroke-free text and skip the empty save_layer in draw_text when no stroke-group opacity is set. Plain text paints into Current without the Fills/Strokes blit.	2026-09-02 16:17:06 +02:00
Andrey Antukh	f5aad7b1ae	Merge remote-tracking branch 'origin/staging' into develop	2026-09-01 12:45:17 +02:00
Andrey Antukh	73d3d63616	✨ Enable a way to provide custom opencode config on starting devenv	2026-08-31 16:31:24 +02:00
Álvaro Tejero-Cantero	38004e6bb2	✨ Add the graph subsystem and graph visualization console to the backend (#11101) ... * 🎉 Basic lbug connection for ingestion * ✨ Add Penpot-to-Ladybug graph ingest vertical slice * ✨ Use embedded Ladybug Java API instead of CLI * ♻️ Share Ladybug connection across ingest and stats * ✨ Validate graph ingest projections with Malli * ✨ Project nested shapes recursively into the graph * ⚡ Load graph ingest via Ladybug COPY bulk import * 🐛 Fix graph COPY ingest for multiline text names * ✨ Add Ladybug graph export to debug UI * ✨ Add debug graph console for in-memory Cypher queries * ✨ Add live file-change feed to debug graph console * ✨ Incrementally sync debug graph from Penpot file changes * ✨ Handle mov-objects in debug graph sync * 🐛 Fix batch delete sync and keep graph console feed alive * ♻️ Derive graph node schema from Malli registry * ✨ Add G6 graph view to debug graph console POC per work/g6/plan.md. New /dbg/actions/graph-data exports the in-memory Ladybug session as plain JSON (per-table node queries + multi-table IsChildOf match, row cap 100k with truncation flag). Console page renders it with AntV G6 v5 (jsDelivr CDN, antv-dagre BT layout, color+glyph per node table, validated palette) and refetches debounced on live :file-change messages. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix list-column CSV ingest and serialize graph session access COPY failed on any file with container shapes: list-typed DDL columns (shapes UUID[], points STRING[], strokes JSON[], ...) were JSON-encoded in staging CSVs, which Ladybug's list parser rejects. Write Kuzu list literals instead, typed per column. Also: value->clj no longer crashes on LIST/STRUCT values (binding lacks value_get_value support; fall back to string), and the debug session Connection is now guarded by a per-session lock — it was shared unsynchronized between the msgbus sync loop and HTTP query/export handlers, and one lost DETACH DELETE was observed under concurrent refetch load. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Split graph console in two columns; add file tree and fullscreen Graph view moves to its own sticky right column (overrides .widget max-width). New /dbg/actions/graph-files endpoint lists teams -> projects -> files for the profile; the console renders it as a collapsible tree where clicking a file loads it. Maximize button fullscreens the graph panel and resizes G6 on fullscreenchange. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ♻️ Replace fullscreen with in-page expand for graph view Fullscreen API took over the whole output and broke window-manager splits (and is denied in some environments). The Expand button now toggles a fixed-position overlay covering the page while keeping browser chrome; Esc restores. Column positioning moved from inline style to the stylesheet so the expanded class can override it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Fold containers as collapsible combos in graph view Non-empty containers (Page, Frame, Group, Boolean, SVGRaw) render as nested G6 rect combos holding their own node plus direct children; Document stays a plain node. Double-click folds/expands (collapse-expand behavior); collapsed combos show a member count and re-route child edges. Fold state is read back from getComboData and re-marked on every refetch, so it survives live redraws. Layout gains sortByCombo to keep same-rank nodes grouped by box. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ⚡ Fix graph view freeze on large files; add fold toggle and root rule Root cause of the tab freeze on ~1700-node files was G6's default entrance animation: measured 1700 nodes at >2 min animated vs 1.5 s with animation: false. Secondary cost was antv-dagre (~7 s at that size); since IsChildOf is a tree, an O(n) tidy layout (depth = rank, post-order leaf slots, parents centered) computed client-side replaces it and renders the same file in ~1.4 s. A guard skips auto-render above 4000 nodes with an explicit Render-anyway button, so opening the console with a huge session loaded stays responsive. Folding is now switchable ('fold containers' checkbox, persisted in localStorage) and generalized: any node with children folds except the IsChildOf root of the loaded graph, so Documents (and later Projects/Teams) fold automatically once they gain a parent node. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add layout dropdown to graph view Adds a layout <select> next to the fold toggle, populated from the LAYOUTS map in the template: 'tree' (the O(n) preset layout, default) plus 13 G6 layouts (antv-dagre, dagre, circular, concentric, radial, grid, force, d3-force, force-atlas2, fruchterman, mds, combo-combined, random), all smoke-tested against combo data on this UMD build. Layout and fold toggle are independent; switching layouts recreates the graph instance (cheap with animation off); both choices persist in localStorage. antv-dagre stays available for when non-tree edges arrive. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add query-result subgraph, ws auto-reconnect, adaptive animation The Cypher result pane now offers 'Show result in graph view': any UUID found in any result cell selects the matching nodes in the cached export and the view renders the induced subgraph (edges kept when both endpoints match); 'Show full graph' resets. No graph reconstruction from the query result is needed. The notifications websocket reconnects automatically (3 s retry) and resubscribes + refetches on reopen, so backend restarts no longer permanently kill the live feed; a lost session now reports 'no graph session (backend restarted?) - reload a file' instead of a bare 404. Animation is size-adaptive: graphs (or filtered subgraphs) up to 100 nodes render animated for didactics, larger ones stay animation-free; crossing the threshold recreates the instance like a layout switch. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph toolbar, animate toggle, filter columns, repaint skip Graph view gains an on-canvas G6 toolbar (auto-fit, expand, restore - the fullscreen icons drive the existing in-page expand), an 'animate' checkbox that disables animation unconditionally when off (persisted, adaptive <=100-node rule applies only when on), and a ResizeObserver on the canvas so the panel follows window/flex resizes without touching the user's viewport. Preset tree positions are now only injected for the built-in tree layout, removing the tree-then-layout flash on animated re-renders under G6 layouts. Refetches skip the repaint when the display projection (nodes, edges, truncated) is byte-identical, so attribute-only change bursts no longer repaint. Console: default query returns s/t name+label over all edges plus filter_src_id/filter_tgt_id columns; filter_* columns are hidden from the results table (client and server render) but still feed the 'Show result in graph view' id harvest, keeping the table legible while the graph filter stays available. The query text persists in localStorage across page reloads (restored only over the default, never over a server-rendered query). Legend shows colored Unicode glyphs matching node shapes instead of squares with textual annotations. Load/Unload buttons share one row (HTML5 form attribute), and the loaded file name links to the Penpot workspace via the legacy /#/workspace/<project-id>/<file-id> route resolved client-side from the files-tree payload. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix runaway graph panel growth and blank canvas; drop Expand button Root cause of 'graph flashes on load then disappears' plus unbounded horizontal growth of the graph panel: fieldsets default to min-inline-size: min-content, so #graph-view-panel sized to its content, and the new ResizeObserver->setSize path closed a feedback loop (setSize -> slightly wider G6 canvas -> wider fieldset -> wider .dashboard flex column -> observer fires) that grew the page ~10px per frame and wiped the painted canvas on every step. Fix severs the feedback path: #graph-view-panel gets min-inline-size: 0, #graph-canvas gets overflow: hidden, and the page section gets flex: 1 1 0 with min-width: 0 so column widths are viewport-driven, never content-driven. This also fixes the original narrow-window scrollbars defect for real. The observer stays (guarded by a current-size comparison) because G6's autoResize is inert on this UMD build (verified: window resizes left the canvas size untouched); the inert autoResize flag is dropped. Legend items now join with spaces so the nowrap spans can wrap between entries. Also removes the header Expand button - the toolbar's expand/exit icons cover it, Esc still restores. Verified against the running devenv with a logged-in profile and variants_simple loaded: graph renders and persists, widths stable over multiple seconds at 1400px and 1000px viewports with no horizontal overflow, canvas follows both window shrink and grow, toolbar expand gives a full-page canvas and Esc restores. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Make the default query self-explanatory; link the Cypher docs The default query is now multi-line with // comments that explain the filter_* column convention in place (Kuzu accepts comments and blank lines mid-statement; verified against an in-memory database through the console query path). The query fieldset is retitled 'LadybugDB Cypher' with the Cypher word linking to https://docs.ladybugdb.com/cypher/. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 📎 Fix linter issues * ✨ Add Component nodes and IsInstanceOf edges * 🐛 Fix memory leak * ✨ Style Component nodes and IsInstanceOf edges in graph console Slice-3 export sends edges with a rel field. Derive tree ranking, combo derivation and fold-ability from IsChildOf only; draw other rels as overlay edges with per-rel styles (EDGE_STYLES: IsInstanceOf violet dashed, matching the new Component diamond in NODE_STYLES). Legend now lists only displayed node tables and rels, re-rendered per redraw; help text trimmed to essentials. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph diff marks with step fade to graph console Each display-changing refetch is a step: added nodes/edges get a green halo, removed ones stay as ghosts with a dashed crimson halo (nodes, fading opacity) or thicker crimson stroke (edges), re-entering layout and combos through their ghost IsChildOf edges. Marks fade linearly and drop after N steps; N is the new "fade" number input (localStorage, 0 = off). Dash + fade carry the added/removed distinction under red-green CVD (#40c057/#c2255c, deutan dE 17.4); diff is vs the previous display step, not arbitrary revisions. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Reserve chroma for changes in graph console diff mode Monochrome entity scheme: all node tables share one slate hue, lightness separates within-glyph siblings (validated, worst pair dE 17.5), SVGRaw becomes the hollow hexagon, both rels go grey with dash as the only separator. Diff marks now own all color: thick green/crimson stroke ring (dashed for removals) plus a larger, subtler halo; the legend gains +/- entries while marks are live. Two additions to guide the eye: a brief DOM-overlay pulse on age-0 elements (independent of the G6 animation gate) and a "fold unchanged" toggle that collapses every combo not on an ancestor path of a changed element. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Expand folded combos that gain changed elements setData merges datum props by id on a live G6 instance, so omitting style.collapsed retained a previous true: with "fold unchanged" on, a change inside a folded combo pulsed but never expanded it. Write the boolean explicitly both ways. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Label edge rels with compact unicode symbols Dash variants alone cannot carry the growing rel roster: EDGE_STYLES entries gain a sym rendered as a small mid-edge label with a white backing (IsInstanceOf = "∈"; IsChildOf stays unlabeled as the background structure), and the legend shows the symbol. Convention from the abacus viewer EDGE_SYM dict. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add node inspector panel to graph console Clicking a node fetches its full attribute row (MATCH (n:`Table` {id: uuid(...)}) RETURN n.*) through the query endpoint and renders non-null attrs into a panel under the canvas (count of empty attrs noted). Panel over tooltip: projected tables carry ~80 columns, and the panel persists for reading without obstructing the graph. Table/id are validated before Cypher interpolation; the listener is re-attached on every instance recreation. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Graph console QoL round "Show result in graph view" moves into an actions bar above the results table; results scroll inside a 45vh container (client and server render paths); the Loaded-session fieldset gains a live "Graph size" line that stays fresh through skipped repaints; IsInstanceOf mid-edge label becomes the spelled-out rel name (∈ read as membership, not derivation) with the legend falling back to the dash-arrow for long syms. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Sync Component library changes into the Ladybug graph * 💄 Polish graph console session panel and edge labels Loaded-session fieldset: graph size gains a resident-memory estimate (fit to graph_sizes.md: ~1.1 MiB floor + ~5.4 KiB/node) with per-table counts on hover, replacing the load-time Projection stats; loaded-at compacts to local HH:MM with the full instant on hover. Edge rel labels drop to 7 px and lose the dashed stroke — the text label alone carries rel identity. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add PNG export and hover tooltips to graph console Toolbar gains an export item: graph.toDataURL({mode: "overall"}) downloads the whole laid-out graph as graph-<revn>.png — page-chrome-free captures, also the fast path for agents debugging the console. A hover tooltip (table, label, id) backs the reduced/absent labels on dense layouts. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Prune graph console layout roster and tune overlap Remove grid, random, force, fruchterman, force-atlas2 (nothing over the kept set) and mds (stress layout degenerates to spokes on tree distances, no collision term to tune). Parameterize the keepers against node overlap — concentric/radial get preventOverlap+nodeSize, d3-force a collide radius — and shrink node labels to 7 px on those layouts (DENSE_LABEL_LAYOUTS), verified against variants_simple (72 nodes). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Highlight clicked node neighborhood in graph console click-select behavior with degree 1: the clicked element keeps a black ring, direct neighbors stay full-strength, everything else dims to 0.2 opacity (inactive state); clicking empty canvas clears. Works on edges too (selects both endpoints) and composes with the node inspector on the same click. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add overview mode: fold containers at or beyond a depth "fold >= depth" number input (root = 0, empty = off, localStorage): every combo whose container sits at that IsChildOf depth or deeper collapses, giving a top-of-file overview (e.g. 2 folds the containers hanging from a Page). Composes with fold-unchanged — depth folds first, changed ancestor paths are then drilled open. Derived fold state overrides manual folds while active. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore fold-containers as the combo master gate Since fold-unchanged and depth folding arrived, withCombos ORed them in, so unchecking "fold containers" could no longer remove the combo boxes. The checkbox is the gate again; the derived fold rules are dormant without it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console UI polish round Merge the load form and files tree into one "Load graph from Penpot" box (tree first, uuid + Load/Unload in a row); Loaded session carries HH:MM in its legend; the Live changes box stays hidden until the first change arrives; query fieldset reads "Query graph (LadybugDB Cypher)" with the link covering both terms. Drop the hover tooltips (distracting, useless zoomed out) and the resident-size estimate (per-table counts stay on hover); every toggle gets a "When set/checked ..." title. Depth fold: 0 now expands every container (no more hunting for max depth). Node inspector: two-column flow, structured or long values folded behind the file-tree disclosure triangle. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Report actual graph memory from the buffer manager graph-data gains bm-bytes (CALL bm_info() -> [mem_limit mem_usage], nil-safe, under the session lock); the session panel shows it as MiB behind the node/edge counts — real resident memory replacing the removed estimate. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console control-bar and session-panel rework Left column narrowed 440->330 px (uuid input flexes). Control bar reordered: layout first, then animate and fade (narrow inputs), then the fold set; "fold containers" renamed "foldable containers" (on = foldable, not folded). Load becomes Reload once a session exists (same operation as the removed Full-reload button — load-session! on the current id; tooltip explains the fallback role) with Unload beside it. Session panel: revisions on one line ("ingested at N · graph now M", hover explains the difference), duplicate uuid after the file name dropped. Tried and rejected: fishbone (no positions on graph data) and compact-box (G6 tree layouts walk parent->child, IsChildOf points child->parent). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Legend entries toggle node-table visibility Clicking a legend entry hides/shows that table across the view (struck-through while hidden, kept listed for re-enabling; pure client-side id filter through filteredGraphData, edges drop with their endpoints, ghosts respect it). Also: setting fold >= depth above 0 now switches foldable containers on — a positive depth was silently inert without combos. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Enable the edge-bundling plugin Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Session breadcrumb, changelog colors, spacing File line becomes team › project › file (clickable) with the resident-memory figure beside it (moved up from the graph-size line; breadcrumb resolves from the files-tree payload, so files outside the profiles teams show plain). add-obj/del-obj in Live changes wear the canvas diff colors. Paragraph margins tightened above Feed; left column 330→350 px. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Guard renders against heavy graphs; add ?safe escape hatch A heavy file could freeze the tab on load-and-render despite the animation gate: the render guard counted nodes only, and the edge-bundling plugin is iteration-heavy in edges. Guard now also trips on edges (8000), edge bundling only activates at <= 300 edges, and /dbg/graph?safe disables auto-render entirely (counts + "Render anyway"), so a page that hung can always be re-entered with the session intact. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🔥 Remove the edge-bundling plugin Bundled edges render unsmooth and ugly on this build; the gating constant goes with it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 One row per operation in the Live changes table Columns revn | op | id: the revn repeats across a batch, the op wears the canvas diff colors (shape/attrs detail on hover), and the id column shows the uuid last group with the full uuid on hover, or N/A for ops without a subject id (e.g. mov-objects). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Use app.system/system in the graph ingest helper develop renamed app.main/system to app.system/system and dropped the app.main require while this branch was away. Rebasing replays the old call, so clj-kondo reports an unresolved namespace and the ns will not load. * ✨ Put the graph subsystem behind a flag, off by default (#11075) `app.graph.ladybug` imports `com.ladybugdb.*` at namespace load. Two namespaces reach the subsystem and both required it at the top level: `app.http.debug`, which registers the `/dbg` routes, and `app.srepl.main`, which loads with the REPL server. Every backend built from this branch therefore linked the Ladybug native library into the JVM at boot, whether or not a graph was ever used. Add a `:graph` flag to `varia`, deliberately absent from `default` so that a released Penpot ships with the subsystem off. Both require sites now resolve `app.graph.*` at call time, so with the flag off no `com.ladybugdb` class is loaded. The nine `/dbg` graph routes are registered only when the flag is on, and 404 otherwise. The `/dbg` admin gate is untouched: the flag decides which routes exist, not who may reach them. When the flag is on, route init requires the subsystem eagerly, so a missing or unusable native library fails the boot rather than the first console request. No tracked file turns the flag on. `backend/scripts/_env` leaves it out, so a devenv boots with the subsystem off exactly as a released build does, and `docker/images/docker-compose.yaml`, the self-hosting distribution, is untouched. Whoever works on the graph turns it on for one checkout through the gitignored `backend/scripts/_env.local`, which every backend and exporter dev script sources right after `_env`. Verified with `-verbose:class` over a boot's namespace load plus `ig/init-key ::routes`: 9 `com.ladybugdb` classes before this change with no flag set, 0 after it with the flag off, 9 with `enable-graph`. * ⬆️ Take Ladybug 0.19.1 `com.ladybugdb/lbug` moves from 0.18.0 to 0.19.1, the current release on Maven Central. The engine fixes a SIGSEGV on an unwrapped parameter and moves parameter coercion out of JNI, so shipping 0.18.0 would land a native library into `develop` with a known crash already fixed upstream. Nothing else changes. This branch has no `app.graph.arrow`, so the top-level Arrow field-name backticking that 0.19.x retires does not exist here and there is no workaround to remove alongside the bump. AI-assisted-by: mixed models * 📎 Pin the graph console's G6 bundle to an exact version The console loaded `@antv/g6@5` from jsDelivr, a floating major range, so the JavaScript served into the page could change without a Penpot release. Pin it to 5.1.1, the version the range resolves to today. Where the dependency finally belongs is an open question for review: vendored into `backend/resources`, declared in `frontend/package.json` if the console moves out of `/dbg`, or left on the CDN. Pinning removes the floating-code problem without pre-empting that decision. AI-assisted-by: mixed models * ✨ Add graph provenance, column naming and two transforms A projected graph is a cache of one file at one revision, built by one schema, and nothing in it said so. `GraphMeta` records the file, the revision, the schema version and the producer, and is written last, so its presence also marks the build complete and its contents say whether a cached database is still worth opening. - `graph/meta.clj`: the `GraphMeta` table and its writer. - `graph/schema/contract.clj`: one place that maps a Penpot key to its graph column. The rule is snake_case of the key; every exception, be it a rename, a drop or a type override, is recorded there with its reason, so a divergence is a diff to review rather than a silent rename. - `graph/project/document.clj`: `page-id` and the inherited `component-id` are written during the tree walk, which already knows both, rather than by a post-ingest statement. `graph/sync.clj` does the same on the incremental path, so a live-synced graph matches a rebuild. - `graph/project/transforms.clj`: a registry, so adding a derived-link pass is one entry. Adds `RefersTo` (from `shape-ref`) and `FillsSwapSlot` (from `swap-slot-*` entries in `touched`, then stripped as `ctk/normal-touched-groups` does). - `graph/debug.clj`, `graph/stats.clj`: enumerate relationship tables from the catalog instead of naming them, so the console's graph view and the ingest counts pick up new edge types without being told. - `graph/debug.clj`, `http/debug.clj`: `graph-export` gains `source=session`, which snapshots the live in-memory console graph through EXPORT/IMPORT DATABASE. Live sync moves that graph away from a fresh projection, and taking it away to query elsewhere is the point of asking for it. AI-assisted-by: mixed models * 🐛 Write graph values Ladybug's CSV reader cannot carry through Cypher Three parity failures against beadpot's suite, all one cause: the bulk loader put compound and multi-line values into CSV, where Ladybug parses a field's *contents* as a literal with no escape mechanism at all. Verified against 0.18: a comma inside a list element ends the element, quotes are kept as part of the value rather than delimiting it, and the parallel reader rejects quoted newlines outright. So a value now goes through CSV only if it cannot be misread there — UUIDs, numbers, booleans, single-line strings, and lists of those. Everything else (MAP, STRUCT, STRING[]/JSON[], any string containing a newline) is written after the COPY by one Cypher statement per row, where `app.graph.ladybug` escapes properly. Parquet removes the distinction entirely and is still the right destination (masterplan P0 T1); this is what CSV can honestly do. Consequences beyond the encoding: - `touched` entries reached the graph as `:swap-slot-…`, keywords stringified with their colon, so `LinkSwapSlots` matched nothing. Keywords now render through `name`. - Shape names lost their newlines to a flattening step that existed only to keep the CSV writer happy. They are preserved. - `applied_tokens` keys are rendered camelCase, the form Penpot's own JSON encoder produces and the one beadpot's `AppliedTokenKey` holds — a MAP column's keys are values, not schema, so they are not snake_cased. - `link-component-instances!` keys on `component-file`, not `component-id` alone. The projection denormalizes `component-id` down the shape tree, after which it no longer tells an instance head from a shape inside one, and the transform linked every descendant frame; `ctk/instance-of?` requires both keys anyway. IsInstanceOf on the variants fixture: 78 -> 60, matching beadpot exactly. `app.graph.schema.nodes/format-column-value` is now the single place that knows a column's type and its contract details, used by the bulk loader and the incremental sync alike so the two cannot disagree about a value's shape. * ✨ Type graph columns as tightly as Ladybug allows Ladybug is schema-first and strongly typed: a property key gets its type at table-creation time and there is no widening later. That makes the Malli to Ladybug mapping the whole of the graph's typing, and it was leaving a lot on the table: a transform stored as `STRING`, a rect as `JSON`, a set of feature flags as a single `STRING`. A column typed `DOUBLE[4]` is four numbers a consumer reads as a tensor row; the same value as JSON is text somebody has to parse and trust. `app.graph.schema.types` now maps, in order: scalars; Penpot value types whose layout is fixed even though Malli only sees a map or a string (`::gmt/matrix` to `DOUBLE[6]`, `::gpt/point` to `DOUBLE[2]`, `::grc/rect` to `DOUBLE[4]`, `::clr/hex-color` to `UINT32`); then structure, with collections to `T[]`, `:map-of` to `MAP(k, v)`, and a closed map of scalars to a `STRUCT`. JSON is the fallback of last resort, for schemas that genuinely admit more than one shape. Two defects fell out. `::sm/set` was unmapped, so `features` and `migrations` were single strings rather than `STRING[]`, and `::sm/one-of`, how Penpot spells a closed set of keywords, was unmapped too, so `blend-mode`, `grow-type`, the constraints and every `layout-*` were mistyped. A tight column is only worth having if the writer fills it in that shape, so `app.graph.schema.values` shapes a value for its type: a matrix record into six doubles, a hex colour into a packed integer, a map into a struct's fields. Both writers go through it, so the bulk load and the incremental sync cannot disagree. What that required: - STRUCT field names must be backticked in the DDL *and* in every literal, because a grid cell has a field named `column`. The catalog reports them bare. - A struct literal's type is its field list, so every declared field must appear, and an absent one needs `cast(NULL, '<type>')`. A bare NULL is typed STRING and changes the struct's type. - `STRUCT(…)[]` starts with `STRUCT(` but is a list, so the list check comes first. - Nested lists cannot be rendered with `str`: Clojure's `[1 2]` is space-separated and Ladybug reads it as a one-element array. Three more corrections in the same area: - `project-attrs` used truthiness where it meant `some?`, so `opacity 0` and `blocked false` projected as absent. - Set-valued columns are written sorted. A set has no order, so the column varied between builds of the same file, which is precisely what stops two builds being diffable. - An empty collection is written as `[]` rather than skipped. A shape with no fills has none; NULL would say "unknown". Renamed the `kuzu-*` helpers to `ladybug-*`: Kùzu is deprecated and Ladybug substitutes it, so a name bearing the engine should bear this one. The one remaining mention cites the upstream issue Ladybug inherits. AI-assisted-by: mixed models * ✨ Add the file-level graph columns and tighten the svg ones Split out of "🐛 Declare the shape attributes stored files carry", which is now #11125 and carries only its `common/` half. This commit is the graph's own side of that change, and it stays on this branch. `app.graph.schema.contract` pins `svg_viewbox` to `DOUBLE[4]` and `svg_transform` to `DOUBLE[6]`. The shape schema types both `:map` on purpose, because legacy files hold them as plain maps rather than as `::grc/rect` and `::gmt/matrix` records, and a tighter *schema* would reject those files. A tighter *column* costs nothing, since `app.graph.schema.values/coerce` reads either form. `app.graph.schema.nodes` declares four file-level attributes as projection `:extra` rather than in `ctf/schema:file`: `:options`, `:backend`, `:comment-thread-seqn`, and `:ignore-sync-until`. Declaring them in the file schema breaks saving, measured at 185 failures, because `app.binfile.common/update-file!` derives its UPDATE column list from a file map's keys and the `file` table has no `backend` column, that value being synthesized on read. An `:extra` is local to the graph and cannot reach a write. `app.graph.project.document` lifts `:options` out of `:data` before the blob is dropped, so a consumer reads file-level configuration without opening the blob. AI-assisted-by: mixed models * ✨ Add the Arrow prerequisites for in-memory bulk load lbug pulls arrow-memory-core and arrow-vector but no allocation-manager implementation, so RootAllocator cannot be constructed; arrow-memory-netty 18.2.0 matches the arrow-vector lbug already brings and pulls only netty-buffer, netty-common, jackson and slf4j-api, all of which the backend already has. --add-opens=java.base/java.nio=ALL-UNNAMED is the second half: without it MemoryUtil's static initializer dies with an InaccessibleObjectException that surfaces as an unhelpful NoClassDefFoundError from anything touching RootAllocator. It has to be present at JVM start, hence all three places. Note app.main/restart will not pick it up — it restarts integrant inside the same JVM, so the process must be restarted. Worth a reviewer's attention: this is a JVM-wide flag added for one subsystem. It is the standard Arrow requirement and grants nothing beyond reflective access to java.nio, but it strengthens the case for putting the whole graph subsystem behind a feature flag. * ✨ Bulk load through in-memory Arrow; delete the CSV loader app.graph.arrow stages rows as Arrow VectorSchemaRoots and COPYs from them. No file is written at any point and no value is rendered as text for the engine to re-parse, so the defect class that produced three of this branch's four backend defects cannot recur. app.graph.bulk is deleted whole. csv-representable?, defer-to-cypher?, multiline?, fixup-statements, ladybug-literal, ladybug-list-element, ladybug-list-cell and staging-dir go with it, along with the post-COPY Cypher pass that emitted one SET per row. Measured before deciding: the fixup pass was ~77% execution, 16-22% parse and 6-7% round-trip, and prepared statements could not have recovered any of it — every fixup row carries a MAP column and Ladybug binds scalars only. So this replaces rather than optimizes. Marginal ingest 4.0 -> 1.21 ms/shape; ~25 s extrapolated at 20k shapes against the ~2 min the CSV path projected. Size unchanged. Four engine facts the implementation rests on, each verified against 0.18.2 with a standalone probe: - An Arrow table is not a COPY source identifier but is a MATCH-able node label. - A MAP vector's entries child must be a non-nullable struct, and MapVector.getWriter promotes it to a sparse union, so map vectors are built from an explicit Field and filled child-first. - Ladybug names a staged table's columns and struct fields from the Arrow field names and quotes none of them, so anything needing quotes must arrive quoted — hence cypher-property-key, not column-name, names the Arrow fields. - createArrowRelTable cannot resolve endpoints against a UUID-keyed node table under any encoding, so edges stage as a node table and the COPY subquery joins them. values/coerce is reused unchanged, so the Arrow and Cypher writers cannot disagree about a value's shape; nodes/column-map-key-fn is extracted so they cannot disagree about a MAP's key spelling either. Verified with pytest --graph-origin=penpot-only unchanged at 225/38/1 and --graph-origin=penpot unchanged at 258 passed / 2 pre-existing failures, both baselines re-established against a reverted backend rather than assumed; with bp graph diff between a CSV-built and an Arrow-built graph reporting "Graphs agree"; and with an adversarial round-trip carrying a quote, a backslash, a newline, a CRLF and a tab through STRING, STRING[] elements and MAP values. The diff was necessary, not belt-and-braces: both parity suites passed an earlier revision of this change that was writing EDN into every JSON column, because beadpot's assertions never parse those columns. It also showed Arrow correcting a CSV defect — an empty Component.path was being stored as NULL, because Ladybug's CSV reader cannot distinguish an empty field from an absent one. * ✨ Add a prepared-statement surface to the graph connection `app.graph.ladybug` could only run Cypher as text. Every value the sync path writes is therefore concatenated into the statement, and nothing can ask the engine whether a statement is even valid without running it. Add the four functions that close both gaps. `prepare-on-connection!` parses and binds without executing. `execute-prepared!` binds a parameter map and runs it. `exec-prepared-on-connection!` prepares every statement in a batch before executing any of them, so a parse or bind failure aborts before the first mutation. `validate-on-connection!` returns `{:ok? :error :read-only?}` instead of raising, which is what a gate wants. `->param-value` is the only `Value` constructor on the write path. It is unconditional: on lbug 0.18.2 an unwrapped parameter does not raise, it SIGSEGVs the JVM inside `lbug_value_clone`. Parameters are scalars only, because the JNI `Value` constructor takes no list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered and the `:else` branch raises rather than crashing. Two departures from the design, both closing a JNI-handle leak on the error path: `prepare-on-connection!` closes the failed `PreparedStatement` before raising, and `execute-prepared!` closes every `Value` it built, including the ones built before a later parameter was rejected. `as-statement` accepts a bare string, so the sync builders can convert to bound parameters one family at a time rather than in one commit. AI-assisted-by: mixed models * 🐛 Write the document revision to the column that exists `set-document-revision-statement` emitted `SET d.revn`, but the column is `revision`: the beadpot contract renames `:revn` and the DDL has followed it since. The statement is the last one in every sync batch, so each batch raised after its mutations had already committed, and the session's in-memory index stayed frozen at its load-time revision. Name the column through `nodes/cypher-property-key` rather than spelling it, so the DDL and the statement cannot disagree again. Found by the binder gate in the next commit, on its first run. AI-assisted-by: mixed models * ✨ Gate every sync statement template through the binder Nothing checked that the eleven Cypher templates `app.graph.sync` emits still bind against the DDL the schema registry generates. A renamed column, a dropped table or a reserved word emitted unquoted surfaced only when a live session ran the statement, and by then the batch's earlier mutations had committed. `backend-tests.graph-binder-gate-test` opens a `:memory:` database, creates the live schema on it, and *prepares* one instance of each template without executing any of them. 14 tests, 51 assertions: the eleven templates, label coverage over all twelve registered node tables, and two assertions on the gate itself, that a `RETURN` reads as read-only and a `SET` does not, and that an unbindable statement is reported rather than thrown. It was not green on HEAD: it caught `set-document-revision-statement` writing a column that no longer exists, fixed in the previous commit. Red on both injected templates tried. No `:jvm-opts` change: CI's `-M:dev:test` already carries the native access flags the engine needs. AI-assisted-by: mixed models * 🐛 Let the engine quote the Arrow field names it interpolates `node-batch` named every top-level Arrow field with backticks, so that a column whose name is a reserved word (`Page.index`, `Document.options`) survived the DDL Ladybug generates for a staged table. The engine now quotes those identifiers itself, and it does not collapse a doubled backtick, so a pre-quoted name reaches the parser as ``index`` and `createArrowTable` fails outright: Parser exception: mismatched input '``' expecting PRIMARY Name the fields with `column-name`. The `COPY` projection is Cypher rather than DDL and keeps its own backticks through `cypher-property-key`, and STRUCT member names keep theirs too: those come out of `LogicalType::toString()`, which the DDL builder does not touch, so an unquoted member called `column` still fails to parse. Measured with `probes/arrow/probe25.clj` against lbug 0.19.1: a plain top-level reserved word loads and reads back, a pre-quoted one fails to parse, a plain STRUCT member fails to parse, and a pre-quoted one loads and reads back. Also re-dates the engine facts in the `app.graph.arrow` docstring to the version they were checked against, drops the SIGSEGV note from `->param-value` now that `Connection.execute` rejects an unwrapped parameter, and removes two references to the CSV loader. AI-assisted-by: mixed models * 📚 State what the graph schema does, not what it mirrors The graph namespaces explained themselves by citing a separate project whose Python pipeline reads the graphs this backend writes. A reader of this repository does not have that project and should not need it, and a docstring that justifies a choice by pointing elsewhere cannot be checked here. Every claim survives; only the framing changes. Column names and types are Penpot's own decision, recorded with the reason for each divergence from the snake_case default. The transform registry describes the edges it materializes. The denormalizations in `app.graph.project.document` are justified by the walk already holding both answers. Three corrections fall out of the rewrite: - `app.graph.schema.contract` claimed a test, `graph_contract_test`, that walks a checked-in schema manifest and fails on any divergence. No such test exists. The paragraph is gone. - `app.graph.project.document` pointed at `app.graph.meta/projection-transforms`, which does not exist. - `app.graph.project.transforms/registry` claimed its entries were "in application order" while `apply-transforms!` reduced over the literal vector. The three registered transforms read disjoint columns, so the order is not load-bearing. The docstring now says so, and the one real ordering constraint is stated where it applies: `link-swap-slots!` strips `swap-slot-*` entries from `touched`, so anything reading `touched` has to run before it. `contract/pending-beadpot-columns` becomes `contract/unprojected-keys`. It is referenced nowhere else. AI-assisted-by: mixed models * ✨ Refuse a mutating query from the graph console `debug/query-session!` ran whatever it was handed against the session connection. A session graph is a projection of a file, rebuilt from that file by Reload, so a mutation from the console produces a graph no rebuild reproduces and no query result explains. Bind the statement against the live schema first. A statement that does not bind reports the binder's own message and executes nothing, which also turns a misspelt table or property into an immediate error instead of an empty result. A statement that binds runs only when the engine's own read/write analysis calls it read-only. The console's query box is labelled read-only. Load, Reload, Unload and live sync are unaffected: they are separate handlers and do not go through this path. AI-assisted-by: mixed models * 🐛 Keep a synced graph equal to a rebuilt one Cold projection and incremental sync are two implementations of one mapping and nothing checked that they agree. They did not. `backend-tests.graph-sync-parity-test` projects a file into one `:memory:` database, applies a change list to that database and the same list to the file data, projects the result into a second database, and diffs the two down to the row and the column. It found four disagreements, each fixed here. **Sibling order was inverted.** A container's stored `:shapes` list runs bottom to top and `IsChildOf.position` numbers children in Penpot z-order, so appending to the list means taking position 0 and pushing every sibling up. Sync instead handed each new child the next free number, so any container edited live carried its children in the opposite order to a rebuild, and a delete left a gap where a rebuild renumbers densely. `insert-position` and `renumber-siblings` put the two paths on the same rule for `:add-obj`, `:mov-objects` and `:del-obj`, including a block move and `:after-shape`. **A moved shape kept its old parent.** `:mov-objects` moved the edge and left the shape's own `parent_id` and `frame_id` columns pointing at the container it came from. Both now follow, and `frame_id` follows through the whole subtree the shape carries, as `app.common.files.changes` does for `:mov-objects`. A top-level shape's column holds `uuid/zero`, the page's root frame, while its edge points at the Page. **A container's `shapes` column went stale.** Nothing maintained it after an add, a move or a delete. It is now rebuilt from the sibling order on every change that touches a container. **Pages came out backwards.** `projection-data` reversed `:pages` before numbering them, which is right for child shapes and wrong for pages: `:pages` is the tab order and has no second ordering to undo. `Page.index` and the page's `IsChildOf.position` are now that order. One defect the test does not reach, fixed on the way past: `index-add-shape!` accepted `:component-ctx` and dropped it, so a shape added under an instance head added in the same session inherited no `component-id`. AI-assisted-by: mixed models * 🐛 Build a synced page node the way the projection does `apply-add-page` sent the new Page node through `nodes/validate-node`, which checks a map against the registry schema and returns it unchanged. Every other node on both write paths goes through `nodes/project-attrs`, which also selects the projected keys and is the single place a column-level rule can live. A rule added there reached a rebuilt page and not a synced one. AI-assisted-by: mixed models * 🐛 Let the graph view's query filter follow the graph "Show result in graph view" froze the set of node ids the query returned and filtered every later repaint against it. Live sync creates ids the set has never seen, so a shape created while a filter was on could not appear in the view at any point, and clicking "Show full graph" was the only way to see it. A node the query would no longer match stayed. Keep the query beside the ids and re-run it whenever the graph repaints, which is only when the projection actually changed. A failed re-run keeps the ids in hand and says so on the status line rather than passing a stale view off as current. `idsInResult` and `presentIds` are extracted from the two places that scraped UUIDs out of a result. Verified in the devenv: with a filter showing 108 of 276 nodes, a `:file-change` adding a Frame published on the session's msgbus topic took the view to 109 of 277, with the new node carrying its added mark, and no interaction. AI-assisted-by: mixed models * ♻️ Rename app.graph.project to app.graph.projection `project` is a Penpot noun: a team holds projects and a project holds files, and the graph will carry a `Project` node table. A namespace called `app.graph.project.document` therefore reads as "the graph of a Penpot project" and means the opposite. `projection` is the word the rest of the subsystem already uses for the operation: `projection-data`, `load-projection!`, `:projection` in the ingest report, and `app.graph.schema.projection`. Pure rename. Both namespaces and every alias move; nothing else changes. AI-assisted-by: mixed models * 📎 Apply the project formatter to the graph namespaces `cljfmt check src/ test/` is a step of the Backend workflow and these two files did not pass it: an import block sorted the way a human reads it rather than the way the formatter sorts it, and a `cond` in `format-typed-value` indented one column short. Formatter output only. No semantic change. AI-assisted-by: mixed models * 📚 Document graph experiment architecture Add Serena memory coverage for the embedded Ladybug graph subsystem.\nDocument projection, incremental sync, console data flow, tests, and operational risks.\n\nAI-assisted-by: gpt-5.6-luna --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Alejandro Alonso <alejandroalonsofernandez@gmail.com> Co-authored-by: Andrey Antukh <niwi@niwi.nz>	2026-08-27 18:54:56 +02:00
Andrey Antukh	0e388442a1	✨ Add storage object status lifecycle and verified dedup (#11345) ... * ♻️ Simplify storage GC delays and add skip-delay task params The touched GC no longer applies an extra deletion-delay when marking storage objects as deleted. By the time a storage object is touched, its referencing domain row has already passed its own deletion delay, and the reference scan is the only safety check needed. Touched objects are now marked with deleted_at = now, so the deleted GC removes them on the next run. For the tempfile bucket, upload chunks now set touched-at in the future (1h, aligned with the upload-session-gc TTL) instead of relying on a special-case deletion delay. Task handlers now read their task props: - storage-gc-touched accepts :skip-delay to process all touched objects immediately, bypassing the min-age threshold. - objects-gc accepts :chunk-size and :skip-delay to process recently deleted rows without waiting for the deletion delay. This allows running the deletion cascade immediately from the REPL via run-task! with the skip-delay option. AI-assisted-by: deepseek-v4-flash * ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking Storage object lifecycle hardening: - Add status column ('valid' | 'pending') as write-ahead marker for object creation. put-object! inserts in 'pending' state, writes blob, then promotes to 'valid'. Failed writes remove the pending row. - Add :storage-pending-gc task to reclaim orphaned pending rows (e.g. after crash between blob write and promotion). - Verify blob existence on every dedup hit via exists-object? (fs stat / s3 headObject). Missing blobs mark the row as deleted and create fresh object. - Add deletion_attempts column (migration 0154) to track physical blob deletion attempts. Restructure gc_deleted to use chunked processing with per-chunk transactions (short lock duration). Failed deletions are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent infinite loops. After 7 attempts, give up and accept orphan. - Change del-objects-in-bulk contract to return #{fail-ids} for precise per-id tracking (fs and s3 backends updated). - Use tmp/tempfile for fs atomic writes with cleanup queue registration (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE POSIX-only assumption. - Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms). - Wrap compensating delete in put-object! catch block to prevent masking original error when connection is aborted. - Fix assert messages in pending_gc.clj and gc_deleted.clj (pool assertion said 'expected valid storage' instead of 'db pool'). - Add pending-objects-excluded-from-gc-deleted test. Use unique path in put-object-write-failure-leaves-no-row test to avoid collisions. AI-assisted-by: qwen3.7-plus * 🐛 Fix review comments on gc-deleted and storage - Fix process-chunk! returning nil causing (+ acc nil) crash - Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent infinite loop when another worker holds locks - Pass :cause to log messages in gc_deleted.clj and s3.clj - Fix extra space in log hint string - Remove unused ::blob-missing? reference from storage memory - Rename test to match actual behavior (leaves pending row) - Add test for gc-deleted giving up after max attempts AI-assisted-by: qwen3.7-plus	2026-08-27 12:37:05 +02:00
Andrey Antukh	87c51090b1	Merge remote-tracking branch 'origin/staging' into develop	2026-08-26 20:14:51 +02:00
Andrey Antukh	6e173a02fb	📚 Split backend testing memory and link from testing skill ... Extract the backend Testing section from backend/core into a dedicated backend/testing memory, following the pattern of common, frontend, and exporter. Update the testing skill and root testing memory to point at the new location, and add exporter/testing to the skill's required reading list. AI-assisted-by: deepseek-v4-flash	2026-08-26 18:11:40 +00:00
Alejandro Alonso	7419bc7007	🐛 Evict multi-scale tile cache on shape edits (#11337) ... those textures across zoom for progressive previews, and invalidate by old∪new document coverage so rotate/move edits do not leave stale fragments on zoom-out.	2026-08-26 08:25:23 +02:00
Alejandro Alonso	57c0e81616	⚡ Present viewport before interest and clamp paint to atlas (#11313) ... Present visible tiles via ViewportReady so zoom settle turns sharp without waiting on the interest ring, and paint at atlas slot size so DPR 2 does not rasterize 1024 only to downscale into 512 slots.	2026-08-25 12:56:07 +02:00
Alejandro Alonso	689d506788	⚡ Render eligible frame drop shadows via direct geometry path ... Add a direct container-geometry path for eligible frames: inline blur when the kernel fits the tile margin, otherwise a cached filter-surface pass reused across tiles via DropShadowFilterCache on both the direct and slow render_shape paths. Move frame shadow logic into shadows.rs. Fix nested/clipped frame shadows by deferring parent clip to composite time, apply negative spread via inset, and allow rotated/transformed frames on the direct path. Skip descendant extrect walks for clipped frames when only nested drop shadows matter, and skip child silhouettes when the container fill already covers shadow descendants.	2026-08-21 11:17:16 +02:00
Alejandro Alonso	2dcf1a8a0a	Merge remote-tracking branch 'origin/staging' into develop	2026-08-20 09:15:03 +02:00