471 Commits

Author	SHA1	Message	Date
alonso.torres	cb26e8a39b	✨ Keep autosave and the workspace alive when saves fail ... - A save that failed on a short platform failure (a lost or refused database connection, lack of resources, a server shutting down, a serialization failure or deadlock, a socket or HTTP timeout, a host that did not resolve, an interruption) answered 500, which stops autosave. It now answers 503, which the frontend retries. Other I/O and database errors keep answering 500, timeouts 504. - A save answered not-found no longer replaces the workspace with the access page. It is a terminal save error: the workspace stays open with the failed save notice, report and reload link. When edit rights are lost, the notice says how many changes could not be saved. - A library sync marked the library synced three seconds later, whatever happened to its save. A rejected sync was undone but stayed marked: copies stayed out of date and the "updates available" notice never came back. The mark now waits for the save, a rollback of the sync brings the notice back, and a failed mark (e.g. the user just became a viewer) is logged instead of leaving the workspace. - While saves fail the workspace is read-only. The view-mode banner says editing is disabled because changes can't be saved, with the Inspect banner's size and bold lead and no Done button. Users can still switch pages, pan, zoom and inspect. The lock lifts when a save lands and brings back the mode the user had or picked meanwhile; the WebGL context lock keeps the workspace read-only if it lasts longer. The lock lives in app.main.data.persistence.lock. - A viewer is never locked. A role change that leaves the user a viewer keeps the edits that lost permission, queued and on screen, and shows the count notice again. The role change that brings edit rights back releases them and rolls them back on screen, as a rejected save would. Any other terminal failure is released by the next role change or file opened. The debug faults `io` (socket timeout, 503) and `eof` (500) cover both sides of the I/O rule.	2026-11-05 23:59:32 +01:00
Andrey Antukh	93d36d04d8	✨ Stabilize fragmented file data across snapshots and GC (#12420) ... * ✨ Stabilize fragmented file data across snapshots and GC Snapshots, automatic and user, now store pointer-based rows that share fragments with the live file instead of duplicating page content. `get-file-snapshot` resolves those pointers on the server so version previews keep rendering plain data, and `restore!` reuses the shared rows. Export still materializes (the package carries no fragments) and import regenerates them. `file-gc` treats the main file and every unpurged snapshot as liveness roots, reads the file unrealized, and reattaches the original pointer wherever the cleaned value is unchanged, so a pass over an untouched file writes no fragments and a pass over an edited file keeps the ids of the pages it did not touch. Snapshot creation and the restore backup take the file row `FOR SHARE`, which serializes them with the pass and closes the race that let a new snapshot reference fragments the pass was about to mark. All backend pointer bindings go through the `fdata/with-pointers`, `with-tracked`, `with-tracked-pointers` and `with-untracked` macros. The loader reads fragments a previous pass marked until `objects-gc` purges them, and the frontend fails loudly with retry instead of dropping an unresolved fragment. Review fixes in this branch: the automatic snapshot attaches before pending migrations, `realize-pointers` is the single verb, `reattach-unchanged` takes the identity fast path, the restore guard takes its revn from the backup read, and `PointerMap.withMeta` generates a fresh id when metadata changes. Tests cover both snapshot writers (red without the lock), snapshot sharing and pin release through the real delete and purge paths, reattach against orphan marking, preview after offload, the pointer binding contracts and the with-meta id change. Closes #12410 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: deepseek-v4.1-flash ♻️ Pin no-churn behavior for untouched pages Pin the behavior the churn verification measured: a save keeps the fragment id of every page it does not touch, with a page of eleven keys (PersistentHashMap), a 46-shape objects map and a nine-page index (hash map). Also pin the mixed case: a data-level objects-map flip rewrites the old pointer pages once and the next save keeps the new ids. The memory records the identity rule behind it: re-assoc with the same value returns the same map in array and hash maps alike. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Stop loading untouched pages when enabling objects-map `enable-objects-map` wrapped every page on every save, and reading a page's objects forced its PointerMap to load the fragment. A save of one page in a seven-page file therefore made seven fragment reads. Skip unloaded pointers: their fragment was already wrapped by the encode-file that wrote it, so loading it only to wrap it again is a no-op. The save now loads only the pages it edits. A page with a plain fragment (a data-level feature flip) keeps its plain fragment until it is first loaded or edited; a migration that enables the feature must rewrite the data itself. Cover the change with load-count tests for save and restore, update the flip test to the new contract, and drop the stale restore comment. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Keep page fragment ids through pending migrations Applying a pending migration on read realized the file, ran the migration and persisted it again; the persist wrapped every plain page in a new PointerMap, so every page got a new fragment id even when the migration did not change it. That leaves the fragment cache cold and rewrites every fragment on the first read after a deploy. Capture the original pointers before realizing, reattach the unchanged positions after migrating (same value comparison file-gc already uses), persist the reattached file and return the realized one: the read contract is plain data. Read-only connections skip the persist and the reattach. A migration that adds objects-map skips the reattach so the pages get wrapped. Move the three helpers to fdata so file-gc and the migration path share them, and cover the path with a real 0031 migration test: the unchanged page keeps its id, the changed page gets a new one, the caller gets realized data and a second read does not migrate again. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Realize library pointers with each library's own loader Fragment rows are keyed by file id, and every caller binds the fragment loader of the file it works on. get-resolved-file-libraries returned libraries with their pages still as pointers, so the first read of a library page under fdata/pointer-map asked the consuming file for the fragment and failed with :fragment-not-found. That broke migrations and file validation for every file using a pointer-mapped library. The weak-map loader now realizes each library's pointers with the library's own id. The test fails without the change. AI-assisted-by: claude-opus-5-5 Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore the pointer-map require the repair path still needs The branch dropped `app.util.pointer-map` from the namespace while the auto-repair path (added to develop before this branch) still rebinds `pmap/*tracked*` around validation and repair in `repair-preexisting-damage` and `process-changes-and-validate`. The namespace did not compile, so the whole backend failed to load. Those toggles do not fit the fdata macros: `with-untracked` binds nil and `with-tracked` creates a fresh atom, while the repair path has to turn tracking off and restore the existing atom (and read it before the repair starts). Restore the require and keep the bindings explicit. Also pin the feature-off path with a test: with `fdata/pointer-map` off (the default), a save creates no fragments, file-gc writes nothing and the snapshot preview still returns plain data. Drop a stray `\n` char literal left at the end of one deftest. Update the memories with the exception to the macro rule, the snapshot FOR SHARE locking and the pre-migration automatic snapshot. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Validate debug files with their own pointer loader The `/dbg/actions` file validation passed the file id where `get-resolved-file-libraries` expects the file map, so the library map was seeded with a nil key. It also validated a file read without realizing it, so the first page lookup of a pointer-mapped file raised "load is not supported when *load-fn* is not bind" and the endpoint could not validate exactly the files the feature touches. Read the file realized (validation walks every page and component anyway) and build the library map from the file. The library loader already resolves each library's pointers with its own id, so the realized file and its libraries validate as plain data. Covered by a handler-level test that fails before the change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Bind the snapshot loader through the fdata helper `reduce-snapshots` hand-rolled the same `binding` that `fdata/with-pointers` builds (it also loaded the function's own pointer loader before entering the reduction). Use the macro so the rule holds there too and the namespace no longer needs `app.util.pointer-map`; the loader is identical (same cfg and file-id) and snapshot walking behaves the same. AI-assisted-by: deepseek-v4.1-flash * 📚 Note the pointer-binding exceptions and snapshot ordering Record what the feature review settled: - Two direct `binding`s stay on purpose: the fine-grained `*tracked*` toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off and restore the existing atom) and `binfile.common/encode-file` (fresh tracked set inheriting the current one). Atom-scoped macros would only be worth it if the set grows. - The automatic snapshot is taken before the pending migration, and the pre-migration window is narrow: the `get-file` RPC migrates on read and its ETag hashes the available migrations, so a post-deploy read cannot be a 304 from the pre-deploy state. The old-format snapshot only appears when the save is the first request to touch the file after the deploy, and `get-file-snapshot` previews it as stored. AI-assisted-by: deepseek-v4.1-flash * 🐛 Apply pending migrations when previewing a version Automatic snapshots are attached before the save applies pending file migrations, so a version can store the old format. The preview RPC handed that data to the UI as stored, and the viewport renders it with the current code. Realize the snapshot with the live file id (fragments are keyed by file, not by snapshot), normalize its migrations to a set (the decoder keeps the pgarray as a vector, and `set/difference` would compare indexes) and apply the pending migrations with the pure `fmg/migrate-file` and the live libraries. Nothing is persisted: the stored version keeps its data and restoring it re-migrates on the next read. The response now carries the snapshot's migrations. Use `:features` the way `get-file` does for `fdata/objects-map`: a client that does not declare it gets plain objects. The historical feature set is not checked against the enabled one on purpose: a version may predate a feature change. Cover the preview of a snapshot with a pending migration and the objects realization; both pin that the stored row is untouched. Update the file-data memory with the preview contract and the vector/set trap. AI-assisted-by: deepseek-v4.1-flash * 🐛 Decode snapshot migrations as an ordered set `file-snapshots/decode-snapshot` handed the `file_change.migrations` pgarray as a vector. `set/difference` falls back to index lookups when the first collection is shorter than the second, so the pending set was right only by accident: a stored subset happens to work, while a stored set larger than `available-migrations` (a snapshot taken by a newer build) marks every available migration as pending and re-applies it to already migrated data. Decode it as a `d/ordered-set`, the type the rest of the code and `schema:file` expect. This also covers `restore!` and the snapshot list, and it lets `get-file-snapshot` drop the `(into #{} ...)` normalization it was carrying. Re-wrap the union in `fmg/migrate` so the invariant holds even when a caller hands a plain set. A snapshot with NULL `migrations` predates SQL migration 0140 and keeps the format version of its time. Pin that legacy path: the preview regenerates the applied set from that version, migrates in memory and leaves the stored row untouched. Update the file-data memory with the invariant and the legacy path. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix flaky waiting-gauge assertion in climit test The throwable-while-waiting test asserted the exact list of waiting-gauge publications, but the component sampler republishes the same count when its first tick lands inside the recorded window under load. Assert only the last value, which is the contract the helper documents. AI-assisted-by: muse-spark-1.3 --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>	2026-11-02 15:02:55 +01:00
Andrey Antukh	29373fff9f	✨ Rework the progress of the import and export dialogs (#12192) ... * ✨ Rework the import dialog The import wizard paints a Check/Upload/Import stepper derived from the dialog status, plus a phase title, a description and a progress bar per phase, and its rows follow the new design: folder icon, file name and a size subtitle. While checking, the dialog shows no rows and no spinner, because the phase header already carries the message. The wizard writes the job milestone once, in the header (`jp/current-milestone-text`), since every file of a container reports the same one. A row keeps its name and its size, says "Ready" once its container finished uploading, and shows the error text when it fails; the button that drops a file only appears while the wizard still offers its import step. The `jobs.queued` label now reads "Waiting job slot". The bar measures the whole import: determinate while the containers are uploaded, weighted by their size (`imp/upload-pct`), and animated otherwise, because the stages of a job are not comparable. The phase title counts the files, and `imp/imported-count` counts them done with the job's own counter, so the number moves while a container rebuilds its files one by one. The analysis reads the entries of a version 3 package in one pass: a file shows what it occupies inside its container, its own entries plus the storage objects its media and thumbnails reference, while what the container weighs stays in `:container-size`, the size the upload bar is weighed on. The ready step waits for the end of the analysis (`imp/ready-to-import?`), not for every known file, so Continue never leaves out the files that arrive later. Tests cover the file sizes, the upload percentage, the done counter, the wizard step and the milestone in course. The change also adds `-XX:NativeMemoryTracking=summary` to the backend devenv so `jcmd` shows native memory, and documents the wizard, the promise call styles (`mem:clojure/promesa`) and two CSS pitfalls: a logical `inline-size` wins over an inline physical `width`, and Rumext leaves a plain `:style` map raw, so inline styles leave CLJS as `#js` objects. AI-assisted-by: glm-5.3-flash AI-assisted-by: deepseek-v4.1-flash * ✨ Rework the export dialog Every exported file is its own job, so each row reports its own state: folder icon, name and a size-plus-`.penpot` subtitle; waiting for a worker; running with the milestone of the job and an animated bar; or exported with a tick. A failed row paints red, takes the close icon and shows the hint of the job, or a generic label, through `files-export-modal.export-error`. Success, failure and cancellation replace each other, so a row never reports two outcomes at once. The ids of the messages are canonicalized to string before they match a row, because they cross serialization boundaries. The export result now carries `:size`, so the subtitle says the format until the job produces the artifact and the size once it exists; a transport failure builds a synthetic error, so a nil `ex-data` no longer leaves the row hanging. The header counts the files and says whether the export runs or ended; since every file has its own job, it paints no global bar. Closing changes too: the Close button is gone. While files are in flight the footer offers Cancel export and the X cancels as well, which marks the loading rows cancelled and leaves them in place, so their outcome can be read; once every row reached a terminal state the footer offers Done, which dismisses the dialog. The retry button of the finished design is left out, because the client has no retry mechanism yet. Styles come along: the rows scroll while the header and the footer stay put, and the new phase header, row and bar follow the tokens of the import wizard. The title key becomes `files-export-modal.title`, and the new keys cover the header, the row states and the buttons. AI-assisted-by: muse-spark-1.3 * ♻️ Inline the binfile job adapters into their task handlers The wrappers in app.binfile.jobs only forwarded to the core, so each task handler now composes its own run: frozen cfg, active check, progress bridge and format dispatch. export-cfg freezes the whole export (file-ids, export-type, output) and v3/export-files! takes it as its single argument; ::bfc/ids becomes ::bfc/file-ids. The bridge keeps one job: bfj/run forwards the cfg to its callback, and check-active answers with it to thread the run. AI-assisted-by: muse-spark-1.3 * 🐛 Migrate binfile tests to the single-arg export-files! The refactor made v3/export-files! take a single cfg carrying ::bfc/file-ids and ::bfc/output, but six calls in binfile_test (added by the tokens handling change) still used the old two-arg form with ::bfc/ids. Move them to the new form, as the rest of the file already does. AI-assisted-by: muse-spark-1.3 * 🐛 Fix export artifact expectations to include file size The export flow now reports the artifact size, but the stale expectations dropped it and failed the frontend suite. Include the size in both assertions so they match the emitted message. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Migrate remaining export-files! test callers to the single-arg form rpc_binfile_test and offload_file_data_test still passed the output apart and used ::bfc/ids; move both into the cfg as ::bfc/file-ids/::bfc/output. * 🐛 Hide the downloads hint when no file was exported After Cancel or when every job failed, the export header said "0 files exported · Check your downloads folder."; drop the hint then. * ♻️ Read the object of a container entry once The index of a container asked twice for the object of the same zip entry: once to test it and once as the key of the update. Resolve it once, in the branch that owns it, so the entries of a file never pay for a question that does not apply to them. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * ♻️ Never size an import row with its container's weight A row of a version 3 package took the size of the container whenever the container held no entry for its file, while the comment beside it promised the opposite. An unknown size now stays nil, which the row already understands: it paints no subtitle instead of a number that belongs to the whole package. The test carries a file the manifest lists and the container does not hold, and the memory records what the row does then. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * 🐛 Retry pool stats read until snapshot is consistent Hikari exposes each pool counter through a separate MXBean call, so a borrow or return between the reads can leave one snapshot off by one. The test read once and failed in CI with total 2 vs active + idle 3. Read up to 20 times with a 10ms pause until total equals active + idle, then assert on the stable snapshot. The invariant stays strict; only the transient race is tolerated. AI-assisted-by: muse-spark-1.3-contributor --------- Co-authored-by: Elena Torro <elenatorro@gmail.com>	2026-11-02 12:12:39 +01:00
𝐧𝐞𝐫𝐬	b0c1d9acda	🐛 Fix the Cache-Control max-age of redirected assets (#12758)	2026-11-01 19:26:14 +01:00
Dr. Dominik Jain	c0f0b14540	🎉 Add direct MCP feedback channel via dedicated tool (#12551) ... * ♻️ Move devenv-only tools to a separate folder * 🎉 Add an MCP feedback channel Add a user-approved feedback tool and an MCP-token-authenticated backend endpoint that queues plain-text email. Enable the tool through the configured backend URI in both local and multi-user mode. Validate subject and feedback limits, enforce per-profile rate limits, and document configuration. Cover authentication, queueing, tool availability, and failure handling with backend and MCP tests. Closes #12490 AI-assisted-by: gpt-6 * ♻️ Move MCP feedback submission into RPC Use the standard RPC wrappers for feedback and remove the dedicated HTTP endpoint, route wiring, and custom rate limiter. Restrict MCP tokens to opted-in routes and commands. Add a shared RPC call helper to the MCP backend client and adapt the existing tests. AI-assisted-by: gpt-6 * 🐛 Harden MCP feedback auth, validation and messages Accept MCP tokens when either the mcp or the access-tokens flag is on, so instances that enable MCP without personal access tokens can send feedback. Personal tokens still need access-tokens. Keep the issuer in one constant, decode tokens without raising to reject unknown issuers, and name the issuer and type checks. Reject control characters in the subject, and in the feedback all but tabs and line breaks, in both the command and the tool, so a NUL answers 400 instead of failing when the mail job is stored. Answer 403 with profile-blocked or profile-not-active for unusable profiles, record only who sent feedback in the audit event, and add doc metadata. Replace on OAuth approval an MCP token issued before MCP tokens had their own issuer, or one this instance cannot decrypt, since the backend rejects both. In the MCP server, mention disabled access tokens on 401, drop the unused 503 case, and say that delivery happens in the background. Document the flags in the README. AI-assisted-by: claude-opus-5-5 * 📚 Document the MCP feedback tool and MCP token rules List submit_feedback among the MCP tools and add an MCP feedback section to the configuration guide covering PENPOT_MCP_BACKEND_URI, PENPOT_MCP_FEEDBACK_EMAIL, the flags and the email settings. Add commented examples of both variables to the compose file. In the agent memories, record when OAuth approval replaces an MCP token, how the MCP server reaches the backend, and which flags, routes and commands accept MCP tokens. AI-assisted-by: claude-opus-5-5	2026-10-29 10:35:46 +01:00
Dr. Dominik Jain	e20a0d748e	✨ Add OAuth support for MCP server (#12480) ... * 🎉 Add OAuth support for remote MCP connections Add MCP-specific discovery, client registration, and approval endpoints. Exchange single-use authorization codes with PKCE for the fixed MCP user token, enabling MCP and creating the token on approval when needed. Use self-contained client IDs without new database tables or stored client records. Keep temporary authorization state in Redis/Valkey. Accept bearer credentials while preserving query-parameter connections. Enable OAuth through PENPOT_MCP_AUTH_URI in multi-user mode. Document the environment variable in the README and implementation details in Serena. Explain protocol and concurrency decisions in the Clojure code. Cover token provisioning, OAuth state, client registration, HTTP flows, and the approval page with tests. AI-assisted-by: gpt-6 * 🐛 Wait for devenv initialization before starting builds Docker reports a container as Running as soon as its main process starts, while the entrypoint may still be preparing the development environment. manage.sh used that status to launch tmux jobs, allowing frontend builds to race with ownership changes to /opt/rustup and Cargo setup. On a fresh container, rustup could fail with Permission denied before the frontend watcher started. Nginx would later serve the existing JavaScript, leaving source changes absent from the running app despite restarting devenv. Wait for nginx's local health endpoint before launching the jobs. Nginx starts after entrypoint setup, so its response confirms that setup has finished. Allow up to 120 seconds and report a timeout without starting the jobs if initialization does not complete. Validation: shell syntax and whitespace checks pass. AI-assisted-by: gpt-6 * ♻️ Share the login modal with MCP authorization Extract the viewer login modal and its styles into a shared auth component, retaining the viewer wrapper and its close behavior. Use the modal on the MCP approval page and preserve the pending request through login. Closing it clears the saved return URL and shows a cancellation message without further actions. Cover return URL handling and cancellation with frontend tests. AI-assisted-by: gpt-6 * ✨ Show client icons in MCP authorization Preserve HTTPS logo URLs from client registration and published metadata and show the icon beside the application name on the approval page. Ignore unusable URLs and hide failed images without blocking approval. Update the access description to refer to MCP connections established from Penpot. Cover logo metadata and approval responses in backend tests. AI-assisted-by: gpt-6 * 🐛 Follow the system theme when no preference is configured Default to the system theme in the shared resolver used by the app and plugins. Preserve explicit light and dark choices, including the legacy default value that means dark. Reuse the reactive theme hook in progress widgets and show the system preference in settings for accounts without a configured theme. Verify preference resolution with regression tests and browser checks. All 1,474 frontend tests pass. AI-assisted-by: gpt-6 * 🐛 Replace expired MCP tokens during OAuth approval Settings can give an MCP token an expiry date. OAuth approval reused the newest MCP token without checking it, so an expired token reached the client and never matched a plugin connection. Approval now treats an expired token as missing and creates a new one. Token exchange rejects a code whose token expired after approval with invalid_grant. AI-assisted-by: claude-opus-5-5 * 🐛 Log unexpected errors on MCP OAuth endpoints The OAuth routes sit outside the shared HTTP middleware, so the shared error handler never saw them, and their own handler turned unexpected failures into a 503 without logging anything. Failures in the session middleware of the approval endpoints had no handler at all. Unexpected failures now also go through the shared error handler for its logging, while clients still get an OAuth error body. The OAuth error handler also wraps the session middleware of the approval endpoints. AI-assisted-by: claude-opus-5-5 * 🔧 Add devenv script to run the MCP server with OAuth start-mcp-devenv-oauth starts the MCP server in multi-user mode with PENPOT_MCP_AUTH_URI derived from PENPOT_PUBLIC_URI, to test OAuth in the devenv. Agentic tooling still uses the single-user start-mcp-devenv. AI-assisted-by: claude-opus-5-5 * 🐛 Fix failing test case --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>	2026-10-28 18:40:04 +01:00
Andrey Antukh	b633093336	Merge remote-tracking branch 'origin/staging' into develop	2026-10-17 16:28:59 +02:00
Andrey Antukh	70b317a976	🐛 Harden upload path handling and validation error bodies (#12067) ... * 🚑 Reject forged upload paths outside multipart Transit bodies can smuggle upload maps whose path points at any server file; the schema only checks the shape and handlers open the path directly. app.rpc now injects a server-side multipart flag that bodies cannot forge, and methods declare their upload keys with the new multipart-only-params metadata, enforced centrally after schema validation with a params-validation error. Covers all nine upload methods, including chunked and management flows. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Sanitize raw explain in generic validation error bodies The generic :validation branch returned ex-data untouched, so errors carrying the raw malli explain (live schema objects, e.g. storage invalid-storage-metadata) broke JSON encoding mid-stream and clients got truncated bodies. Apply the same clean treatment the known codes already use: drop the raw explain and attach its humanized text. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Add e2e path traversal tests for binfile and media upload Add two e2e suites that hand-write transit bodies with a forged ["~#path", ...] upload path (no transit library needed) and expect 400 validation/params-validation. They cover import-binfile :file, upload-file-media-object :content and upload-chunk :content, plus a plain-JSON string-path control case. Before the multipart-path fix they failed (500 io-exception, invalid-image, 200 stored chunk); after it they pass. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Use multipart helper in binfile SVG test setup The SVG import setup uploaded through plain command!, so the new multipart-only guard rejected it with params-validation. Switch the setup to multipart-command! like the other upload tests. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Use multipart helper in remaining upload tests The multipart-only upload guard rejects plain RPC calls carrying upload maps. Five tests with real files still used command! and failed with params-validation. Switch them to multipart-command! while keeping the forged-content tests on plain command!. AI-assisted-by: muse-spark-1.3-contributor	2026-10-17 16:01:04 +02:00
Andrey Antukh	15606a3f85	🐛 Answer 400 on a request body the parser cannot read ... A body declared as JSON that clojure.data.json cannot read ended as a 500 :unexpected with the whole stack trace in the log: wrap-parse-request classified the failure by exception class, and the library signals its syntax errors with a bare java.lang.Exception. Classify it where it happens instead: read-body-params reads the body and raises :validation :malformed-json (400) for anything it cannot parse, keeps RequestTooBigException at 413 and re-raises a structured error from deeper untouched. The body is read outside the handler try, so the error goes out as a throw like the other specific-exception branches. The failure is now one log line (path, peer address, content-type, parser message) without the stack trace, and it is counted: the middleware takes the metrics instance as a middleware argument, so the backend exports penpot_http_malformed_requests_total and nexus declares its own nexus_http_malformed_requests_total. Tests: parse-request-malformed-json-body reproduces the production body and asserts the 400 and the counter; the existing parse-request tests now pass a metrics stub. AI-assisted-by: deepseek-v4.1-flash	2026-10-17 15:54:52 +02:00
Andrey Antukh	0c0336da42	Merge remote-tracking branch 'origin/staging' into develop	2026-10-13 21:04:58 +02:00
Andrey Antukh	eafa483a18	Merge remote-tracking branch 'origin/main' into staging	2026-10-13 14:56:45 +02:00
Andrey Antukh	dd6cc835ef	🎉 Add graceful instance drain via /readyz ... Add a per-process availability flag (app.health) that /readyz reports as 503 OFFLINE while offline, so the load balancer drains the instance and in-flight requests finish untouched. Operator surface is app.srepl.main/set-offline and app.srepl.main/set-online; a restart comes back online. Covered by backend-tests.health-test; HA docs updated. AI-assisted-by: muse-spark-1.3-contributor	2026-10-13 14:33:32 +02:00
Andrey Antukh	9abcfebf60	✨ Run binfile import and export as durable jobs (#12056) ... * ✨ Run binfile import and export as durable jobs Import and export of `.penpot` files become durable jobs: two commands create them and answer at once with a job id the caller follows, while a worker of the `binfile` queue does the work. Each job keeps its own ledger (owner, params frozen at creation, result, progress history and an expiry), and the legacy commands and their SSE keep working. Why: both operations ran inside the request. A large import held it open for minutes, progress only existed while the caller kept the stream open, and a client that went away left work nobody recorded, retried or could cancel. The unified `job` substrate already dispatched, leased, retried and logged events, but it had no user-facing ledger. How: - The substrate gains the pieces a user-facing job needs: optional job-def metadata (family, resource-role), an expiry on submit, the owner in the runner context, and job resources in their own bucket, written with the profile of the job and never deduplicated. - One generic command per family: the caller names the job and sends its business params, the registry checks the family, the job-def decodes and validates, and permission and quota are checked first. - Progress taps of the core become milestones of the job: a shared stage vocabulary with self-contained counters, throttled heartbeats, and an interrupt that stops the run inside its own transaction on cancel. - Cancel reaches pending and running jobs from both modals, and closing a modal mid-run cancels the job it was following. - The export re-checks the read permission at run time and answers with the package descriptor; the import re-checks the edition permission, runs the core in a transaction so a cancellation rolls it back, and releases the consumed package and its temporary copy. AI-assisted-by: deepseek-v4.1-flash * ♻️ Handle externally-cancelled binfile jobs end to end The `cancelled` outcome of a job nobody followed no longer hangs its screens: the data layers emit an explicit terminal message for it. - The export emits `{:cancelled true}` per file and the modal paints a neutral cancelled row instead of success; a message without an artifact never reaches the download branch. - The import closes every entry with a dedicated cancelled message through the existing error rendering, so the wizard completes. - Both flows are pinned by data-layer tests, including the red state they replace. Also in this change: the progress-contract comment of the management API, early business-params validation in create-import-job (before assembling the upload), a `release-resource` helper without fabricated job ids, a note on the heartbeat write cost, and a memory cleanup. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Pre-translate the cancelled import message The cancelled entry of the import wizard carried its message as a key for the UI to translate, which needed a rehash marker comment. Like its neighboring errors, it now translates eagerly in the data layer: the literal call keeps rehash clean with no marker, and the tests assert through the same call instead of the raw key. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Move cancelled strings under generic labels keys The dashboard-specific cancelled key never shipped: before use it moves under `labels.*` next to the generic strings both flows already use, with Spanish translations following the established wording. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add minor improvements * ♻️ Rename export job command to create-export-binfile-job Export now has one command per job type, the shape the :export-assets command needs: create-export-binfile-job freezes exactly the :export-binfile job, so :name no longer travels in the body and only :params does. Import keeps the generic envelope. submit-job reads the queue from the job-def metadata ::jobs/queue-name (falling back to :binfile), read by the command and never by the substrate, so a job-def can route its work to another queue without a change in the command. The rename reaches the frontend and the tests: the binfile export is asked for under its own name, and the exporter service keeps create-export-job. AI-assisted-by: deepseek-v4.1-flash * ✨ Polish the import and export status rows The status row of a file already carries what the file is doing, so the generic "Uploading file…"/"Downloading file…" lines are gone: the import wizard drops its own (the export one went with the jobs refactor), along with the now-unused style. In the export, every file is queued up front, even the ones whose job is created only after the previous one is over, and a file stays queued until its first milestone: a worker picking the job up no longer ends the queue. The queued text is a step smaller than the milestone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Show the cancelled export and fix the import error The cancelled export was unreachable: mark-file-cancelled sets :loading false and the label lived inside the loading branch, so a cancelled file only painted a neutral notification. The terminal state now carries the label, and the export flags lose the `?` suffix (predicates keep it; a data prop is not a question). The import wizard rendered (:error entry) through tr, but every producer writes user-facing text (a hint or a translated label), never a key: it renders the text as-is now, and the dynamic-key ignores and the stale comment are gone. The total-text-limit comment no longer references the 200MiB it replaced; the value is 500MiB in develop already. AI-assisted-by: deepseek-v4.1-flash * ♻️ Reflect the real job params in the creation commands The creation commands declared their params as an open empty map, so the RPC contract and the generated docs said nothing about them. Each command now imports the schema from the job namespace (`export-binfile/schema:params`, `import-binfile/schema:create-params`), and the RPC validates the business params before the command runs. The job-def still owns the params and re-validates the assembled ones at run time. The commands no longer decode the params again: the RPC did it. Import joins export as one command per job type: `create-import-job` becomes `create-import-binfile-job`, its body no longer carries `:name` and the job-def is resolved by name like export's. Its creation schema keeps the version optional, because the command reads it from the package header when the caller does not say it, and fills the manifest metadata kept for audit. A version out of range is now refused by the schema (`:params-validation`) instead of the command's own `:unsupported-version` check, which is gone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Stop the test system from running job runners The test system is main/system-config plus main/worker-config with a few components dissoc'ed. The runners of default, webhook and cron were listed by hand; the binfile runner added with the durable binfile jobs was not, so a real penpot/job-runner/binfile/0 thread ran during the whole suite. That runner races the tests that drive the loop by hand (run-batch plus run-worker-loop): it takes the job the test just dispatched and completes it, and the test then reads a state it did not produce. That is the reported flake of binfile-jobs-integration-test, whose middle assertion waits for a runner of another queue to not take the job. It only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant baked into main/worker-config differ from the one the test builds, so the two used different queue keys locally. Drop every [<profile> :app.worker/runner] by key shape instead of listing the known queues, so a new queue cannot bring the race back. AI-assisted-by: deepseek-v4.1-flash * ✨ Upload the package one chunk at a time and as a percentage The upload milestone counts chunks, not units of work, so the row wrote "Uploading the package 3/9" over a file the user had uploaded once: it read as if nine packages were on their way. Write the stage as a percentage of the file instead ("Uploading file… (33%)"), and make the English text of the stage say file, not package. The chunks also go one at a time now (concat-all instead of merge-all with two in flight), so the percentage moves in a straight line instead of jumping. upload-blob-chunked is the generic uploader, so media and font uploads become sequential too. AI-assisted-by: deepseek-v4.1-flash * 🌐 Complete the Spanish catalogue es.po was 25 keys short of en.po: the 15 that the durable binfile jobs introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps that predate them (errors.connection-error, errors.save-retrying, the find shortcuts, workspace.header.retrying, workspace.tokens.stroke-width and the mcp session labels). sync -l es then sorts the entries, syncs the #: comments from en and drops the 7 dashboard.import.progress.* keys en no longer has. The msgid sets of en and es are now identical (0 missing, 0 extra). AI-assisted-by: deepseek-v4.1-flash * ♻️ Move the job status keys into the jobs namespace The three status words the durable import and export rows render were filed under labels.*, next to the generic labels bucket, while their progress stages live under jobs.progress.stage.*: the strings of one feature sat in two places. Rename them so the whole job vocabulary is one namespace: labels.queued -> jobs.queued labels.export-cancelled -> jobs.export-cancelled labels.import-cancelled -> jobs.import-cancelled No other locale has these keys yet, so the rename costs no translation. The convention (a key names the feature that renders it; labels.* holds the generic reusable words) is written down in the frontend/translations memory. AI-assisted-by: deepseek-v4.1-flash * 🔥 Remove the translation keys nothing references Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...) call site left in frontend/src or common/src, and no literal reference either: labels.uploading-file a duplicate of jobs.progress.stage.upload since the upload stage says "Uploading file…" labels.downloading-file dead since the durable jobs replaced the old import and export progress rows inspect.attributes.stroke.alignment.center inspect.attributes.stroke.alignment.inner inspect.attributes.stroke.alignment.outer workspace.toolbar.mcp-connect-here The first two are leftovers of the durable binfile jobs, the other four predate them. Both catalogues keep an identical msgid set afterwards. AI-assisted-by: deepseek-v4.1-flash	2026-10-07 11:52:48 +02:00
Andrey Antukh	d3ca9a0b83	✨ Add unified jobs substrate replacing legacy one-shot task system (#11542) ... * 🎉 Move backend jobs to the unified job table substrate Replace the generic 'task' table with a unified 'job' table as the durable substrate for all backend jobs, and migrate every producer and consumer to it. The legacy 'task' table stops receiving writes but stays dormant (still cleaned by tasks-gc); upload-session chunking and the other post-branch upstream changes are preserved. New 0154 migration creates the 'job' table: dispatch and lifecycle columns (name, queue, priority, scheduled_at, retries, status, timestamps, props) plus optional user-facing ledger columns (profile_id, target, progress, error, result, resource_id, expires_at). No modified_at trigger: the application updates it on every write for lease and orphan detection. Partial indexes cover the dispatcher claim query, orphan scanning, and per-profile listing. New app.jobs public API: submit!/invoke!, heartbeats and progress reporting, a precompiled job-def registry (name, schema, handler, decoder, validator), an ephemeral request/response mode, and a management API for external workers. The dispatcher claims new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis payload and marks lease-expired running rows as orphaned; the runner executes per-queue with the same retry conventions. New jobs GC (daily): deletes expired rows and retains internal terminal rows per :jobs-retention, touching referenced storage objects so storage-gc-touched reclaims them. Job resources live in a dedicated job-resource bucket with a reclaim branch in storage-gc-touched. All production call-sites (sendmail, delete-object, demo-purge, file-gc, offload, webhooks, quotes, nitrate bulk delete, snapshots, telemetry) and the ten cron tasks now submit through app.jobs; cron is a pure scheduler and the legacy worker registry, task-table inserts, and per-namespace handlers are removed. Includes migrations, dispatcher, runner, jobs, request/response, jobs GC, management API, and cron test suites. AI-assisted-by: glm-5.3-flash AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: kimi-k3 AI-assisted-by: mimo-v2.5 * ✨ Address review comments on jobs substrate Resolves the four open review threads on the jobs PR. Orphaned jobs are now marked aborted, a system-side terminal state with no retry: the dispatcher sweep stores the end event with outcome aborted in a single bulk insert in the same transaction, records the terminal outcome next to the orphaned counter, and reports an error log; orphans keep alerting through the log and the counter, with no user notification. The aborted state is terminal everywhere: the status CHECK, the sweep, the lifecycle writers, the jobs GC retention, the backlog gauges and the user status mapping, with tests covering each transition. The claim comment, the 2.20 doc version and the legacy task table notes are corrected, and objects-gc heartbeats per chunk. AI-assisted-by: muse-spark-1.3-contributor	2026-10-06 07:33:12 +02:00
Andrey Antukh	b2c3fd872a	♻️ Scope organization notifications to specific WebSocket topics (#11460) ... * ♻️ Scope organization notifications to specific WebSocket topics Publish team/org notifications to team-id and organization-id topics instead of broadcasting to all connections via uuid/zero. Dashboard and workspace now subscribe to their current team and organization on initialization, receiving only relevant events. Backend: added subscribe-organization/unsubscribe-organization WebSocket handlers and updated :close to clean up organization subscriptions. Modified notify-team-change, notify-organization-deletion, and notify-organization-change-sso to publish to specific topics. Frontend: dashboard and workspace now subscribe to team-id and organization-id (when applicable) on initialization, with nil-guard in topic filters. Closes #11455 AI-assisted-by: longcat-2.0 * 🔧 Remove unused session-id binding in unsubscribe-organization Clj-kondo lint fix. AI-assisted-by: longcat-2.0 * 🐛 Fix permission check on team ws connection --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>	2026-10-01 13:16:12 +02:00
Andrey Antukh	e6983a1e4e	✨ Enforce idle and absolute session expiration (#11654) ... * ✨ Enforce idle and absolute session expiration Sessions now expire on two server-side conditions: an idle window (PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE, default 7d) and an absolute cap from creation (PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE_ABSOLUTE, default 30d, enforced by the token :exp claim). The daily session-gc task deletes rows that exceed either window, so idle sessions can no longer be replayed and active sessions are not deleted at the idle window. Also remove the legacy v1 HTTP sessions: the http_session table and the string-id / :ver 0 token code paths are gone. Any v1 cookie now requires a fresh login. Document the session expiration configuration in the technical guide and add a backend memory describing the token, renewal and GC model. Closes #11646 AI-assisted-by: deepseek-v4.1-flash * 🐛 Address session-expiration review findings F1-F4 Fix the unreadable test (a stray paren broke whole-suite discovery), enforce idle expiration on every request in wrap-authz, fail boot fast when the absolute cap sits below the idle window, and align config defaults with the memory rule while fixing its migration number and stale reference. Closes #11646 AI-assisted-by: muse-spark-1.3-contributor	2026-10-01 10:12:05 +02:00
Andrey Antukh	acd146f6f4	🐛 Restore rate-limit headers and add Retry-After on 429 (#11895) ... * 🐛 Restore rate-limit headers and add Retry-After The account-lockout change replaced the header-forwarding 429 handler with a body-only one, so existing RPC rate-limit responses lost their x-rate-limit-remaining and x-rate-limit-reset headers. Account lockout never sent Retry-After either. Make handle-error :rate-limit preserve ::http/headers and add a retry-after header when the exception carries a non-nil :ttl in seconds, keeping the current JSON body. Add focused tests for both the lockout and the RPC limiter paths. Document activation, defaults, password/LDAP scope, Redis fail-open behavior, and the lockout risk, and record the final HTTP contract in the backend auth memory. Refs #11397 AI-assisted-by: deepseek-v4.1-flash * 🐛 Add Retry-After to RPC 429 and expose headers in CORS Address review follow-ups on the account-lockout 429 contract: - The RPC limiter now sets retry-after in its 429 headers (seconds until the longest rejecting limit resets), so it matches the account-lockout response and the HTTP standard. - CORS exposes retry-after, x-rate-limit-remaining, and x-rate-limit-reset so browser clients can read them. - Use backticks for Retry-After and account-locked in the docs for consistency with nearby sections. Refs #11397 AI-assisted-by: deepseek-v4.1-flash	2026-09-24 12:19:29 +02:00
Andrey Antukh	452f38cf5d	✨ Add Prometheus metrics for storage operations (#11700) ... * ✨ Add storage operation metrics for S3 and buckets Expose Prometheus metrics for the object storage subsystem. The S3 backend now attaches an AWS SDK MetricPublisher that counts API calls, retries and latency per operation and target. The storage layer counts logical operations and deduplication outcomes per Penpot bucket, and the assets handlers count served requests per route. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Fix storage metrics labels, errors and test gaps Address the review findings on the storage metrics commit. Label reads with the object's own backend, count failed asset serving as errors without swallowing them, and cover the failed S3 call, S3 asset path and permission-denied branches with tests. Also share the label helper and reuse the metrics test helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics and fill test gaps Address the second-round review findings on storage metrics. Unknown backends now fail explicitly and count as errors, exists stays paired with its dedup outcome, and the thumbnail, missing storage, expired reads, unknown buckets and write failure paths are covered by tests. Label coercion goes through the shared metrics helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics accuracy and coverage Address the full-branch review findings on storage metrics. Touch and delete emit only on changed rows, reads emit after the backend fetch, unknown backends fail explicitly, and tempfile mismatches count as unauthorized. Publisher nil policy, pairing rules and attempt semantics are documented and covered by tests. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Address full-branch review findings on storage metrics Touch and delete resolve labels from the row, reads stay paired, failures are covered by tests, and logging, ranges and docs are tightened. Includes the label helper unit tests and the retries wording clarification. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ⚡ Label touch and del metrics from UPDATE RETURNING The storage metrics change resolved metric labels for touch-object! and del-object! with an extra SELECT per id-based call. Since app.main instruments storage unconditionally, every GC collector and binfile import paid that extra round trip: deleting a team with 10k media objects doubled the storage_object statements exactly on the paths that already process the most rows. touch-object! and del-object! now take only the object id (UUID) and read the labels from the updated row itself via RETURNING id, backend, metadata: one statement, no pre-read, and labels that always match the row actually mutated. del-object! additionally guards on deleted_at IS NULL, so a repeated delete returns false and emits no metric. Also from the review of the full branch: extract the duplicated serve/emit/rethrow block in app.http.assets into one helper; give penpot_storage_s3_timing explicit histogram buckets up to 60s (the default cap at 7.5s hid the slow S3 calls the metric exists for); drop the unused ::target-id config key from the S3 backend and hardcode the :default target label until per-bucket routing lands. AI-assisted-by: glm-5.3-flash * ✨ Harden storage metric recording and definitions The metric definition schema is now closed and declares every key the collectors read: buckets, quantiles, max-age and reg. A typo such as a misspelled ::mdef/buckets used to compile and silently fall back to the default histogram buckets; it now fails at startup. The asset result-label fallback coerced an absent status to 500, so a future serve path without a status would have counted successes as errors. The mapping is now explicit and documented: served below 400, unauthorized for 401/403, not-found for 404, and error for everything else, including an absent status. The never-fail try/catch around metric recording existed four times with drift. One app.metrics/run-safe! helper replaces them: it no-ops on a nil metrics instance and logs the first failure per hint at warn level, then at debug, so a broken setup surfaces once without flooding the log. The S3 publisher keeps its outer try/catch: it is the SDK MetricPublisher contract boundary. AI-assisted-by: glm-5.3-flash * ✨ Make metrics mandatory and run! safe by default Recording a metric must never change the behavior of the operation being measured, so `run!` now catches recording failures itself: the first failure per metric id logs at warn, later ones at debug. This replaces the `run-safe!` helper, whose four copies had drifted, and applies the guarantee to every emit site instead of only storage. The metrics instance precondition is a plain assert, and the collector lookup stays outside the recording guard, so a missing instance fails hard even when asserts are disabled. Metrics is therefore no longer optional: the storage, s3-backend and db-pool schemas require `::mtx/metrics`, and the assets handler cfg always carries it. `wrap-publisher` no longer returns nil for a nil instance, and the db pool wires the prometheus tracker unconditionally. AI-assisted-by: deepseek-v4.1-flash	2026-09-23 15:25:21 +02:00
Andrey Antukh	1c7a73ec16	✨ Add account lockout after failed login attempts (#11402) ... * ✨ Add account lockout after failed login attempts Implement per-account brute-force protection using a Redis-backed failed-login counter. After 5 failed attempts within 15 minutes, the account is temporarily locked out and all login attempts (including with the correct password) are rejected with a 429 response. Closes #11397 AI-assisted-by: longcat-2.0 * 🐛 Bind LDAP session to directory-verified profile The account-lockout change added a shortcut that preferred the profile matching the typed email over the one returned by the LDAP directory. These can differ with aliases, UPNs, or multi-valued mail attributes, letting a user with valid LDAP credentials bind a session to another Penpot account. Keep the typed-email profile only for lockout checks. After LDAP succeeds, resolve the session profile from the directory identity as before and clear failed attempts on that profile. AI-assisted-by: deepseek-v4.1-flash	2026-09-23 12:31:50 +02:00
María Valderrama	3a5481c577	✨ Error page for instances without admin-console configuration (#11812) ... * ✨ Error page for instances without admin-console configuration (#11649) * ✨ Error page for instances without admin-console configuration * 📎 Code review * 🐛 Fix link nitrate-not-configured	2026-09-23 10:36:40 +02:00
Andrey Antukh	38212e757c	🐛 Bound decompressed size of JSON entries on binfile v3 import (#11607) ... * 🐛 Bound decompressed size of JSON entries on binfile v3 import Every JSON/text zip entry (manifest, files, pages, shapes, colors, components, typographies, tokens, plugin-data) was decompressed without any size limit, letting a small .penpot archive exhaust the backend heap (GHSA-qcw7-v626-g6cf). Only binary storage blobs were guarded. Reuse the existing size-limiting-stream guard on the text path: 20 MiB cap per entry, 200 MiB cumulative budget per import job, plus a cheap declared-size pre-check. Both limits are configurable and wired through the binfile, management and debug entry points. Adds zip-bomb regression tests for the file entry, the synchronous manifest read and the cumulative budget. Closes #11606 AI-assisted-by: muse-spark-1.3-contributor * ♻️ Uniform binfile import limits behind init-limits Move the binfile import limits to a single source of truth in app.binfile.common (default-* vars) and drop the duplicated entries from config/default; env overrides keep working through the schema. Resolve all limits once per job with init-limits (::max-size, ::total-max, ::current-size, ::max-object-size, ::max-zip-entries) instead of rebuilding the map per zip entry. Thread cfg as the first arg through the v3 readers, collapse read-plain-entry into read-entry, and give size-limiting-stream a single explicit-counter arity. v1 keeps using the compiled default (mechanical rename only). No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename binfile limits to text-entry/binary-entry terms Use text-entry/binary-entry vocabulary consistently across config keys, bfc input keys, default-* vars and the limits resolved by setup-limits (::max-text-entry-size, ::max-text-total-size, ::current-text-size, ::max-binary-entry-size). Rename init-limits to setup-limits. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename cumulative text counter and document binary limit Rename ::current-text-size to ::accumulated-total-text-size for clarity and expand the default-max-binary-entry-size comment to match the other limit vars. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Harden binfile guards and prove budget accumulation Add a regression test that only passes when text bytes accumulate across entries (budget between largest entry and summed total; verified red against a per-entry atom). Include the entry name in streaming-guard errors, count skipped bytes against the budget with a direct unit test, and forward all four limit keys in get-manifest. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Thread cfg through get-manifest get-manifest now takes the caller cfg and resolves limits with setup-limits like the import job itself, instead of building a single-use mini-cfg from cf/get. No behavior change. AI-assisted-by: muse-spark-1.3-contributor	2026-09-22 15:16:59 +02:00
Andrey Antukh	d9e28fb1a4	🐛 Add authorization checks to WebSocket subscriptions (#11069) ... Subscribe-file and subscribe-team handlers now verify the requesting profile has read permissions on the target resource before creating a subscription. Pointer-update handler now validates that the message file-id matches the subscribed file-id before publishing. Closes #11067 AI-assisted-by: mimo-v2.5-pro	2026-09-22 11:16:14 +02:00
Andrey Antukh	69111accfe	📎 Fix copyright owner on several files	2026-09-15 19:22:56 +02:00
elhombretecla	a91d81c695	🎉 Add link preview metadata for shared links	2026-09-15 17:10:54 +02:00
María Valderrama	66876f16b2	✨ Error page for instances without admin-console configuration (#11649) ... * ✨ Error page for instances without admin-console configuration * 📎 Code review	2026-09-15 07:24:55 +02:00
Andrey Antukh	ba608f8c77	Merge remote-tracking branch 'origin/staging' into develop	2026-09-14 14:50:32 +02:00
Andrey Antukh	f4cf6f46f8	🐛 Use constant-time comparison for management API shared key auth (#11534) ... The management API shared-key-auth middleware was using the standard = operator for key comparison, which is vulnerable to timing attacks. The RPC middleware already uses constant-time comparison via MessageDigest/isEqual. This change: - Makes constant-time-eq? public in app.http.middleware - Updates app.http.management/shared-key-auth to use mw/constant-time-eq? - Fixes an inconsistency where the nil-key branch returned a 2-arg function - Adds comprehensive tests for the management shared-key-auth middleware Closes #11426 AI-assisted-by: qwen3.7-plus	2026-09-14 13:26:16 +02:00
Andrey Antukh	37dab75e1a	Merge remote-tracking branch 'origin/staging' into develop	2026-09-10 20:29:44 +02:00
Andrey Antukh	4ce459d720	🐛 Escape LDAP filter values and use directory email in retrieve-user (#11085) ... Fix LDAP injection vulnerability (T5-N1-03) where the client-supplied email was used directly in the LDAP search filter without escaping RFC 4515 special characters (*, (, ), \, NUL), and the profile email was taken from client input instead of the LDAP directory attribute. Changes: - Add escape-ldap-filter-value per RFC 4515 section 3 - Apply escaping in search-user before building LDAP filter - Add get-attr helper for multi-valued LDAP attributes - Fix retrieve-user to use directory email (attrs-email) instead of client email - Use cuerdas.core instead of clojure.string Closes #11084 AI-assisted-by: mimo-v2.5-pro	2026-09-10 16:39:35 +02:00
Andrey Antukh	d6e82f44a6	✨ Enforce absolute maximum session duration (#11447) ... Session tokens now carry an :exp claim anchored to created-at (not modified-at), so activity cannot extend the session beyond the absolute maximum (default 30 days, configurable via PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE_ABSOLUTE). The existing token verification already rejects expired tokens, so enforcement is automatic. Also extends the GC task to purge expired http_session_v2 rows, which were previously never cleaned up. Closes #11444 AI-assisted-by: longcat-2.0	2026-09-09 11:45:28 +02:00
Pablo Alba	d263c23a58	🐛 Add ssrf check for nitrate sso and add timeouts to http client (#11576)	2026-09-09 11:23:53 +02:00
Pablo Alba	fb6ece7a7e	Revert "🐛 Enforce SSRF checks and add timeouts to HTTP client (#11474)" (#11556) ... This reverts commit ff63668c1ef61928878cded13bc1c2734c983aff.	2026-09-08 13:38:00 +02:00
Pablo Alba	ff63668c1e	🐛 Enforce SSRF checks and add timeouts to HTTP client (#11474)	2026-09-07 15:59:35 +02:00
Elena Torró	00e0492bb5	🐛 Fix import libraries missing team-id (#11505)	2026-09-04 13:49:37 +02:00
Andrey Antukh	f5aad7b1ae	Merge remote-tracking branch 'origin/staging' into develop	2026-09-01 12:45:17 +02:00
Andrey Antukh	92c2079ae2	🐛 Add configurable limits for ZIP entry count and object size in v3 import (#11022) ... * 🐛 Add configurable limits for ZIP entry count and object size in v3 import Add binfile-import-max-zip-entries (default 500,000) and binfile-import-max-object-size (default 100 MiB) config entries. Both are configurable via PENPOT_BINFILE_IMPORT_MAX_ZIP_ENTRIES and PENPOT_BINFILE_IMPORT_MAX_OBJECT_SIZE env vars. Entry count is checked before processing begins. Per-object size is checked after each storage object content is resolved. AI-assisted-by: mimo-v2.5-pro * 🐛 Enforce actual decompressed byte limits on v3 import The previous object-size check trusted the ZIP entry header's declared size (ZipEntry.getSize()), which a malicious zip-bomb can forge. The check would pass, then the full decompressed payload would be read anyway during hashing and storage persistence. Add size-limiting-stream, a FilterInputStream wrapper that counts actual bytes read and raises :validation :max-file-size-reached when the configured limit is exceeded. Wire it into zip-entry-storage-content so both the hash calculation and storage write paths are bounded by real decompressed bytes, not declared header size. Also wire import limits into management.clj (clone-template) and debug.clj (import-handler + clone path) for defense-in-depth, and add a test that exercises the object-size limit with a real storage object in the exported ZIP. AI-assisted-by: mimo-v2.5-pro	2026-09-01 08:44:06 +02:00
Elena Torró	66b4a99ac3	🎉 Implement export jobs to process export requests (#11296) ... * ✨ Add export job model, store and scheduler to exporter * ✨ Render wasm exports on pooled worker threads * ✨ Add export job REST API to exporter * ✨ Use export job API and allow cancelling wasm exports * 🔧 Show export jobs in the internal debug panel * 🔧 Pass flags and export job settings to the exporter container * 📚 Document the exporter job API and its redis layout	2026-08-31 14:42:51 +02:00
Andrey Antukh	292390f0eb	Merge remote-tracking branch 'origin/staging' into develop	2026-08-31 11:52:26 +02:00
Marina López	f06d1df5be	🐛 Align management API with payments service (#11414)	2026-08-28 11:54:02 +02:00
Álvaro Tejero-Cantero	38004e6bb2	✨ Add the graph subsystem and graph visualization console to the backend (#11101) ... * 🎉 Basic lbug connection for ingestion * ✨ Add Penpot-to-Ladybug graph ingest vertical slice * ✨ Use embedded Ladybug Java API instead of CLI * ♻️ Share Ladybug connection across ingest and stats * ✨ Validate graph ingest projections with Malli * ✨ Project nested shapes recursively into the graph * ⚡ Load graph ingest via Ladybug COPY bulk import * 🐛 Fix graph COPY ingest for multiline text names * ✨ Add Ladybug graph export to debug UI * ✨ Add debug graph console for in-memory Cypher queries * ✨ Add live file-change feed to debug graph console * ✨ Incrementally sync debug graph from Penpot file changes * ✨ Handle mov-objects in debug graph sync * 🐛 Fix batch delete sync and keep graph console feed alive * ♻️ Derive graph node schema from Malli registry * ✨ Add G6 graph view to debug graph console POC per work/g6/plan.md. New /dbg/actions/graph-data exports the in-memory Ladybug session as plain JSON (per-table node queries + multi-table IsChildOf match, row cap 100k with truncation flag). Console page renders it with AntV G6 v5 (jsDelivr CDN, antv-dagre BT layout, color+glyph per node table, validated palette) and refetches debounced on live :file-change messages. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix list-column CSV ingest and serialize graph session access COPY failed on any file with container shapes: list-typed DDL columns (shapes UUID[], points STRING[], strokes JSON[], ...) were JSON-encoded in staging CSVs, which Ladybug's list parser rejects. Write Kuzu list literals instead, typed per column. Also: value->clj no longer crashes on LIST/STRUCT values (binding lacks value_get_value support; fall back to string), and the debug session Connection is now guarded by a per-session lock — it was shared unsynchronized between the msgbus sync loop and HTTP query/export handlers, and one lost DETACH DELETE was observed under concurrent refetch load. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Split graph console in two columns; add file tree and fullscreen Graph view moves to its own sticky right column (overrides .widget max-width). New /dbg/actions/graph-files endpoint lists teams -> projects -> files for the profile; the console renders it as a collapsible tree where clicking a file loads it. Maximize button fullscreens the graph panel and resizes G6 on fullscreenchange. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ♻️ Replace fullscreen with in-page expand for graph view Fullscreen API took over the whole output and broke window-manager splits (and is denied in some environments). The Expand button now toggles a fixed-position overlay covering the page while keeping browser chrome; Esc restores. Column positioning moved from inline style to the stylesheet so the expanded class can override it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Fold containers as collapsible combos in graph view Non-empty containers (Page, Frame, Group, Boolean, SVGRaw) render as nested G6 rect combos holding their own node plus direct children; Document stays a plain node. Double-click folds/expands (collapse-expand behavior); collapsed combos show a member count and re-route child edges. Fold state is read back from getComboData and re-marked on every refetch, so it survives live redraws. Layout gains sortByCombo to keep same-rank nodes grouped by box. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ⚡ Fix graph view freeze on large files; add fold toggle and root rule Root cause of the tab freeze on ~1700-node files was G6's default entrance animation: measured 1700 nodes at >2 min animated vs 1.5 s with animation: false. Secondary cost was antv-dagre (~7 s at that size); since IsChildOf is a tree, an O(n) tidy layout (depth = rank, post-order leaf slots, parents centered) computed client-side replaces it and renders the same file in ~1.4 s. A guard skips auto-render above 4000 nodes with an explicit Render-anyway button, so opening the console with a huge session loaded stays responsive. Folding is now switchable ('fold containers' checkbox, persisted in localStorage) and generalized: any node with children folds except the IsChildOf root of the loaded graph, so Documents (and later Projects/Teams) fold automatically once they gain a parent node. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add layout dropdown to graph view Adds a layout <select> next to the fold toggle, populated from the LAYOUTS map in the template: 'tree' (the O(n) preset layout, default) plus 13 G6 layouts (antv-dagre, dagre, circular, concentric, radial, grid, force, d3-force, force-atlas2, fruchterman, mds, combo-combined, random), all smoke-tested against combo data on this UMD build. Layout and fold toggle are independent; switching layouts recreates the graph instance (cheap with animation off); both choices persist in localStorage. antv-dagre stays available for when non-tree edges arrive. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add query-result subgraph, ws auto-reconnect, adaptive animation The Cypher result pane now offers 'Show result in graph view': any UUID found in any result cell selects the matching nodes in the cached export and the view renders the induced subgraph (edges kept when both endpoints match); 'Show full graph' resets. No graph reconstruction from the query result is needed. The notifications websocket reconnects automatically (3 s retry) and resubscribes + refetches on reopen, so backend restarts no longer permanently kill the live feed; a lost session now reports 'no graph session (backend restarted?) - reload a file' instead of a bare 404. Animation is size-adaptive: graphs (or filtered subgraphs) up to 100 nodes render animated for didactics, larger ones stay animation-free; crossing the threshold recreates the instance like a layout switch. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph toolbar, animate toggle, filter columns, repaint skip Graph view gains an on-canvas G6 toolbar (auto-fit, expand, restore - the fullscreen icons drive the existing in-page expand), an 'animate' checkbox that disables animation unconditionally when off (persisted, adaptive <=100-node rule applies only when on), and a ResizeObserver on the canvas so the panel follows window/flex resizes without touching the user's viewport. Preset tree positions are now only injected for the built-in tree layout, removing the tree-then-layout flash on animated re-renders under G6 layouts. Refetches skip the repaint when the display projection (nodes, edges, truncated) is byte-identical, so attribute-only change bursts no longer repaint. Console: default query returns s/t name+label over all edges plus filter_src_id/filter_tgt_id columns; filter_* columns are hidden from the results table (client and server render) but still feed the 'Show result in graph view' id harvest, keeping the table legible while the graph filter stays available. The query text persists in localStorage across page reloads (restored only over the default, never over a server-rendered query). Legend shows colored Unicode glyphs matching node shapes instead of squares with textual annotations. Load/Unload buttons share one row (HTML5 form attribute), and the loaded file name links to the Penpot workspace via the legacy /#/workspace/<project-id>/<file-id> route resolved client-side from the files-tree payload. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix runaway graph panel growth and blank canvas; drop Expand button Root cause of 'graph flashes on load then disappears' plus unbounded horizontal growth of the graph panel: fieldsets default to min-inline-size: min-content, so #graph-view-panel sized to its content, and the new ResizeObserver->setSize path closed a feedback loop (setSize -> slightly wider G6 canvas -> wider fieldset -> wider .dashboard flex column -> observer fires) that grew the page ~10px per frame and wiped the painted canvas on every step. Fix severs the feedback path: #graph-view-panel gets min-inline-size: 0, #graph-canvas gets overflow: hidden, and the page section gets flex: 1 1 0 with min-width: 0 so column widths are viewport-driven, never content-driven. This also fixes the original narrow-window scrollbars defect for real. The observer stays (guarded by a current-size comparison) because G6's autoResize is inert on this UMD build (verified: window resizes left the canvas size untouched); the inert autoResize flag is dropped. Legend items now join with spaces so the nowrap spans can wrap between entries. Also removes the header Expand button - the toolbar's expand/exit icons cover it, Esc still restores. Verified against the running devenv with a logged-in profile and variants_simple loaded: graph renders and persists, widths stable over multiple seconds at 1400px and 1000px viewports with no horizontal overflow, canvas follows both window shrink and grow, toolbar expand gives a full-page canvas and Esc restores. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Make the default query self-explanatory; link the Cypher docs The default query is now multi-line with // comments that explain the filter_* column convention in place (Kuzu accepts comments and blank lines mid-statement; verified against an in-memory database through the console query path). The query fieldset is retitled 'LadybugDB Cypher' with the Cypher word linking to https://docs.ladybugdb.com/cypher/. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 📎 Fix linter issues * ✨ Add Component nodes and IsInstanceOf edges * 🐛 Fix memory leak * ✨ Style Component nodes and IsInstanceOf edges in graph console Slice-3 export sends edges with a rel field. Derive tree ranking, combo derivation and fold-ability from IsChildOf only; draw other rels as overlay edges with per-rel styles (EDGE_STYLES: IsInstanceOf violet dashed, matching the new Component diamond in NODE_STYLES). Legend now lists only displayed node tables and rels, re-rendered per redraw; help text trimmed to essentials. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph diff marks with step fade to graph console Each display-changing refetch is a step: added nodes/edges get a green halo, removed ones stay as ghosts with a dashed crimson halo (nodes, fading opacity) or thicker crimson stroke (edges), re-entering layout and combos through their ghost IsChildOf edges. Marks fade linearly and drop after N steps; N is the new "fade" number input (localStorage, 0 = off). Dash + fade carry the added/removed distinction under red-green CVD (#40c057/#c2255c, deutan dE 17.4); diff is vs the previous display step, not arbitrary revisions. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Reserve chroma for changes in graph console diff mode Monochrome entity scheme: all node tables share one slate hue, lightness separates within-glyph siblings (validated, worst pair dE 17.5), SVGRaw becomes the hollow hexagon, both rels go grey with dash as the only separator. Diff marks now own all color: thick green/crimson stroke ring (dashed for removals) plus a larger, subtler halo; the legend gains +/- entries while marks are live. Two additions to guide the eye: a brief DOM-overlay pulse on age-0 elements (independent of the G6 animation gate) and a "fold unchanged" toggle that collapses every combo not on an ancestor path of a changed element. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Expand folded combos that gain changed elements setData merges datum props by id on a live G6 instance, so omitting style.collapsed retained a previous true: with "fold unchanged" on, a change inside a folded combo pulsed but never expanded it. Write the boolean explicitly both ways. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Label edge rels with compact unicode symbols Dash variants alone cannot carry the growing rel roster: EDGE_STYLES entries gain a sym rendered as a small mid-edge label with a white backing (IsInstanceOf = "∈"; IsChildOf stays unlabeled as the background structure), and the legend shows the symbol. Convention from the abacus viewer EDGE_SYM dict. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add node inspector panel to graph console Clicking a node fetches its full attribute row (MATCH (n:`Table` {id: uuid(...)}) RETURN n.*) through the query endpoint and renders non-null attrs into a panel under the canvas (count of empty attrs noted). Panel over tooltip: projected tables carry ~80 columns, and the panel persists for reading without obstructing the graph. Table/id are validated before Cypher interpolation; the listener is re-attached on every instance recreation. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Graph console QoL round "Show result in graph view" moves into an actions bar above the results table; results scroll inside a 45vh container (client and server render paths); the Loaded-session fieldset gains a live "Graph size" line that stays fresh through skipped repaints; IsInstanceOf mid-edge label becomes the spelled-out rel name (∈ read as membership, not derivation) with the legend falling back to the dash-arrow for long syms. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Sync Component library changes into the Ladybug graph * 💄 Polish graph console session panel and edge labels Loaded-session fieldset: graph size gains a resident-memory estimate (fit to graph_sizes.md: ~1.1 MiB floor + ~5.4 KiB/node) with per-table counts on hover, replacing the load-time Projection stats; loaded-at compacts to local HH:MM with the full instant on hover. Edge rel labels drop to 7 px and lose the dashed stroke — the text label alone carries rel identity. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add PNG export and hover tooltips to graph console Toolbar gains an export item: graph.toDataURL({mode: "overall"}) downloads the whole laid-out graph as graph-<revn>.png — page-chrome-free captures, also the fast path for agents debugging the console. A hover tooltip (table, label, id) backs the reduced/absent labels on dense layouts. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Prune graph console layout roster and tune overlap Remove grid, random, force, fruchterman, force-atlas2 (nothing over the kept set) and mds (stress layout degenerates to spokes on tree distances, no collision term to tune). Parameterize the keepers against node overlap — concentric/radial get preventOverlap+nodeSize, d3-force a collide radius — and shrink node labels to 7 px on those layouts (DENSE_LABEL_LAYOUTS), verified against variants_simple (72 nodes). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Highlight clicked node neighborhood in graph console click-select behavior with degree 1: the clicked element keeps a black ring, direct neighbors stay full-strength, everything else dims to 0.2 opacity (inactive state); clicking empty canvas clears. Works on edges too (selects both endpoints) and composes with the node inspector on the same click. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add overview mode: fold containers at or beyond a depth "fold >= depth" number input (root = 0, empty = off, localStorage): every combo whose container sits at that IsChildOf depth or deeper collapses, giving a top-of-file overview (e.g. 2 folds the containers hanging from a Page). Composes with fold-unchanged — depth folds first, changed ancestor paths are then drilled open. Derived fold state overrides manual folds while active. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore fold-containers as the combo master gate Since fold-unchanged and depth folding arrived, withCombos ORed them in, so unchecking "fold containers" could no longer remove the combo boxes. The checkbox is the gate again; the derived fold rules are dormant without it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console UI polish round Merge the load form and files tree into one "Load graph from Penpot" box (tree first, uuid + Load/Unload in a row); Loaded session carries HH:MM in its legend; the Live changes box stays hidden until the first change arrives; query fieldset reads "Query graph (LadybugDB Cypher)" with the link covering both terms. Drop the hover tooltips (distracting, useless zoomed out) and the resident-size estimate (per-table counts stay on hover); every toggle gets a "When set/checked ..." title. Depth fold: 0 now expands every container (no more hunting for max depth). Node inspector: two-column flow, structured or long values folded behind the file-tree disclosure triangle. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Report actual graph memory from the buffer manager graph-data gains bm-bytes (CALL bm_info() -> [mem_limit mem_usage], nil-safe, under the session lock); the session panel shows it as MiB behind the node/edge counts — real resident memory replacing the removed estimate. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console control-bar and session-panel rework Left column narrowed 440->330 px (uuid input flexes). Control bar reordered: layout first, then animate and fade (narrow inputs), then the fold set; "fold containers" renamed "foldable containers" (on = foldable, not folded). Load becomes Reload once a session exists (same operation as the removed Full-reload button — load-session! on the current id; tooltip explains the fallback role) with Unload beside it. Session panel: revisions on one line ("ingested at N · graph now M", hover explains the difference), duplicate uuid after the file name dropped. Tried and rejected: fishbone (no positions on graph data) and compact-box (G6 tree layouts walk parent->child, IsChildOf points child->parent). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Legend entries toggle node-table visibility Clicking a legend entry hides/shows that table across the view (struck-through while hidden, kept listed for re-enabling; pure client-side id filter through filteredGraphData, edges drop with their endpoints, ghosts respect it). Also: setting fold >= depth above 0 now switches foldable containers on — a positive depth was silently inert without combos. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Enable the edge-bundling plugin Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Session breadcrumb, changelog colors, spacing File line becomes team › project › file (clickable) with the resident-memory figure beside it (moved up from the graph-size line; breadcrumb resolves from the files-tree payload, so files outside the profiles teams show plain). add-obj/del-obj in Live changes wear the canvas diff colors. Paragraph margins tightened above Feed; left column 330→350 px. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Guard renders against heavy graphs; add ?safe escape hatch A heavy file could freeze the tab on load-and-render despite the animation gate: the render guard counted nodes only, and the edge-bundling plugin is iteration-heavy in edges. Guard now also trips on edges (8000), edge bundling only activates at <= 300 edges, and /dbg/graph?safe disables auto-render entirely (counts + "Render anyway"), so a page that hung can always be re-entered with the session intact. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🔥 Remove the edge-bundling plugin Bundled edges render unsmooth and ugly on this build; the gating constant goes with it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 One row per operation in the Live changes table Columns revn | op | id: the revn repeats across a batch, the op wears the canvas diff colors (shape/attrs detail on hover), and the id column shows the uuid last group with the full uuid on hover, or N/A for ops without a subject id (e.g. mov-objects). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Use app.system/system in the graph ingest helper develop renamed app.main/system to app.system/system and dropped the app.main require while this branch was away. Rebasing replays the old call, so clj-kondo reports an unresolved namespace and the ns will not load. * ✨ Put the graph subsystem behind a flag, off by default (#11075) `app.graph.ladybug` imports `com.ladybugdb.*` at namespace load. Two namespaces reach the subsystem and both required it at the top level: `app.http.debug`, which registers the `/dbg` routes, and `app.srepl.main`, which loads with the REPL server. Every backend built from this branch therefore linked the Ladybug native library into the JVM at boot, whether or not a graph was ever used. Add a `:graph` flag to `varia`, deliberately absent from `default` so that a released Penpot ships with the subsystem off. Both require sites now resolve `app.graph.*` at call time, so with the flag off no `com.ladybugdb` class is loaded. The nine `/dbg` graph routes are registered only when the flag is on, and 404 otherwise. The `/dbg` admin gate is untouched: the flag decides which routes exist, not who may reach them. When the flag is on, route init requires the subsystem eagerly, so a missing or unusable native library fails the boot rather than the first console request. No tracked file turns the flag on. `backend/scripts/_env` leaves it out, so a devenv boots with the subsystem off exactly as a released build does, and `docker/images/docker-compose.yaml`, the self-hosting distribution, is untouched. Whoever works on the graph turns it on for one checkout through the gitignored `backend/scripts/_env.local`, which every backend and exporter dev script sources right after `_env`. Verified with `-verbose:class` over a boot's namespace load plus `ig/init-key ::routes`: 9 `com.ladybugdb` classes before this change with no flag set, 0 after it with the flag off, 9 with `enable-graph`. * ⬆️ Take Ladybug 0.19.1 `com.ladybugdb/lbug` moves from 0.18.0 to 0.19.1, the current release on Maven Central. The engine fixes a SIGSEGV on an unwrapped parameter and moves parameter coercion out of JNI, so shipping 0.18.0 would land a native library into `develop` with a known crash already fixed upstream. Nothing else changes. This branch has no `app.graph.arrow`, so the top-level Arrow field-name backticking that 0.19.x retires does not exist here and there is no workaround to remove alongside the bump. AI-assisted-by: mixed models * 📎 Pin the graph console's G6 bundle to an exact version The console loaded `@antv/g6@5` from jsDelivr, a floating major range, so the JavaScript served into the page could change without a Penpot release. Pin it to 5.1.1, the version the range resolves to today. Where the dependency finally belongs is an open question for review: vendored into `backend/resources`, declared in `frontend/package.json` if the console moves out of `/dbg`, or left on the CDN. Pinning removes the floating-code problem without pre-empting that decision. AI-assisted-by: mixed models * ✨ Add graph provenance, column naming and two transforms A projected graph is a cache of one file at one revision, built by one schema, and nothing in it said so. `GraphMeta` records the file, the revision, the schema version and the producer, and is written last, so its presence also marks the build complete and its contents say whether a cached database is still worth opening. - `graph/meta.clj`: the `GraphMeta` table and its writer. - `graph/schema/contract.clj`: one place that maps a Penpot key to its graph column. The rule is snake_case of the key; every exception, be it a rename, a drop or a type override, is recorded there with its reason, so a divergence is a diff to review rather than a silent rename. - `graph/project/document.clj`: `page-id` and the inherited `component-id` are written during the tree walk, which already knows both, rather than by a post-ingest statement. `graph/sync.clj` does the same on the incremental path, so a live-synced graph matches a rebuild. - `graph/project/transforms.clj`: a registry, so adding a derived-link pass is one entry. Adds `RefersTo` (from `shape-ref`) and `FillsSwapSlot` (from `swap-slot-*` entries in `touched`, then stripped as `ctk/normal-touched-groups` does). - `graph/debug.clj`, `graph/stats.clj`: enumerate relationship tables from the catalog instead of naming them, so the console's graph view and the ingest counts pick up new edge types without being told. - `graph/debug.clj`, `http/debug.clj`: `graph-export` gains `source=session`, which snapshots the live in-memory console graph through EXPORT/IMPORT DATABASE. Live sync moves that graph away from a fresh projection, and taking it away to query elsewhere is the point of asking for it. AI-assisted-by: mixed models * 🐛 Write graph values Ladybug's CSV reader cannot carry through Cypher Three parity failures against beadpot's suite, all one cause: the bulk loader put compound and multi-line values into CSV, where Ladybug parses a field's *contents* as a literal with no escape mechanism at all. Verified against 0.18: a comma inside a list element ends the element, quotes are kept as part of the value rather than delimiting it, and the parallel reader rejects quoted newlines outright. So a value now goes through CSV only if it cannot be misread there — UUIDs, numbers, booleans, single-line strings, and lists of those. Everything else (MAP, STRUCT, STRING[]/JSON[], any string containing a newline) is written after the COPY by one Cypher statement per row, where `app.graph.ladybug` escapes properly. Parquet removes the distinction entirely and is still the right destination (masterplan P0 T1); this is what CSV can honestly do. Consequences beyond the encoding: - `touched` entries reached the graph as `:swap-slot-…`, keywords stringified with their colon, so `LinkSwapSlots` matched nothing. Keywords now render through `name`. - Shape names lost their newlines to a flattening step that existed only to keep the CSV writer happy. They are preserved. - `applied_tokens` keys are rendered camelCase, the form Penpot's own JSON encoder produces and the one beadpot's `AppliedTokenKey` holds — a MAP column's keys are values, not schema, so they are not snake_cased. - `link-component-instances!` keys on `component-file`, not `component-id` alone. The projection denormalizes `component-id` down the shape tree, after which it no longer tells an instance head from a shape inside one, and the transform linked every descendant frame; `ctk/instance-of?` requires both keys anyway. IsInstanceOf on the variants fixture: 78 -> 60, matching beadpot exactly. `app.graph.schema.nodes/format-column-value` is now the single place that knows a column's type and its contract details, used by the bulk loader and the incremental sync alike so the two cannot disagree about a value's shape. * ✨ Type graph columns as tightly as Ladybug allows Ladybug is schema-first and strongly typed: a property key gets its type at table-creation time and there is no widening later. That makes the Malli to Ladybug mapping the whole of the graph's typing, and it was leaving a lot on the table: a transform stored as `STRING`, a rect as `JSON`, a set of feature flags as a single `STRING`. A column typed `DOUBLE[4]` is four numbers a consumer reads as a tensor row; the same value as JSON is text somebody has to parse and trust. `app.graph.schema.types` now maps, in order: scalars; Penpot value types whose layout is fixed even though Malli only sees a map or a string (`::gmt/matrix` to `DOUBLE[6]`, `::gpt/point` to `DOUBLE[2]`, `::grc/rect` to `DOUBLE[4]`, `::clr/hex-color` to `UINT32`); then structure, with collections to `T[]`, `:map-of` to `MAP(k, v)`, and a closed map of scalars to a `STRUCT`. JSON is the fallback of last resort, for schemas that genuinely admit more than one shape. Two defects fell out. `::sm/set` was unmapped, so `features` and `migrations` were single strings rather than `STRING[]`, and `::sm/one-of`, how Penpot spells a closed set of keywords, was unmapped too, so `blend-mode`, `grow-type`, the constraints and every `layout-*` were mistyped. A tight column is only worth having if the writer fills it in that shape, so `app.graph.schema.values` shapes a value for its type: a matrix record into six doubles, a hex colour into a packed integer, a map into a struct's fields. Both writers go through it, so the bulk load and the incremental sync cannot disagree. What that required: - STRUCT field names must be backticked in the DDL *and* in every literal, because a grid cell has a field named `column`. The catalog reports them bare. - A struct literal's type is its field list, so every declared field must appear, and an absent one needs `cast(NULL, '<type>')`. A bare NULL is typed STRING and changes the struct's type. - `STRUCT(…)[]` starts with `STRUCT(` but is a list, so the list check comes first. - Nested lists cannot be rendered with `str`: Clojure's `[1 2]` is space-separated and Ladybug reads it as a one-element array. Three more corrections in the same area: - `project-attrs` used truthiness where it meant `some?`, so `opacity 0` and `blocked false` projected as absent. - Set-valued columns are written sorted. A set has no order, so the column varied between builds of the same file, which is precisely what stops two builds being diffable. - An empty collection is written as `[]` rather than skipped. A shape with no fills has none; NULL would say "unknown". Renamed the `kuzu-*` helpers to `ladybug-*`: Kùzu is deprecated and Ladybug substitutes it, so a name bearing the engine should bear this one. The one remaining mention cites the upstream issue Ladybug inherits. AI-assisted-by: mixed models * ✨ Add the file-level graph columns and tighten the svg ones Split out of "🐛 Declare the shape attributes stored files carry", which is now #11125 and carries only its `common/` half. This commit is the graph's own side of that change, and it stays on this branch. `app.graph.schema.contract` pins `svg_viewbox` to `DOUBLE[4]` and `svg_transform` to `DOUBLE[6]`. The shape schema types both `:map` on purpose, because legacy files hold them as plain maps rather than as `::grc/rect` and `::gmt/matrix` records, and a tighter *schema* would reject those files. A tighter *column* costs nothing, since `app.graph.schema.values/coerce` reads either form. `app.graph.schema.nodes` declares four file-level attributes as projection `:extra` rather than in `ctf/schema:file`: `:options`, `:backend`, `:comment-thread-seqn`, and `:ignore-sync-until`. Declaring them in the file schema breaks saving, measured at 185 failures, because `app.binfile.common/update-file!` derives its UPDATE column list from a file map's keys and the `file` table has no `backend` column, that value being synthesized on read. An `:extra` is local to the graph and cannot reach a write. `app.graph.project.document` lifts `:options` out of `:data` before the blob is dropped, so a consumer reads file-level configuration without opening the blob. AI-assisted-by: mixed models * ✨ Add the Arrow prerequisites for in-memory bulk load lbug pulls arrow-memory-core and arrow-vector but no allocation-manager implementation, so RootAllocator cannot be constructed; arrow-memory-netty 18.2.0 matches the arrow-vector lbug already brings and pulls only netty-buffer, netty-common, jackson and slf4j-api, all of which the backend already has. --add-opens=java.base/java.nio=ALL-UNNAMED is the second half: without it MemoryUtil's static initializer dies with an InaccessibleObjectException that surfaces as an unhelpful NoClassDefFoundError from anything touching RootAllocator. It has to be present at JVM start, hence all three places. Note app.main/restart will not pick it up — it restarts integrant inside the same JVM, so the process must be restarted. Worth a reviewer's attention: this is a JVM-wide flag added for one subsystem. It is the standard Arrow requirement and grants nothing beyond reflective access to java.nio, but it strengthens the case for putting the whole graph subsystem behind a feature flag. * ✨ Bulk load through in-memory Arrow; delete the CSV loader app.graph.arrow stages rows as Arrow VectorSchemaRoots and COPYs from them. No file is written at any point and no value is rendered as text for the engine to re-parse, so the defect class that produced three of this branch's four backend defects cannot recur. app.graph.bulk is deleted whole. csv-representable?, defer-to-cypher?, multiline?, fixup-statements, ladybug-literal, ladybug-list-element, ladybug-list-cell and staging-dir go with it, along with the post-COPY Cypher pass that emitted one SET per row. Measured before deciding: the fixup pass was ~77% execution, 16-22% parse and 6-7% round-trip, and prepared statements could not have recovered any of it — every fixup row carries a MAP column and Ladybug binds scalars only. So this replaces rather than optimizes. Marginal ingest 4.0 -> 1.21 ms/shape; ~25 s extrapolated at 20k shapes against the ~2 min the CSV path projected. Size unchanged. Four engine facts the implementation rests on, each verified against 0.18.2 with a standalone probe: - An Arrow table is not a COPY source identifier but is a MATCH-able node label. - A MAP vector's entries child must be a non-nullable struct, and MapVector.getWriter promotes it to a sparse union, so map vectors are built from an explicit Field and filled child-first. - Ladybug names a staged table's columns and struct fields from the Arrow field names and quotes none of them, so anything needing quotes must arrive quoted — hence cypher-property-key, not column-name, names the Arrow fields. - createArrowRelTable cannot resolve endpoints against a UUID-keyed node table under any encoding, so edges stage as a node table and the COPY subquery joins them. values/coerce is reused unchanged, so the Arrow and Cypher writers cannot disagree about a value's shape; nodes/column-map-key-fn is extracted so they cannot disagree about a MAP's key spelling either. Verified with pytest --graph-origin=penpot-only unchanged at 225/38/1 and --graph-origin=penpot unchanged at 258 passed / 2 pre-existing failures, both baselines re-established against a reverted backend rather than assumed; with bp graph diff between a CSV-built and an Arrow-built graph reporting "Graphs agree"; and with an adversarial round-trip carrying a quote, a backslash, a newline, a CRLF and a tab through STRING, STRING[] elements and MAP values. The diff was necessary, not belt-and-braces: both parity suites passed an earlier revision of this change that was writing EDN into every JSON column, because beadpot's assertions never parse those columns. It also showed Arrow correcting a CSV defect — an empty Component.path was being stored as NULL, because Ladybug's CSV reader cannot distinguish an empty field from an absent one. * ✨ Add a prepared-statement surface to the graph connection `app.graph.ladybug` could only run Cypher as text. Every value the sync path writes is therefore concatenated into the statement, and nothing can ask the engine whether a statement is even valid without running it. Add the four functions that close both gaps. `prepare-on-connection!` parses and binds without executing. `execute-prepared!` binds a parameter map and runs it. `exec-prepared-on-connection!` prepares every statement in a batch before executing any of them, so a parse or bind failure aborts before the first mutation. `validate-on-connection!` returns `{:ok? :error :read-only?}` instead of raising, which is what a gate wants. `->param-value` is the only `Value` constructor on the write path. It is unconditional: on lbug 0.18.2 an unwrapped parameter does not raise, it SIGSEGVs the JVM inside `lbug_value_clone`. Parameters are scalars only, because the JNI `Value` constructor takes no list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered and the `:else` branch raises rather than crashing. Two departures from the design, both closing a JNI-handle leak on the error path: `prepare-on-connection!` closes the failed `PreparedStatement` before raising, and `execute-prepared!` closes every `Value` it built, including the ones built before a later parameter was rejected. `as-statement` accepts a bare string, so the sync builders can convert to bound parameters one family at a time rather than in one commit. AI-assisted-by: mixed models * 🐛 Write the document revision to the column that exists `set-document-revision-statement` emitted `SET d.revn`, but the column is `revision`: the beadpot contract renames `:revn` and the DDL has followed it since. The statement is the last one in every sync batch, so each batch raised after its mutations had already committed, and the session's in-memory index stayed frozen at its load-time revision. Name the column through `nodes/cypher-property-key` rather than spelling it, so the DDL and the statement cannot disagree again. Found by the binder gate in the next commit, on its first run. AI-assisted-by: mixed models * ✨ Gate every sync statement template through the binder Nothing checked that the eleven Cypher templates `app.graph.sync` emits still bind against the DDL the schema registry generates. A renamed column, a dropped table or a reserved word emitted unquoted surfaced only when a live session ran the statement, and by then the batch's earlier mutations had committed. `backend-tests.graph-binder-gate-test` opens a `:memory:` database, creates the live schema on it, and *prepares* one instance of each template without executing any of them. 14 tests, 51 assertions: the eleven templates, label coverage over all twelve registered node tables, and two assertions on the gate itself, that a `RETURN` reads as read-only and a `SET` does not, and that an unbindable statement is reported rather than thrown. It was not green on HEAD: it caught `set-document-revision-statement` writing a column that no longer exists, fixed in the previous commit. Red on both injected templates tried. No `:jvm-opts` change: CI's `-M:dev:test` already carries the native access flags the engine needs. AI-assisted-by: mixed models * 🐛 Let the engine quote the Arrow field names it interpolates `node-batch` named every top-level Arrow field with backticks, so that a column whose name is a reserved word (`Page.index`, `Document.options`) survived the DDL Ladybug generates for a staged table. The engine now quotes those identifiers itself, and it does not collapse a doubled backtick, so a pre-quoted name reaches the parser as ``index`` and `createArrowTable` fails outright: Parser exception: mismatched input '``' expecting PRIMARY Name the fields with `column-name`. The `COPY` projection is Cypher rather than DDL and keeps its own backticks through `cypher-property-key`, and STRUCT member names keep theirs too: those come out of `LogicalType::toString()`, which the DDL builder does not touch, so an unquoted member called `column` still fails to parse. Measured with `probes/arrow/probe25.clj` against lbug 0.19.1: a plain top-level reserved word loads and reads back, a pre-quoted one fails to parse, a plain STRUCT member fails to parse, and a pre-quoted one loads and reads back. Also re-dates the engine facts in the `app.graph.arrow` docstring to the version they were checked against, drops the SIGSEGV note from `->param-value` now that `Connection.execute` rejects an unwrapped parameter, and removes two references to the CSV loader. AI-assisted-by: mixed models * 📚 State what the graph schema does, not what it mirrors The graph namespaces explained themselves by citing a separate project whose Python pipeline reads the graphs this backend writes. A reader of this repository does not have that project and should not need it, and a docstring that justifies a choice by pointing elsewhere cannot be checked here. Every claim survives; only the framing changes. Column names and types are Penpot's own decision, recorded with the reason for each divergence from the snake_case default. The transform registry describes the edges it materializes. The denormalizations in `app.graph.project.document` are justified by the walk already holding both answers. Three corrections fall out of the rewrite: - `app.graph.schema.contract` claimed a test, `graph_contract_test`, that walks a checked-in schema manifest and fails on any divergence. No such test exists. The paragraph is gone. - `app.graph.project.document` pointed at `app.graph.meta/projection-transforms`, which does not exist. - `app.graph.project.transforms/registry` claimed its entries were "in application order" while `apply-transforms!` reduced over the literal vector. The three registered transforms read disjoint columns, so the order is not load-bearing. The docstring now says so, and the one real ordering constraint is stated where it applies: `link-swap-slots!` strips `swap-slot-*` entries from `touched`, so anything reading `touched` has to run before it. `contract/pending-beadpot-columns` becomes `contract/unprojected-keys`. It is referenced nowhere else. AI-assisted-by: mixed models * ✨ Refuse a mutating query from the graph console `debug/query-session!` ran whatever it was handed against the session connection. A session graph is a projection of a file, rebuilt from that file by Reload, so a mutation from the console produces a graph no rebuild reproduces and no query result explains. Bind the statement against the live schema first. A statement that does not bind reports the binder's own message and executes nothing, which also turns a misspelt table or property into an immediate error instead of an empty result. A statement that binds runs only when the engine's own read/write analysis calls it read-only. The console's query box is labelled read-only. Load, Reload, Unload and live sync are unaffected: they are separate handlers and do not go through this path. AI-assisted-by: mixed models * 🐛 Keep a synced graph equal to a rebuilt one Cold projection and incremental sync are two implementations of one mapping and nothing checked that they agree. They did not. `backend-tests.graph-sync-parity-test` projects a file into one `:memory:` database, applies a change list to that database and the same list to the file data, projects the result into a second database, and diffs the two down to the row and the column. It found four disagreements, each fixed here. **Sibling order was inverted.** A container's stored `:shapes` list runs bottom to top and `IsChildOf.position` numbers children in Penpot z-order, so appending to the list means taking position 0 and pushing every sibling up. Sync instead handed each new child the next free number, so any container edited live carried its children in the opposite order to a rebuild, and a delete left a gap where a rebuild renumbers densely. `insert-position` and `renumber-siblings` put the two paths on the same rule for `:add-obj`, `:mov-objects` and `:del-obj`, including a block move and `:after-shape`. **A moved shape kept its old parent.** `:mov-objects` moved the edge and left the shape's own `parent_id` and `frame_id` columns pointing at the container it came from. Both now follow, and `frame_id` follows through the whole subtree the shape carries, as `app.common.files.changes` does for `:mov-objects`. A top-level shape's column holds `uuid/zero`, the page's root frame, while its edge points at the Page. **A container's `shapes` column went stale.** Nothing maintained it after an add, a move or a delete. It is now rebuilt from the sibling order on every change that touches a container. **Pages came out backwards.** `projection-data` reversed `:pages` before numbering them, which is right for child shapes and wrong for pages: `:pages` is the tab order and has no second ordering to undo. `Page.index` and the page's `IsChildOf.position` are now that order. One defect the test does not reach, fixed on the way past: `index-add-shape!` accepted `:component-ctx` and dropped it, so a shape added under an instance head added in the same session inherited no `component-id`. AI-assisted-by: mixed models * 🐛 Build a synced page node the way the projection does `apply-add-page` sent the new Page node through `nodes/validate-node`, which checks a map against the registry schema and returns it unchanged. Every other node on both write paths goes through `nodes/project-attrs`, which also selects the projected keys and is the single place a column-level rule can live. A rule added there reached a rebuilt page and not a synced one. AI-assisted-by: mixed models * 🐛 Let the graph view's query filter follow the graph "Show result in graph view" froze the set of node ids the query returned and filtered every later repaint against it. Live sync creates ids the set has never seen, so a shape created while a filter was on could not appear in the view at any point, and clicking "Show full graph" was the only way to see it. A node the query would no longer match stayed. Keep the query beside the ids and re-run it whenever the graph repaints, which is only when the projection actually changed. A failed re-run keeps the ids in hand and says so on the status line rather than passing a stale view off as current. `idsInResult` and `presentIds` are extracted from the two places that scraped UUIDs out of a result. Verified in the devenv: with a filter showing 108 of 276 nodes, a `:file-change` adding a Frame published on the session's msgbus topic took the view to 109 of 277, with the new node carrying its added mark, and no interaction. AI-assisted-by: mixed models * ♻️ Rename app.graph.project to app.graph.projection `project` is a Penpot noun: a team holds projects and a project holds files, and the graph will carry a `Project` node table. A namespace called `app.graph.project.document` therefore reads as "the graph of a Penpot project" and means the opposite. `projection` is the word the rest of the subsystem already uses for the operation: `projection-data`, `load-projection!`, `:projection` in the ingest report, and `app.graph.schema.projection`. Pure rename. Both namespaces and every alias move; nothing else changes. AI-assisted-by: mixed models * 📎 Apply the project formatter to the graph namespaces `cljfmt check src/ test/` is a step of the Backend workflow and these two files did not pass it: an import block sorted the way a human reads it rather than the way the formatter sorts it, and a `cond` in `format-typed-value` indented one column short. Formatter output only. No semantic change. AI-assisted-by: mixed models * 📚 Document graph experiment architecture Add Serena memory coverage for the embedded Ladybug graph subsystem.\nDocument projection, incremental sync, console data flow, tests, and operational risks.\n\nAI-assisted-by: gpt-5.6-luna --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Alejandro Alonso <alejandroalonsofernandez@gmail.com> Co-authored-by: Andrey Antukh <niwi@niwi.nz>	2026-08-27 18:54:56 +02:00
Andrey Antukh	a3feb4ef3b	Merge remote-tracking branch 'origin/staging' into develop	2026-08-27 11:54:55 +02:00
Andrés Moya	582187dd03	🎉 Allow to validate and repair files from /dbg (#11335)	2026-08-27 11:50:14 +02:00
Andrey Antukh	87c51090b1	Merge remote-tracking branch 'origin/staging' into develop	2026-08-26 20:14:51 +02:00
Andrey Antukh	7079d33ae1	🐛 Enforce ownership check on tempfile bucket access (#11270) ... * 🐛 Enforce ownership check on tempfile bucket access The upload-tempfile RPC stores profile-id with tempfile objects, but objects-handler never verified the requester was the owner. Any authenticated user who knew the UUID could access the tempfile. Add ownership check: tempfile bucket now requires the request's profile-id to match the stored profile-id. Returns 404 on mismatch (not 403) to avoid leaking object existence. Legacy tempfiles without stored profile-id remain accessible to any authenticated user for backward compatibility. Closes #11269 AI-assisted-by: qwen3.7-plus * ♻️ Extract tempfile-bucket constant and fix docstring indentation Extract the 'tempfile' bucket string literal into a named constant (sto/tempfile-bucket) to prevent typos and make future bucket renames trivial. Updated 9 occurrences across 7 files. Also fixed minor docstring indentation inconsistency in authenticated? function. AI-assisted-by: qwen3.7-plus * ♻️ Refactor process-bucket! and authenticated? helpers Replace case with cond in process-bucket! to properly resolve sto/tempfile-bucket var from another namespace (case does not evaluate qualified vars at compile time). Redefine authenticated? in terms of request-profile-id to remove duplicated lookup logic. Closes #11269 AI-assisted-by: mimo-v2.5-pro	2026-08-26 13:05:36 +02:00
Andrey Antukh	d655aa9c63	Merge remote-tracking branch 'origin/staging' into develop	2026-08-25 20:46:47 +02:00
Andrey Antukh	b79680eeb7	🐛 Fix asset 404 for unauthenticated share-link viewers (#11342) ... PR #11036 added a per-request permission check to the file-media asset endpoints (/assets/by-file-media-id/:id and the /thumbnail variant) using bfc/get-file-permissions. Anonymous share-link viewers were then rejected because they have neither a session nor an access token, the asset URL carries no share context, and the 2-arg get-file-permissions short-circuits to nil when profile-id is nil. Make the asset endpoints share-link aware, mirroring how get-view-only-bundle already authorizes the same scenario: read the share-id from the query string, validate it as a UUID, and call the 3-arg perms/get-file-read-permissions (which chains the existing 2-arg bfc lookup, the bfc share-link fallback, and the Nitrate org-owner fallback). On the frontend, extend cf/resolve-file-media with an optional share-id arg and pass it from the WASM viewer render path using the share-id already present in [:viewer-local :share-id]. Non-viewer call sites (workspace, clipboard, code-gen) keep the original URL shape because the new arg defaults to nil. Closes #11338 AI-assisted-by: minimax-m3	2026-08-25 13:10:55 +02:00
Andrey Antukh	81c3b3cd56	📎 Update copyright name on file header (#11346)	2026-08-25 11:55:10 +02:00
Andrey Antukh	64a52d6b04	Merge remote-tracking branch 'origin/staging' into develop	2026-08-23 21:01:31 +02:00
Andrey Antukh	7c85837290	🐛 Fix session invalidation on logout to prevent token replay (#11317) ... Logout only cleared the auth-token cookie but never deleted the server-side row because delete-fn read ::id which wrap-authz no longer sets since 363b4e3778. Make delete-fn delete via ::session/:id attached by wrap-authz so replayed tokens are rejected (CWE-613, GHSA-mj9f-5cwq-7p3q). Add regression tests covering invalidation, idempotency and isolation of other sessions. Fix verified with Red→Green TDD and full backend suite (677 tests). Closes #11316 AI-assisted-by: muse-spark-1.2-contributor	2026-08-21 12:55:24 +02:00
Andrey Antukh	dd4a163217	🐛 Remove internal error details from HTTP error responses (#11288) ... * 🚑 Remove internal error details from HTTP error responses PostgreSQL exceptions, I/O exceptions, and unhandled errors were leaking raw database messages (table names, constraint names, SQLSTATE codes), filesystem paths, and internal exception details to API clients via :hint, :state, and :path response fields. Remove these fields from server-error responses while keeping full error context in server-side logs for operators. Closes #11287 AI-assisted-by: mimo-v2.5-pro * 🚑 Strip internal fields and map PG errors to safe messages Complete the security fix for GHSA-r8wx-23q6-w3gf by addressing the incomplete redaction found in code review. Add strip-internal-fields helper to dissoc :hint, :state, :path, and :context from error response data in three handlers that previously passed raw ex-data through to clients: - handle-error :internal - handle-exception :default (else branch) - handle-error :assertion (else branch) Add pgsql-state->message to map PostgreSQL SQLSTATE codes to safe, client-facing messages (e.g. 23505 → "A conflicting entry already exists") instead of returning raw PG error text. Include :message in all PSQLException response branches. Add regression tests asserting :hint, :state, :path, :context are absent from responses for :internal and unhandled ex-info errors. Closes #11287 AI-assisted-by: mimo-v2.5-pro * 🚑 Keep :hint in error protocol, fix unsafe sources Refine the security fix based on code review feedback. Keep :hint as part of the error protocol — it is essential for controlled error communication. Remove it from strip-internal-fields (which now only strips :state, :path, :context). Fix the actual sources of unsafe :hint values: - http/middleware.clj: replace (ex-message cause) with safe static strings for IllegalArgumentException, RequestTooBigException, and EOFException. These :validation errors return ex-data verbatim to clients, so raw exception messages were leaking internals. - PSQLException handler: use :hint instead of :message for the SQLSTATE-mapped messages, staying consistent with the error protocol. Update tests to assert :hint is present (with safe static values) in :internal and unhandled ex-info responses, and absent only from bare RuntimeException and IOException responses. Closes #11287 AI-assisted-by: mimo-v2.5-pro	2026-08-21 12:43:23 +02:00