7 Commits

Author	SHA1	Message	Date
Andrey Antukh	93d36d04d8	✨ Stabilize fragmented file data across snapshots and GC (#12420) ... * ✨ Stabilize fragmented file data across snapshots and GC Snapshots, automatic and user, now store pointer-based rows that share fragments with the live file instead of duplicating page content. `get-file-snapshot` resolves those pointers on the server so version previews keep rendering plain data, and `restore!` reuses the shared rows. Export still materializes (the package carries no fragments) and import regenerates them. `file-gc` treats the main file and every unpurged snapshot as liveness roots, reads the file unrealized, and reattaches the original pointer wherever the cleaned value is unchanged, so a pass over an untouched file writes no fragments and a pass over an edited file keeps the ids of the pages it did not touch. Snapshot creation and the restore backup take the file row `FOR SHARE`, which serializes them with the pass and closes the race that let a new snapshot reference fragments the pass was about to mark. All backend pointer bindings go through the `fdata/with-pointers`, `with-tracked`, `with-tracked-pointers` and `with-untracked` macros. The loader reads fragments a previous pass marked until `objects-gc` purges them, and the frontend fails loudly with retry instead of dropping an unresolved fragment. Review fixes in this branch: the automatic snapshot attaches before pending migrations, `realize-pointers` is the single verb, `reattach-unchanged` takes the identity fast path, the restore guard takes its revn from the backup read, and `PointerMap.withMeta` generates a fresh id when metadata changes. Tests cover both snapshot writers (red without the lock), snapshot sharing and pin release through the real delete and purge paths, reattach against orphan marking, preview after offload, the pointer binding contracts and the with-meta id change. Closes #12410 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: deepseek-v4.1-flash ♻️ Pin no-churn behavior for untouched pages Pin the behavior the churn verification measured: a save keeps the fragment id of every page it does not touch, with a page of eleven keys (PersistentHashMap), a 46-shape objects map and a nine-page index (hash map). Also pin the mixed case: a data-level objects-map flip rewrites the old pointer pages once and the next save keeps the new ids. The memory records the identity rule behind it: re-assoc with the same value returns the same map in array and hash maps alike. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Stop loading untouched pages when enabling objects-map `enable-objects-map` wrapped every page on every save, and reading a page's objects forced its PointerMap to load the fragment. A save of one page in a seven-page file therefore made seven fragment reads. Skip unloaded pointers: their fragment was already wrapped by the encode-file that wrote it, so loading it only to wrap it again is a no-op. The save now loads only the pages it edits. A page with a plain fragment (a data-level feature flip) keeps its plain fragment until it is first loaded or edited; a migration that enables the feature must rewrite the data itself. Cover the change with load-count tests for save and restore, update the flip test to the new contract, and drop the stale restore comment. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Keep page fragment ids through pending migrations Applying a pending migration on read realized the file, ran the migration and persisted it again; the persist wrapped every plain page in a new PointerMap, so every page got a new fragment id even when the migration did not change it. That leaves the fragment cache cold and rewrites every fragment on the first read after a deploy. Capture the original pointers before realizing, reattach the unchanged positions after migrating (same value comparison file-gc already uses), persist the reattached file and return the realized one: the read contract is plain data. Read-only connections skip the persist and the reattach. A migration that adds objects-map skips the reattach so the pages get wrapped. Move the three helpers to fdata so file-gc and the migration path share them, and cover the path with a real 0031 migration test: the unchanged page keeps its id, the changed page gets a new one, the caller gets realized data and a second read does not migrate again. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Realize library pointers with each library's own loader Fragment rows are keyed by file id, and every caller binds the fragment loader of the file it works on. get-resolved-file-libraries returned libraries with their pages still as pointers, so the first read of a library page under fdata/pointer-map asked the consuming file for the fragment and failed with :fragment-not-found. That broke migrations and file validation for every file using a pointer-mapped library. The weak-map loader now realizes each library's pointers with the library's own id. The test fails without the change. AI-assisted-by: claude-opus-5-5 Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore the pointer-map require the repair path still needs The branch dropped `app.util.pointer-map` from the namespace while the auto-repair path (added to develop before this branch) still rebinds `pmap/*tracked*` around validation and repair in `repair-preexisting-damage` and `process-changes-and-validate`. The namespace did not compile, so the whole backend failed to load. Those toggles do not fit the fdata macros: `with-untracked` binds nil and `with-tracked` creates a fresh atom, while the repair path has to turn tracking off and restore the existing atom (and read it before the repair starts). Restore the require and keep the bindings explicit. Also pin the feature-off path with a test: with `fdata/pointer-map` off (the default), a save creates no fragments, file-gc writes nothing and the snapshot preview still returns plain data. Drop a stray `\n` char literal left at the end of one deftest. Update the memories with the exception to the macro rule, the snapshot FOR SHARE locking and the pre-migration automatic snapshot. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Validate debug files with their own pointer loader The `/dbg/actions` file validation passed the file id where `get-resolved-file-libraries` expects the file map, so the library map was seeded with a nil key. It also validated a file read without realizing it, so the first page lookup of a pointer-mapped file raised "load is not supported when *load-fn* is not bind" and the endpoint could not validate exactly the files the feature touches. Read the file realized (validation walks every page and component anyway) and build the library map from the file. The library loader already resolves each library's pointers with its own id, so the realized file and its libraries validate as plain data. Covered by a handler-level test that fails before the change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Bind the snapshot loader through the fdata helper `reduce-snapshots` hand-rolled the same `binding` that `fdata/with-pointers` builds (it also loaded the function's own pointer loader before entering the reduction). Use the macro so the rule holds there too and the namespace no longer needs `app.util.pointer-map`; the loader is identical (same cfg and file-id) and snapshot walking behaves the same. AI-assisted-by: deepseek-v4.1-flash * 📚 Note the pointer-binding exceptions and snapshot ordering Record what the feature review settled: - Two direct `binding`s stay on purpose: the fine-grained `*tracked*` toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off and restore the existing atom) and `binfile.common/encode-file` (fresh tracked set inheriting the current one). Atom-scoped macros would only be worth it if the set grows. - The automatic snapshot is taken before the pending migration, and the pre-migration window is narrow: the `get-file` RPC migrates on read and its ETag hashes the available migrations, so a post-deploy read cannot be a 304 from the pre-deploy state. The old-format snapshot only appears when the save is the first request to touch the file after the deploy, and `get-file-snapshot` previews it as stored. AI-assisted-by: deepseek-v4.1-flash * 🐛 Apply pending migrations when previewing a version Automatic snapshots are attached before the save applies pending file migrations, so a version can store the old format. The preview RPC handed that data to the UI as stored, and the viewport renders it with the current code. Realize the snapshot with the live file id (fragments are keyed by file, not by snapshot), normalize its migrations to a set (the decoder keeps the pgarray as a vector, and `set/difference` would compare indexes) and apply the pending migrations with the pure `fmg/migrate-file` and the live libraries. Nothing is persisted: the stored version keeps its data and restoring it re-migrates on the next read. The response now carries the snapshot's migrations. Use `:features` the way `get-file` does for `fdata/objects-map`: a client that does not declare it gets plain objects. The historical feature set is not checked against the enabled one on purpose: a version may predate a feature change. Cover the preview of a snapshot with a pending migration and the objects realization; both pin that the stored row is untouched. Update the file-data memory with the preview contract and the vector/set trap. AI-assisted-by: deepseek-v4.1-flash * 🐛 Decode snapshot migrations as an ordered set `file-snapshots/decode-snapshot` handed the `file_change.migrations` pgarray as a vector. `set/difference` falls back to index lookups when the first collection is shorter than the second, so the pending set was right only by accident: a stored subset happens to work, while a stored set larger than `available-migrations` (a snapshot taken by a newer build) marks every available migration as pending and re-applies it to already migrated data. Decode it as a `d/ordered-set`, the type the rest of the code and `schema:file` expect. This also covers `restore!` and the snapshot list, and it lets `get-file-snapshot` drop the `(into #{} ...)` normalization it was carrying. Re-wrap the union in `fmg/migrate` so the invariant holds even when a caller hands a plain set. A snapshot with NULL `migrations` predates SQL migration 0140 and keeps the format version of its time. Pin that legacy path: the preview regenerates the applied set from that version, migrates in memory and leaves the stored row untouched. Update the file-data memory with the invariant and the legacy path. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix flaky waiting-gauge assertion in climit test The throwable-while-waiting test asserted the exact list of waiting-gauge publications, but the component sampler republishes the same count when its first tick lands inside the recorded window under load. Assert only the last value, which is the contract the helper documents. AI-assisted-by: muse-spark-1.3 --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>	2026-11-02 15:02:55 +01:00
Andrey Antukh	3cc91e8ba2	♻️ Take the command limits before the root ones in the RPC limiter (#12686) ... * ♻️ Take the command limits before the root ones in the RPC limiter The RPC concurrency limiter took the root limiters (the ones every request passes through) before the ones a command declares, so a request waiting for the limits of its own command held the root permits of its profile while it waited: a profile with a pile-up of heavy requests saw its other requests queue behind them and be rejected with 429. That order was forced by two commands that entered the limiter from inside their handler: a command that declares no limits of its own can only take them deeper than the root ones, so two paths of one profile could take the same limiters in opposite order and park for good. The commands declare their limits in their metadata now and their handlers call the work directly, so no command enters the limiter from inside its handler and every path takes the limiters in the same order. The root limiters are appended to the sequence first, so the limits a command declares end up outermost and are taken first: a request waiting for them does not hold the root permits of its profile. The photo of a team keeps its limits, because it declares them now that the helper does not. A call from inside a handler can only reenter the limits that thread already holds, and it is refused when it asks for a new one, because that would take it after the root ones and invert the order against the other paths. The key of a limit is resolved the same way in every path, so both paths point at the same limiter and the reentrancy check sees it. Every entry of the shipped climit.edn declares a queue, so the number of requests waiting for the same limit is bounded instead of piling up until the timeout. The production file lives outside the repository and has to be mirrored to pick them up. The tests cover the acquisition order of a command with and without limits of its own, the order of the limits as a whole (a ring of any size fails, not only a pair in opposite order), the nested call that asks for a limit the thread does not hold, and the queues and timeouts of the shipped configuration. The import of a binfile declares its request as multipart in its test, the way the commands that take an upload map do. AI-assisted-by: minimax-m3.1-flash AI-assisted-by: deepseek-v4.1-flash * 🔧 Keep the commands' limits and the shipped config in sync `profile-plugin-ops` guarded only an UPDATE on the caller's own profile row, which the row lock already serializes, so its global cap could only reject a burst of different profiles toggling plugins. Both commands declare the by-profile limit alone now, and the example config drops the entry. A declared id without an entry does not limit anything: `wrap` logs a warning and runs the handler with no limiter at all, which is how a command ends up unbounded in production. A test fails now when a command declares an id the config does not list, and the header of the example says the production file mirrors the shape and lists every declared id, not the values. AI-assisted-by: deepseek-v4.1-flash * 📚 Record that background work is not waited for with a sleep A long command moved to the background reports its own completion, so polling it with a sleep inside a foreground command only blocks the session and the notification arrives anyway. The rule says to keep working on something else, or to end the response and wait for it. AI-assisted-by: minimax-m3.1-flash	2026-10-28 18:44:36 +01:00
Andrey Antukh	fdd66978fd	Merge branch 'main' into staging	2026-10-19 20:10:20 +02:00
Andrey Antukh	84d92cfd5a	✨ Add tail buckets and update-file validation timings (#12409) ... * ✨ Extend backend histogram buckets beyond 7.5s Give every backend timing histogram a tail past the shared 7500ms cap so slow RPC calls, CLIMIT waits and background tasks report real quantiles instead of clipping at the top bucket. Request and wait metrics grow to 60s, background tasks to 300s and Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the history comparable. Closes #12408 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix flaky assertion in climit republish test Read the last published waiting value instead of the exact publication list. The sampler thread started with the component runs its first tick asynchronously, and when it lands inside the recording window it publishes the same count twice, which made the full-suite CI run fail while focused runs stayed green. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add file_data blob to JSON decode helper Allow decoding file_data blobs (app.util.blob) to JSON for investigations with jq, offline from a blob file or from nREPL with the backend running. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add validation step timings to update-file logs and metrics Measure the six phases inside process-changes-and-validate (resolve-libs, process-changes, soft and hard affected and schema validations) with tpoint and try/finally, so timings are recorded even when a hard validation throws. Publish each phase to a new histogram penpot_rpc_update_file_validation_timing with a fixed step label using the request buckets (up to 60000 ms), and extend the update-file debug log with the same *-ms fields plus file, project and team ids. The deferred log only runs on success, so failures get an inline error log with partial timings. AI-assisted-by: muse-spark-1.3-contributor * 💄 Translate decode-file-data comments to English Translate the Spanish docstrings, section headers and test messages of the file_data decode helper to English, keeping code untouched. The repo convention is English in code and comments; the helper was first written in Spanish. AI-assisted-by: muse-spark-1.3-contributor	2026-10-19 13:01:35 +02:00
Andrey Antukh	57ed11942c	🐛 Fix the broken restore test and the flaky climit test ... The merge of main into staging brought the closed storage metadata schema, which requires file-id and id on file-data objects. The restore test built one without those keys, so it failed with an invalid metadata error. Its helper now takes extra metadata and the test passes the ids of the file and of its file-data row, as production does. The republish test asserted the exact list of waiting-gauge publications, but the component's sampler starts its first tick right away and under load lands inside the recorded window, publishing the same count twice. The test now builds the counters alone, so the republish call is the only publisher. AI-assisted-by: deepseek-v4.1-flash	2026-10-13 18:44:40 +00:00
Andrey Antukh	5c476b4d9c	✨ Make the CLIMIT queue visible with an accumulated gauge ... The queue gauge was written by whichever limiter of a name ran last, so a per-profile pile-up never showed: a healthy profile writing 0 overwrote the 15 of the profile that was queueing (4 of 1441 samples in production), and a limiter whose requests never come back read 0 while 72 threads stayed parked. A queue is an entry and an exit, so the new gauge is accumulated: one counter per limiter name and instance, up when a call starts waiting for a permit, down when it stops, so a call that never leaves stays visible. It is always written with `:val`, and a daemon sampler republishes every count every 15s so a name whose traffic died does not go stale in the scrape; halting the component stops that thread. Rejections had no metric at all: penpot_rpc_climit_rejected_total counts them by name and reason (queue-full or timeout), and the log lines now carry queue-max, permits, code and reason, so an operator can tell whether the limiter did its job or was saturated. The gauges that could not tell the truth are gone: queue and permits (same last-write-wins problem, and waiting plus the wait histogram replace them) and http_worker_busy_threads (reads 0 on an instance with 72 blocked threads; pool_size is the signal). No other dashboard or alert rule reads them. AI-assisted-by: deepseek-v4.1-flash	2026-10-13 14:33:32 +02:00
Andrey Antukh	f242ff9574	🐛 Keep concurrency limits from deadlocking on nested calls ... An instance left the audit events pipeline dead: every push-audit-events answered with 429 and 69 worker threads parked for good inside a limiter. Two code paths of one profile acquired the same two limiters in opposite order: a command declaring its own limits (upload-file-media-object) took process-image/by-profile and then root/by-profile, while a command with no limits of its own took the roots first and then re-entered climit from inside its handler with the same ids (create-file-media-object-from-url). Each path held what the other one waited for, and no limiter had a timeout to break the wait. Now wrap puts the root limiters after the command's own, so they end up outermost and every path meets them first: the nested chains carry only the command's own limits and run inside them, which puts both paths in the same order. impl-invoke also keeps, per thread, the set of limiters that thread is inside of and runs the handler inline when a call asks for one it already holds, so a handler can no longer queue behind itself. The shipped climit.edn sets a timeout on every limiter (60s for operations that can be slow, 30s otherwise), so no wait can park a thread for ever. The production file lives outside the repository and has to be mirrored. Tests cover the acquisition order, the deadlock, the reentrancy and the timeouts of the shipped config. AI-assisted-by: deepseek-v4.1-flash	2026-10-13 14:33:32 +02:00