mirror of
https://github.com/penpot/penpot.git
synced 2026-10-29 14:00:37 +00:00
bb76b29fe2
Teach the file-gc task to heal files whose stored data fails schema or semantic validation instead of failing the nightly run: read once (sanitize, migrate in memory, check the scheduled revn), run the shared best-effort repair loop, then take the row lock and persist the snapshot, marking the file trimmed. A new revision orphaned the current file thumbnail (the list joins on revn), so a run that changes bytes touches modified-at instead (the ETag covers it) and no-op runs touch nothing; the change is detected against the decoded pre-sanitize data, which get-file now returns as :orig-data. The handler reports a result map with a keyword reason (:cleaned, :gone, :stale-revn, :unfixable, :error) and the tiered-storage offload submit runs outside the result handling, so a queue failure after a good write stays truthful and only logs its own error. The library-user scan moved out of the write transaction, so long scans no longer trip idle-in-transaction timeouts. Repair convergence lives in one shared loop used by file-gc, the debug endpoint and the srepl helper: validate, apply the computed repair changes and repeat up to ten times, returning the input untouched when it throws or does not converge. file-gc revalidates after the loop and raises :repair-did-not-converge, so a partial cleanup is never stored as clean: the classifier reports :unfixable, the file stays unmarked and the scheduler retries it. Validation skips shapes without :shape-ref (copy roots legitimately carry none), so the repair converges instead of crashing on the get-shape precondition; a dedicated repair for copies that genuinely lost their ref is future work. Libraries load through the weak-map loader, and that loader was broken: the threading macro passed :data to update as the map itself, so every library load threw and the repair loop swallowed the exception, silently skipping every repair for files with linked libraries. The pipeline is now explicit: load, realize pointers (they take [cfg file], so the reversed -> threading was returning the Integrant system map) and then sanitize. Unconverted types broke ref resolution (detaching copies) and hid real uses from instance-of?, so libraries are sanitized on load, user files are sanitized inside the library-user scan, and the srepl helper and the debug endpoint sanitize too. Only migrate when migrations pend. sanitize-file-data now returns the input instance unchanged when neither the nil-entry drop nor the schema decode repairs anything: lazy containers (objects-map) keep their identity and file-gc can tell a real fix from a no-op by comparing with the pre-sanitize data. The walk still happens on every call, and that cost is documented on the loader. Migration 0027 heals stored data durably with the same recipe the read path applies: schema-decode unconverted types (precompiled decoders, explicit component-objects pass) plus the legacy-57 nil page cleanup that reappeared through a write path. clean-file keeps only its post-decode fixes and documents the single sanitize pass. Change-level guards reject :add-page without a valid page id (structured error, not assert, so release CLJS still enforces it) and conflicting id/page pairs. Tests pin the contract with production-shaped corruption builders: file-gc repair and the outcome matrix, loop convergence and give-up branches, the :shape-ref guard both ways, page guards, sanitize idempotence and identity (plain and objects-map), the 0027 migration, a remote copy resolved through the loaded library and the non-convergence report; existing file suites move to the result map through the run-file-gc! helper. Closes #12433 Follow-up: https://github.com/penpot/penpot/issues/12509 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: glm-5.3-flash