Binfile import ran the whole import in a single transaction with
idle_in_transaction_session_timeout disabled (= 0), so a stalled
import could retain a connection pool slot indefinitely. Set a
finite 20 minutes ceiling via SET LOCAL instead (a compile-time
constant interpolated into the SQL; PostgreSQL does not accept bind
parameters on SET).
The v3 importer also located each file data by rescanning the full
zip entry collection once per manifest file and once per page,
making the cost close to quadratic on large files. Replace the
per-file regex matchers with a single classification pass that
groups entries by their raw path shape; consumers now lookup their
entries per file and page. As a deliberate tightening, the .json
suffix is matched literally: the previous regexes left the dot
unescaped, so crafted paths like files/<f>/tokensXjson or
objects/x-json matched by accident and are now ignored.
Closes#11579
AI-assisted-by: omen-alpha
* 🐛 Bound decompressed size of JSON entries on binfile v3 import
Every JSON/text zip entry (manifest, files, pages, shapes, colors, components, typographies, tokens, plugin-data) was decompressed without any size limit, letting a small .penpot archive exhaust the backend heap (GHSA-qcw7-v626-g6cf). Only binary storage blobs were guarded. Reuse the existing size-limiting-stream guard on the text path: 20 MiB cap per entry, 200 MiB cumulative budget per import job, plus a cheap declared-size pre-check. Both limits are configurable and wired through the binfile, management and debug entry points. Adds zip-bomb regression tests for the file entry, the synchronous manifest read and the cumulative budget. Closes#11606
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Uniform binfile import limits behind init-limits
Move the binfile import limits to a single source of truth in app.binfile.common (default-* vars) and drop the duplicated entries from config/default; env overrides keep working through the schema. Resolve all limits once per job with init-limits (::max-size, ::total-max, ::current-size, ::max-object-size, ::max-zip-entries) instead of rebuilding the map per zip entry. Thread cfg as the first arg through the v3 readers, collapse read-plain-entry into read-entry, and give size-limiting-stream a single explicit-counter arity. v1 keeps using the compiled default (mechanical rename only). No behavior change. AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Rename binfile limits to text-entry/binary-entry terms
Use text-entry/binary-entry vocabulary consistently across config keys, bfc input keys, default-* vars and the limits resolved by setup-limits (::max-text-entry-size, ::max-text-total-size, ::current-text-size, ::max-binary-entry-size). Rename init-limits to setup-limits. No behavior change. AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Rename cumulative text counter and document binary limit
Rename ::current-text-size to ::accumulated-total-text-size for clarity and expand the default-max-binary-entry-size comment to match the other limit vars. No behavior change. AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Harden binfile guards and prove budget accumulation
Add a regression test that only passes when text bytes accumulate across entries (budget between largest entry and summed total; verified red against a per-entry atom). Include the entry name in streaming-guard errors, count skipped bytes against the budget with a direct unit test, and forward all four limit keys in get-manifest. No behavior change. AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Thread cfg through get-manifest
get-manifest now takes the caller cfg and resolves limits with setup-limits like the import job itself, instead of building a single-use mini-cfg from cf/get. No behavior change. AI-assisted-by: muse-spark-1.3-contributor
The FilterInputStream proxy only implemented read() and
read(byte[], int, int). Buffered reads call read(byte[]) (and
read(byte[], int) via Clojure interop), causing ArityException
while hashing storage objects and breaking v3 imports.
Implement all read overloads and extract shared byte-count logic.
* 🐛 Add configurable limits for ZIP entry count and object size in v3 import
Add binfile-import-max-zip-entries (default 500,000) and
binfile-import-max-object-size (default 100 MiB) config entries.
Both are configurable via PENPOT_BINFILE_IMPORT_MAX_ZIP_ENTRIES and
PENPOT_BINFILE_IMPORT_MAX_OBJECT_SIZE env vars.
Entry count is checked before processing begins. Per-object size is
checked after each storage object content is resolved.
AI-assisted-by: mimo-v2.5-pro
* 🐛 Enforce actual decompressed byte limits on v3 import
The previous object-size check trusted the ZIP entry header's declared
size (ZipEntry.getSize()), which a malicious zip-bomb can forge. The
check would pass, then the full decompressed payload would be read
anyway during hashing and storage persistence.
Add size-limiting-stream, a FilterInputStream wrapper that counts
actual bytes read and raises :validation :max-file-size-reached when
the configured limit is exceeded. Wire it into zip-entry-storage-content
so both the hash calculation and storage write paths are bounded by
real decompressed bytes, not declared header size.
Also wire import limits into management.clj (clone-template) and
debug.clj (import-handler + clone path) for defense-in-depth, and
add a test that exercises the object-size limit with a real storage
object in the exported ZIP.
AI-assisted-by: mimo-v2.5-pro
* ♻️ Simplify storage GC delays and add skip-delay task params
The touched GC no longer applies an extra deletion-delay when marking
storage objects as deleted. By the time a storage object is touched, its
referencing domain row has already passed its own deletion delay, and the
reference scan is the only safety check needed. Touched objects are now
marked with deleted_at = now, so the deleted GC removes them on the next
run.
For the tempfile bucket, upload chunks now set touched-at in the future
(1h, aligned with the upload-session-gc TTL) instead of relying on a
special-case deletion delay.
Task handlers now read their task props:
- storage-gc-touched accepts :skip-delay to process all touched objects
immediately, bypassing the min-age threshold.
- objects-gc accepts :chunk-size and :skip-delay to process recently
deleted rows without waiting for the deletion delay.
This allows running the deletion cascade immediately from the REPL via
run-task! with the skip-delay option.
AI-assisted-by: deepseek-v4-flash
* ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking
Storage object lifecycle hardening:
- Add status column ('valid' | 'pending') as write-ahead marker for
object creation. put-object! inserts in 'pending' state, writes blob,
then promotes to 'valid'. Failed writes remove the pending row.
- Add :storage-pending-gc task to reclaim orphaned pending rows (e.g.
after crash between blob write and promotion).
- Verify blob existence on every dedup hit via exists-object? (fs stat /
s3 headObject). Missing blobs mark the row as deleted and create fresh
object.
- Add deletion_attempts column (migration 0154) to track physical blob
deletion attempts. Restructure gc_deleted to use chunked processing
with per-chunk transactions (short lock duration). Failed deletions
are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent
infinite loops. After 7 attempts, give up and accept orphan.
- Change del-objects-in-bulk contract to return #{fail-ids} for precise
per-id tracking (fs and s3 backends updated).
- Use tmp/tempfile for fs atomic writes with cleanup queue registration
(crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE
POSIX-only assumption.
- Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms).
- Wrap compensating delete in put-object! catch block to prevent
masking original error when connection is aborted.
- Fix assert messages in pending_gc.clj and gc_deleted.clj (pool
assertion said 'expected valid storage' instead of 'db pool').
- Add pending-objects-excluded-from-gc-deleted test. Use unique path in
put-object-write-failure-leaves-no-row test to avoid collisions.
AI-assisted-by: qwen3.7-plus
* 🐛 Fix review comments on gc-deleted and storage
- Fix process-chunk! returning nil causing (+ acc nil) crash
- Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent
infinite loop when another worker holds locks
- Pass :cause to log messages in gc_deleted.clj and s3.clj
- Fix extra space in log hint string
- Remove unused ::blob-missing? reference from storage memory
- Rename test to match actual behavior (leaves pending row)
- Add test for gc-deleted giving up after max attempts
AI-assisted-by: qwen3.7-plus
* ✨ Auto-link libraries during import based on slugified name
When a Penpot file is exported without bundled libraries and then
imported into a different environment, external library links are
broken because library UUIDs differ across environments.
This feature adds a heuristic to auto-relink libraries by matching
slugified library names against shared files in the target team:
- Export: embed external library metadata (id, name, slug, used-by)
in the manifest when libraries are not included in the export.
- Import: resolve external libraries by slugifying shared file names
in the destination team and matching against manifest slugs.
- Single match: auto-link silently (creates file-library-rel row).
- Multiple matches: emit SSE event so the frontend shows a selection
dialog for the user to pick the correct library.
- No match: import continues without linking (current behavior).
Backend changes:
- Extended manifest schema with optional :external-libraries field
- Added slugify-name, get-files-names, get-shared-files-for-team,
find-shared-files-by-slug helpers in app.binfile.common
- Threaded team-id into import cfg from RPC layer
- Added resolve-external-libraries and auto-link-libraries in v3
- Emit :library-candidates SSE event for multi-match cases
Frontend changes:
- Worker captures library-candidates SSE events and forwards them
- Import dialog shows auto-link notification and multi-match
selection UI with select dropdowns
- Added link-files-to-library! RPC helper for user selections
- Added en/es translations for new UI strings
Closes#9263
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Add UI for the auto-link plumbing
* ♻️ Consolidate auto-link libraries with unified export-type and fix ref integrity
Consolidates the auto-link libraries feature into a single coherent implementation:
- Unify boolean flags (embed-assets, include-libraries, link-later)
into single ::bfc/export-type parameter
- Fix critical reference-integrity bug: pre-resolution no longer remaps
:component-file refs when no link is created (multi-match / no-permission)
- compute-link-decisions as single source of truth for auto-link logic
- 80+ backend tests covering round-trip, cross-team, permissions,
edge cases, and reference integrity
AI-assisted-by: longcat-2.0
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Eva Marco <evamarcod@gmail.com>
Persist binfile manifest metadata in file_data on import so file
statistics are available at open-workspace time. Emit a new
open-workspace-file audit event enriched with file statistics:
page count, shape count, component count, linked libraries,
design tokens, and whether the file is a shared library.
Closes#11106
AI-assisted-by: mimo-v2.5-pro
Capture unique constraint violation in insert-file! and return
generic :not-found error instead of propagating raw PostgreSQL
exception, preventing file existence oracle.
AI-assisted-by: mimo-v2.5-pro
Prevent unbounded memory allocation when a crafted binfile specifies
an excessively large object size. Apply the same 100 MiB limit that
read-stream! already enforces.
AI-assisted-by: mimo-v2.5
* 🐛 Fix library updates reappear after file is reloaded
Summary
Migrate synced_at timestamps to a standalone file_library_sync table to ensure sync state is tracked for both direct and transitive libraries.
Problem
Transitive libraries (libraries imported by other libraries) are not stored as direct rows in file_library_rel. Because the system previously coupled synced_at directly to the file_library_rel schema, transitive libraries lacked a persistent location for their sync timestamps. This caused sync states to be lost or incorrectly reported for nested dependencies.
Changes
Schema Migration: Created file_library_sync and migrated existing synced_at values from file_library_rel.
Decoupling: Removed tight Foreign Key coupling to allow sync rows to exist independently of specific relationship records.
Persistent Writes: Added upsert-file-library-sync! helper. Updated all import, duplication, and RPC write paths (v1/v2/v3 importers, link-file-library) to ensure every write persists a sync row.
Unified Reads: Updated both direct and recursive/transitive library queries to fetch synced_at from the new table.
Testing: Added regression tests to verify that sync rows are correctly created/updated even when a transitive relation is absent in file_library_rel.
Impact
This fix ensures that the system accurately records and retrieves sync states for the entire library dependency tree, resolving the bug where nested libraries appeared out of sync.
* ✨ MR review
The recursive `read-items` function in `app.util.sse/read-stream`
caused a synchronous stack overflow when reading buffered stream
data. Each `rx/mapcat` call chained another recursive invocation
on the same call stack without yielding to the event loop.
Replace the recursive pattern with an `rx/create`-based async pump
that uses Promise `.then()` chaining, keeping the call stack depth
constant regardless of stream size.
Also add progress reporting with names and IDs during binfile
export and import, and bump `eventsource-parser` dependency.
Closes#9470
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The recursive `read-items` function in `app.util.sse/read-stream`
caused a synchronous stack overflow when reading buffered stream
data. Each `rx/mapcat` call chained another recursive invocation
on the same call stack without yielding to the event loop.
Replace the recursive pattern with an `rx/create`-based async pump
that uses Promise `.then()` chaining, keeping the call stack depth
constant regardless of stream size.
Also add progress reporting with names and IDs during binfile
export and import, and bump `eventsource-parser` dependency.
Closes#9470
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The current binfile export process uses a streaming technique. The
major problem with the streaming approach is the case when an error
happens on the middle of generation, because we have no way to
notify the user about the error (because the response is already
is sent and contents are streaming directly to the user
client/browser).
This commit replaces the streaming with temporal files and SSE
encoded response for emit the export progress events; once the
exportation is finished, a temporal uri to the exported artifact
is emited to the user via "end" event and the frontend code
will automatically trigger the download.
Using the SSE approach removes possible transport timeouts on export
large files by sending progress data over the open connection.
This commit also removes obsolete code related to old binfile
formats.
Caused when file is deleted in the middle of an exportation. The
current export process is not transactional, and on file deletion
several queries can start return not-found exception because of
concurrent file deletion.
With the changes on this PR we allow query deleted files internally
on the exportation process and make it resilent to possible
concurrent deletion.
Replace general usage of virtual threads with platform threads
and use virtual threads for lightweight procs such that websocket
connections. This decision is made mainly because virtual threads
does not appear on thread dumps in an easy way so debugging issues
becomes very difficult.
The threads requirement of penpot for serving http requests
is not very big so having so this decision does not really affects
the resource usage.
We don't migrate the libraries for avoid cascade migration; it is not ideal
but it reduces the total of the required memory needed for process a single
file migration that requires libraries to be loaded.
