* 🐛 Fix duplicate-file crash on nil keys in deleted component objects
* ♻️ Withdraw the legacy-36 migration in favor of 0036
legacy-36 and 0036-clear-nil-keys-in-objects perform the same
cleanup of nil keys in `:objects`. Migrations only clean up
invalid data here, so they carry no order dependency between
them and running both is wasteful.
Remove the legacy-36 defmethod and its id from
`available-migrations`, the same way 0021-fix-shape-svg-attrs
was withdrawn in favor of 0035. Files that recorded legacy-36
re-migrate through 0036; their persisted `:migrations` lists
keep the old id, and the playwright fixtures recording it stay
untouched.
Add a migrate-file test proving a file that recorded legacy-36
is re-armed for 0036 and passes schema validation after.
Document the withdrawal pattern in the migration subtleties
memory.
AI-assisted-by: inkling-2-small-free
---------
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
The `move-unit-left` shortcut was bound to the Right arrow and moved the
selection right, and `move-unit-right` the reverse, in both the workspace
and the path editor. Arrow keys behaved correctly, but the shortcuts panel
and the shortcut settings showed "Move left" next to the Right arrow, and
customizing "Move left" rebound the rightward move.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Roshan Ramani <roshanramani.dev@gmail.com>
* 🐛 Fix duplicate-file crash on nil keys in deleted component objects
* ♻️ Withdraw the legacy-36 migration in favor of 0036
legacy-36 and 0036-clear-nil-keys-in-objects perform the same
cleanup of nil keys in `:objects`. Migrations only clean up
invalid data here, so they carry no order dependency between
them and running both is wasteful.
Remove the legacy-36 defmethod and its id from
`available-migrations`, the same way 0021-fix-shape-svg-attrs
was withdrawn in favor of 0035. Files that recorded legacy-36
re-migrate through 0036; their persisted `:migrations` lists
keep the old id, and the playwright fixtures recording it stay
untouched.
Add a migrate-file test proving a file that recorded legacy-36
is re-armed for 0036 and passes schema validation after.
Document the withdrawal pattern in the migration subtleties
memory.
AI-assisted-by: inkling-2-small-free
---------
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
toggle-token and apply-token-from-input each worked out inside the
event which attributes a token goes on, and repeated most of it. The
decision now lives in pure helpers both events share: `toggle-action`
(remove the token or apply it), `apply-action` (on which attributes)
and `target-shapes` (a selected group stands for its children).
`action->event` turns the result into the event to run. Nothing
changes in behavior; apply-spacing-token-separated loses its `attr`
option, which every caller left empty.
Unit tests cover the decision, with the cases the store tests
lacked: a color token comes off the stroke too, a shape without the
token completes the others, and a spacing token on chosen attributes.
Store tests cover what had none: applying from the fill and stroke
rows, the opacity, position and size fields, the min width of a flex
child and the selection colors, choosing the token a shape already
has, and toggling a token on a group.
Signed-off-by: Florian Schroedl <flo.schroedl@gmail.com>
legacy-db is no longer a write backend: fdata/upsert! rejects it
(:legacy-backend-read-only), handle-persistence stops writing
file.data and file_change.data, and the update-params schema of
fdata no longer accepts it. Reading old payloads keeps working, and
the file-data-backend default stays db.
The configuration guide, the backend config (a startup warning reads
PENPOT_FILE_DATA_BACKEND=legacy-db as db instead of failing every
save) and the storage memories say so, with the upgrade note for
deployments that still set the variable.
AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free
* 🐛 Keep app booting when SES bootstrap fails
Wrap repairIntrinsics in a guarded bootstrap so a SES Temporal check failure
no longer aborts bundle evaluation and blanks the whole app. When it fails,
plugins and MCP stay unavailable for that session, a telemetry event reports
the error, and trying to open a plugin shows an environment warning.
Add vitest coverage for both bootstrap paths and a CLJS regression test for
the runtime-unavailable toast.
Closes#12924
AI-assisted-by: hy4-flash-free
* 🐛 Fix CI format check and plugin test entry point
Wrap the long Prettier line in bootstrap.spec.ts so `pnpm run format:check`
passes again in the plugins module, and document that the duplicated SES
options are an intentional contract snapshot.
Switch the runtime-unavailable CLJS test to the public open-plugin! entry
point instead of the private load-plugin! function. The M2 test for
signal-runtime-ready was dropped: loading app.plugins in the Node test
runner needs packaging and DOM shim changes outside this fix.
Closes#12924
AI-assisted-by: hy4-flash-free
* 🐛 Build the SSO redirect target on the server
The organization SSO flow took the destination URL from the client and
redirected to it after the identity provider round-trip, so any
authenticated profile could point that redirect at another domain: the
server never checked where the URL came from.
The RPC now takes the route token of the client (?screen=...¶ms)
and the server builds the destination out of it: the public URI plus
the query string, parsed and rendered back. The destination can no
longer leave the public URI, and the callback checks it again before
redirecting, which covers the state tokens minted before the change.
The state token also carries the profile that started the flow, so
only that session is granted the organization SSO entry: a state
prepared by someone else cannot mark the session of the one that
completes it. Whenever the RPC receives an organization id (it always
comes from the client), it also checks that the profile is a member of
the organization or its owner, instead of letting any profile start a
flow for an organization it does not belong to.
Tests cover the destination builder, a callback that ignores a
destination outside the public URI, a state that does not grant the SSO
entry to another session, and the organization access check.
AI-assisted-by: minimax-m3.1-flash
* 🐛 Check the SSO redirect target against the public URI
The callback compared the destination with a string prefix, so the two
bypasses the advisory documents (a userinfo prefix and a look-alike
host) were closed only by the trailing slash of the public URI. The
check now compares the parsed origin (scheme, host case-insensitive,
port with the default made explicit) and requires the path to be under
the base path of the public URI, so neither bypass passes.
The client sent `location.search`, the history token, whose leading `?`
is URL syntax and no part of a query string. It sends the query string
proper now (`screen=…¶ms`) through `rt/get-current-query-string`,
and `rt/token->query-string` is the single bridge between the two
shapes: every other consumer concatenates the token over the base path.
The server puts the separator back when it joins the destination, and
trims one that still arrives, so a browser holding the previous bundle
keeps its destination while the new one rolls out.
Building the destination can no longer break the flow: it catches its
own errors and lands on the public URI, which matters in the
deployments where the browser origin differs from the configured one.
A value that is not a route is ignored with a warning, decided by the
keys it parses into (a URL becomes a single key holding the whole URL),
and a route with no `screen`, like the root params of the application
(`template=…`), stays a usable destination.
The organization membership lookup is now shared by the check that
gates the SSO flow and the one that raises, so the two cannot drift
apart, and the docstring of the RPC lists `:no-organization-access`
next to `:no-team-access`.
`rt/nav-raw` returns its `:href`/`:uri` through `@event`, as its
sibling `rt/navigate` already did. Nothing in the app listens to
`::rt/nav-raw`, so the callers behave the same; the SSO test now reads
the destination from the event instead of intercepting the helper.
Tests cover the two bypasses and the base-path rule, the RPC with an
unusable destination, the value that is not a route, the request that
carries both ids, an organization that does not exist, the query string
the frontend sends, and the two router helpers.
AI-assisted-by: minimax-m3.1-flash
* ♻️ Deduplicate SSO destination URL building
The backend had an unreachable blank check and a one-line wrapper around
the URI helper in build-dest-url. On the client, the SSO retry and error
flows each assembled the destination from public-uri and the route params.
Flatten build-dest-url with a cond and call u/query-string->map directly.
Add rt/token->url as the single client helper for absolute route URLs, with
tests for both token shapes and for a blank token. Update the routing memory
with the new helper contract.
AI-assisted-by: hy4-flash-free
Move what the token group works out into pure functions with tests:
whether a type's section is open, whether a text is being edited, and
what clicking a token pill does. Nothing changes in behavior.
Signed-off-by: Florian Schroedl <flo.schroedl@gmail.com>
* 🐛 Show resolved color in alias token edit form swatch
The color token form opened every alias token with an empty swatch.
The resolved value only reached :color-result once the value field
was touched, so the swatch had nothing to show on open.
Keep the resolved color for the swatch when the field is untouched,
without showing the "resolved value" hint. Add an e2e test that loads
a fixture with a color token and an alias to it.
Closes#12672
AI-assisted-by: claude-opus-5-5
* 🐛 Show resolved value when opening an alias token edit form
The token edit form only showed "Resolved value" after typing in the
value field: the single-value inputs dropped the first resolution while
the field was untouched, and the typography/shadow reference field read
its first value from [:value :reference], which an alias token does not
have (its value is the reference string).
Show the hint on open when the resolved value differs from the written
one, so a plain `16` gets no `Resolved value: 16`. Start the reference
field stream with the alias string. Extend the e2e fixture with font
size and typography aliases and add a test for them.
Closes#12788
AI-assisted-by: claude-opus-5-5
* 🐛 Carry the team id in every team scoped navigation
The eleven `go-to-*` helpers of `app.main.data.common` resolved the
team with a plain `or` against `:current-team-id`, or read that key
alone, so an event emitted while no team was initialized built a
dashboard or workspace route without a `team-id`. `team-container*`
raises `:not-found :invalid-team-id` for such a route and
`exception-section*` does not report `:not-found`, so the user landed
on the not-found page and PostHog never saw it.
Add `resolve-team-id`, which falls back to the team of the route
already on screen and then to the profile `:default-team-id`. The
route goes first on purpose: an emitter that runs from a team screen
knows which team the user is looking at, and the personal team would
open their file under another team. The `:default` sentinel keeps its
meaning. With none of the four the helper still navigates with a nil
id, which is the documented last resort.
Harden the team-container test at the same time: it now renders the
same component with the store team the caller wants, captures markup
and errors per render, unmounts the root from a `finally`, and covers
the guard that compares the route id with the store team. React
reports a hook-order violation through `console.error` and recovers,
and only when a render adds hooks, so the harness captures that too
and the "id leaves" direction stays a guard test.
Closes#12962
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Measure the navigation tests by termination
The `emitted` helper of `common_test` waited for the first event, with a
budget of 50 macrotasks, so a helper that emitted anything before its
navigation could be disposed early, and a helper that stopped closing
its stream passed on the first event. Wait for termination instead:
`async/observe` over a subject, and make `go-to-dashboard-search`
complete by pushing the `::rt/navigated` it listens for, which is the
event it waits for in the app to focus the search box. Every helper of
the family is measured the same way now, `-search` included.
`nav-params` returns nil when no navigation matched instead of calling
`deref` on nil, as its docstring promised. Add the mirror test for the
`:default` sentinel of `go-to-dashboard-deleted`, the second copy of
the branch that `-recent` already had covered.
Correct two memories: the resolver decides for every team scoped
helper, not for every `go-to-*` and not alone, since two direct
`rt/nav` emitters are the residual; and `(rx/empty)` is for a stream
that must not emit, not for one that must stay open.
Relates to #12962
AI-assisted-by: deepseek-v4.1-flash
The analysis of a package reads the metadata of every media and
thumbnail of its container to attribute the shared objects to each
file. It read them all at once, one promise per entry, and the
container of a big file holds thousands of them: every read opens its
own decompression stream, and Chromium fails the whole analysis under
that load while Firefox keeps up. The failure is masked by zip.js with
a generic "Cannot close a ERRORED writable stream".
The references now go through an observable, one entry at a time, and a
reference that cannot be read leaves its file without a size instead of
failing the package: a number that misses one of the objects of the
file is worse than no number.
Closes#13022
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Invalidate all sessions on password recovery
recover-profile validated the recovery token and updated the
password, but never deleted the existing server-side sessions in
http_session_v2. A previously captured auth-token kept working
after the victim "recovered" the account (GHSA-3j2p-qr45-q365,
CWE-613).
The command now calls session/invalidate-all for the recovered
profile. The command already runs with ::db/transaction true, so
the delete joins the same transaction as the password UPDATE: both
commit or roll back together. There is no current session to
preserve — the RPC is unauthenticated — which is the same contract
the profile deletion path uses.
Tests in rpc_auth_test.clj pin the behavior: the victim's sessions
die, other profiles' sessions survive, and a failed recovery is a
no-op. Both invalidation tests failed before the fix.
AI-assisted-by: inkling-2-small-free
* 🐛 Pin recovery-transaction atomicity after review
Review of 92a65f544b found the tests pinned the end state but not
the rollback guarantee the fix relies on: the session DELETE joins
the password UPDATE in one transaction.
Add recover-profile-rollback-keeps-password-and-sessions: send-template
throws after the update, so the command must fail with password and
sessions both untouched. recover-profile-failure-keeps-sessions now
also asserts the password, matching its docstring. The invalidate-all
docstring mentions the recovery caller.
Addresses review findings M1, M2 and M3.
AI-assisted-by: inkling-2-small-free
* 🔥 Drop recovery test that never reached the fix
The failure test used a token with the :authentication issuer, so
tokens/verify rejected it before any write. It passed with or without
the invalidation fix and exercised no new code. The rollback test
already covers that a failed recovery leaves no partial effects.
AI-assisted-by: hy4-flash-free
When dragging a shape each tile that needs repainted is repainted via
draw_current_tile_into_backbuffer in fast mode. So if you drag over a tile it
ends up dropping drop shadows etc.
IIUC, each tile is 512x512px but drawn on a 1024px surface with 256px margins
so that the background blur has something to read. In a regular render we
render the 1024px surface but only copy the center 512px of the tile into the
backbuffer. But in draw_current_tile_into_backbuffer we were copying the whole
1024px surface. This draws over the other tiles with a fast mode render, even
though those other tiles don't need repainted.
This fixes it by reusing draw_surface_src_rect_to_dst which clips the canvas
properly so we only draw the 512x512px of the tile.
AI-assisted-by: claude-fable-5-2
On a test file `mem_benchmark` with 30k boards on one page, around 11ms is
spent each time the view is panned/zoomed iterating over each board and
checking has-grid?, which after https://github.com/penpot/penpot/pull/12790 is
now slightly slower because it involves an array scan.
We don't need to compute the list of boards with grids on every step of the pan
and can memoize it, which saves 30k calls to has-grid? on each frame.
AI-assisted-by: claude-opus-5-5
* 📚 Name a worker count in the frontend testing memory
The memory gave the command and left the count to the config, whose
default of one worker makes the whole suite slow, and CI's two per
shard reads as the number to copy.
Records the count to reach for and what it costs: about 8 GB of
browser memory at four workers, so a 64 GB host takes four beside a
watching devenv, and a 32 GB host takes two, or four on a slim devenv.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Gate the Playwright worker count on cores and memory
Four workers keep about nine hardware threads busy, measured as the
container's CPU seconds over wall time, and hold about 8 GB of browser
processes. States the cost per worker, so a reader on a smaller machine
does not oversubscribe it.
AI-assisted-by: deepseek-v4.1-flash
The worker draws local exports and looks custom fonts up in its own store
to find their files. Only the main thread fetches the team's fonts, so the
worker's store had none, and exports drew text set in a custom font with
the fallback font.
The main thread now sends the worker the team's fonts each time they
change, and the worker keeps them in its store.
AI-assisted-by: claude-opus-5-5
This PR moves component thumbnail rendering off the main thread
and onto the worker, which avoids the freezes.
The main thread will send the worker over a request containing the component's
shape data, and the worker will render it in batches. Once finished it will
both persist the thumbnails to the server if requested and send back the PNG
data to the main thread.
AI-assisted-by: claude-opus-5-5
The wasm path drew file thumbnails on a 252px wide canvas, wrapped the
PNG in an SVG and sent it to the rasterizer, which scaled it up to its
default 300px width. The result was blurry on the dashboard on high-DPI
screens and in the link preview card.
Render the wasm thumbnail at 600px wide, twice the old size, and upload
the canvas PNG as is, without the rasterizer round trip. The SVG render
path keeps its current size.
Closes#13025
AI-assisted-by: claude-opus-5-5
FontForge opens a CID-keyed OpenType font as several sub-fonts, and
Generate() writes only the active one. Most CJK fonts use this layout,
so the TTF used by the render-wasm renderer kept only a few hundred
glyphs (the Latin sub-font) and lost its family name. Merge the
sub-fonts with CIDFlatten() before generating, both in the backend and
in the media-processor. The call is guarded because it fails on fonts
that are not CID-keyed.
Add a small synthetic CID-keyed font with two sub-fonts as a fixture
and check that the converted TTF keeps all its glyphs.
Closes#10976
AI-assisted-by: claude-opus-5-5
The subscription to the bus reaches redis through an asynchronous agent
action, so a message published right after the connection opens can be
lost before redis knows the subscription. The delivery assertions of the
websocket tests published once and bet on winning that race, which fails
on a loaded CI runner.
`poll-published!` publishes, drains a window and publishes again only
when the window saw nothing, up to three attempts. The assertion still
proves that the client receives a message it published, and the negative
assertions keep the single drained window they need. A test pins the
retry so it does not go back to a single publish.
AI-assisted-by: deepseek-v4.1-flash
Add the 2.21.0 changelog section, generated from the milestone
issues and the advisory source of truth.
The 🔒 entry references GHSA-m53j-2766-6jqw with the fixing PR
#12189. The advisory now declares patched version 2.21.0 only: the
fix reached develop after 2.20.0-RC1, so the 🔒 section of
2.20.0, which claimed that release fixes the vulnerability, is
removed as well.
Entries cover the closed milestone issues after the dev-only and
exclusion filters; #9263 stays in 2.21.0 by maintainer decision and
is flagged in the anomaly report as milestone-mismatched with PR
#9958.
AI-assisted-by: inkling-2-small-free
Add the 2.19.2 section with the six bug fixes of the milestone, in
merge order, following the old 2.19.x layout: 🐛 Bugs fixed only,
no 💥 (no breaking changes, consistent with patch releases), no
🚀 (patches carry none), no 🔒 (no advisory declares
2.19.2). #12622 carries community attribution. The section stays
marked (Unreleased) until the release commit.
AI-assisted-by: inkling-2-small-free
`only_unreleased_envs` compared versions as major.minor, so for the
2.19.2 milestone the released 2.19.0 counted as "this milestone" and
bugs naming it as an affected version were listed as dev-only
candidates (e.g. #13019, "only seen on 2.19.0 and later").
Compare full semver instead, with a missing patch read as 0: only the
milestone version itself counts as an unreleased signal, and any other
version is a production hint that keeps the bug in the changelog, as
the documented contract already said.
AI-assisted-by: inkling-2-small-free
Regenerate the 2.20.0 section for the new layout: `💥` and
`🔒` are generated from the breaking change notes and the
advisory (patched_versions includes 2.20.0); features and
enhancements move to their own `🎉` and `✨` sections
instead of one mixed section; `🚀` lists the two milestone
issues labelled `release highlight` (#11759 and #11761, no closing
PR in the milestone).
Add the issues closed since the section was written: #10436,
#12052, #12135, #12615, #12728, #12738, #12740, #12948 and #13028
as bugs, and #10465, #12265, #12796, #12902, #13029 and #13030 as
enhancements. Bugs that never reached production stay out per
`dev-only-bugs` (#12860, #12138) and manual review (#12901).
#12977 leaves the section: its fix (PR #12978, merged 2026-11-13
in develop) carries milestone 2.21.0, so the issue moves there and
will be picked up by the 2.21.0 changelog.
AI-assisted-by: inkling-2-small-free
The `release blocker` label marks work that blocks the release; it is
not a signal about whether users will notice a change. Drop it from
the changelog exclusions and keep only `no changelog`.
Closed bugs carrying `release blocker` are analyzed by `dev-only-bugs`
like any other bug: a bug found while testing an unreleased feature
never reached production, so the tool is the right filter for them.
Closed non-bug issues with the label are surfaced by a manual review
step in the flow: `gh.py issues --label "release blocker"` lists them
and the user decides between `no changelog` and keeping the entry.
Update the memories, the update-changelog skill and the tests to match.
AI-assisted-by: inkling-2-small-free
* ✨ Disable the admin-console by default in the devenv frontend
The devenv does not run the admin-console service, but the frontend
showed its UI by default. The devenv containers now build the
frontend with PENPOT_DEVENV=true, which prepends disable-admin-console
ahead of the developer's config.js flags. Flags parse in order, so
enable-admin-console in config.js re-enables it; builds outside
devenv are unchanged.
Closes#13060
AI-assisted-by: inkling-2-small-free
* 📚 Require English in all code-facing text
Add a hard rule to AGENTS.md: everything in code must be in
English — identifiers, comments, docstrings, test descriptions,
log hints — with tests not excepted. The only allowed place for
another language is product copy rendered through the i18n
system. Also restated in mem:clojure/design-rules and mem:testing.
AI-assisted-by: inkling-2-small-free
Opening a file connected to a broken shared library showed the
"Internal Error" page: every connected library was loaded in a single
stream, so one failing get-file broke the whole workspace.
Catch the error per library. When the backend fails to load it (a
server error, never a transport or transient error), a directly
connected library is disconnected and a toast tells the user; an
indirect one is only dropped from the state.
Closes#12429
AI-assisted-by: claude-opus-5-5
team-container* called mf/deref only after returning early for a team id
that is not a uuid. When the same component rendered once without a team
id and then with one, the number of hooks changed between the two renders
and React threw "Rendered more hooks than during the previous render"
(error #310), which sent the user to the internal error page. The reverse
order throws "Rendered fewer hooks than expected".
Call the hook on every render and keep the uuid check in the condition
that builds the providers.
Two tests mount the component in jsdom and render it with a team id that
arrives and with one that leaves; both fail without this change.
Closes#12728
AI-assisted-by: claude-opus-5-5
AI-assisted-by: claude-sonnet-5-5
Signed-off-by: Filip Sajdak <filip.sajdak@siili.com>
* 🐛 Fix database pool min-size falling back to max-size
Since 2.16 the pool minimum defaults to the maximum (60), so an idle
backend opens and holds 60 connections at startup. Restore the pre-2.16
default of 0 so the pool grows with load and idle connections expire.
Closes#12622
Signed-off-by: Usama Sadiq <usama7274@gmail.com>
* 📚 Document database connection pool settings
Describe PENPOT_DATABASE_MIN_POOL_SIZE and PENPOT_DATABASE_MAX_POOL_SIZE,
their defaults, and sizing on a PostgreSQL shared with other apps.
Signed-off-by: Usama Sadiq <usama7274@gmail.com>
---------
Signed-off-by: Usama Sadiq <usama7274@gmail.com>
* ✨ Reference the file data storage object with a foreign key
The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON
key. The storage GC parsed it to find live references, and nothing stopped it
from physically deleting an object a live row still pointed at: a failure in
that scan lost the blob.
Migration 0159 moves the pointer to a nullable `storage_object_id` column on
`file_data`, backfills it from the metadata key, removes the key from the
rows it backfills, indexes the column and rebuilds the foreign key to
`storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`.
`INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the
chunk transaction, while a deferred check would fire at commit, outside it,
and the sweep would never move past the blocked chunk.
The column is the only source of truth. The `storage-ref-id` entry is gone
from the file metadata schema, and the single point that serializes the
metadata drops it, so no write can resurrect it. `upsert!` does not query
the row: the pointer of the row being replaced travels in its params, and
the object is touched before the row is written. Fragments and snapshots
pass nil because a modified pointer always gets a new id; the offload passes
its own row; `insert-file!` drops it, since an INSERT never replaces a
persisted object.
`update-file!` takes the file map read from the database plus the fields to
change and merges them, so the pointer comes from the read map and no caller
can lose it by building a partial map. A map without `:data` raises
`:missing-file-data`: it is not `assert`, which is compiled out in the
deployed binary. The params cannot override the pointer or retarget the
write with `:id`/`:file-id`.
`bfc/file->response` strips the pointer from every response (`get-file`,
`create-file`, `duplicate-file`, the view-only bundle, the team access
request, the comments read, the library summary, the snapshot read and the
debug download). A `file-data` object is authentication-only, so a leaked id
would let any logged-in profile fetch the blob from `/assets/by-id`.
`get-file-fragment` resolves through the file data backend, so a fragment on
cold storage no longer hands nil to `blob/decode`, and answers only the keys
of its result schema.
The storage GC reference scan, the restore union and `objects-gc` read the
column. The offload reads `SELECT fd.*`, so its rows go through a
`decode-row` that decodes jsonb metadata and drops a JSON null, which the row
schema rejected; a row without a readable object raises
`:missing-storage-object` with the file and row ids. A chunk blocked by the
foreign key (SQLState 23503) is a real anomaly, since the reference scan
should have frozen the object: it stays an error with its own hint and the
chunk is retried on the next run until the reference is fixed.
The release is a coordinated restart, not a rolling deploy, and not
rollback-safe for the rows it writes: the previous code writes and scans the
metadata pointer, so a key it writes misses the backfill and removing it
loses the reference. `operations/2.20.0-file-data-storage-object-id.md`
builds the column, the index and the foreign key online and runs the
backfill with every instance stopped, then marks the step as applied; a
fresh or small instance does the whole job in the migration. The runbook
keeps the validation scan out of the stopped window and adds a lock timeout
and retry to the online statements.
Tests cover the foreign key shape, the backfill and its corrupt and orphan
guards, the rebuild of a missing index and of a wrong, non-deferrable or
deferred constraint, the deletion refused by a live reference, the metadata
cleanup, the offload row normalizations, the touched objects of the unshare
and the restore, the response stripping on every call site and the chunk
blocked by the foreign key. The backend memories and the developer guide
document the pointer, the GC behavior and the two migration paths.
Closes#12768
AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free
* 🐛 Hold the file row lock across the migration write
`get-file` applies the pending migrations in memory and, when it finds
any, persists the result. That write read the row without a lock and
`update-file!` is an unconditional UPDATE, so a save that committed in
between was overwritten: the saved content, `revn` and `modified-at`
went back, the next save from that client failed with `:revn-conflict`
and the object the save left became an orphan the storage GC cannot
revisit once it is frozen.
The read now runs in a transaction of its own (`get-file` calls
`get-file*` inside `db/tx-run!` and accepts a cfg map only), so the
`file` row lock covers the re-check and the write. `get-file*` reads
the row once (`sql:get-file` plus `resolve-applied-migrations`) and
completes it with `attach-file-data`; only when a migration is pending
it reads the file again under `FOR UPDATE`, re-checks `need-migration?`
on that fresh map and persists it (`migrate-file`). The common case
stays at one row read and one migrations resolution, the migration case
keeps a single decoded copy in the heap, and a row that changed in
between is not overwritten. A migration found on the unlocked read is
applied in memory only, for the next read to persist.
`migrate-file` returns the reattached map when it persists, with the
`:storage-object-id` of the row the write left, and the realized map
when it does not; `:realize?` is what returns the read fully plain.
`fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead
of a boolean, `update-file!` propagates it and the migrating read puts
it back in the map it returns, so a caller that persists again touches
the current object instead of the replaced one. `insert-file!` keeps
nil: an INSERT never replaces a persisted object.
The write also compares the stored pointer with the caller's value
(`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it
affects no row. Every writer holds the `file` row lock, so a pointer
that no longer matches is a protocol violation, not a race to retry.
The expected pointer is a separate argument because the storage branch
replaces `:storage-object-id` before the upsert, and the check lives in
`upsert-in-database`: the `legacy-db` branches do not use that SQL and
their count can legitimately be zero.
The save, snapshot and restore-backup reads take `FOR UPDATE` instead
of `FOR SHARE`: they inherit the lock, and the migration branch asks
for the exclusive one in the same transaction, so a shared lock that
upgrades deadlocks two concurrent readers. A data read rejects
`:lock-for-share?` for the same reason; only row-only reads accept it.
`sql:get-file-without-data` is a plain select now, not a CTE, which
makes locking legal without loading the data. Realization has a single
owner: `attach-file-data` only resolves and decodes, and `get-file*`
realizes once per path. The applied migrations no longer travel in the
`get-file-with-summary` response, cache hit and cache miss alike.
`embed-assets` reads each library without `:realize?`, so its
`:components` is an unloaded pointer map; `absorb-assets` walks it with
`vals`, `PointerMap.seq` calls `load!` and that raised with no loader
bound, so every export with `:merge-libraries` of a library with
components failed. The absorb runs under the library's own loader now.
The in-place binfile import (`import-file-and-overwrite*`,
`::overwrite`, `::bfc/file-id`) had no product caller and cannot carry
the previous `storage_object_id`, so overwriting a storage-backed row
would leave its object unreclaimed. `import-files!` rejects a cfg with
`::bfc/file-id` (`:not-supported`) and the helpers only that path used
go with it (`invalidate-thumbnails`, `*reference-file*`, the
`::overwrite` id branch of `index-object` and of `save-file!`, the
`on-conflict-do-nothing?` flag of the media insert).
Tests: a new `file-migration-read-test` pins the single read, the
locked re-read, the concurrent save interleaving, the double migration
and the read-only path; the upsert tests cover the return shape and the
stale pointer; the SQL recorder they share lives in
`backend-tests.helpers`, and the save, snapshot, restore and summary
sequences follow the new reads. The backend memories document the read
contract and the pointer.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Harden the get-file guard and skip its read-only transaction
The cfg guard of `get-file` was an `assert`, which `app.config` elides
when `:backend-asserts` is off, so a deployed binary accepted a bare
pool or connection while the docstring promised it raises. It throws
`IllegalArgumentException` now, the class the option guards of the same
namespace already use, and the existing test expects it instead of
`AssertionError`.
A read with `:read-only?` applies the pending migrations in memory and
writes nothing, so the transaction the migration write needs only costs
two round trips there. `get-file` runs it with `db/run!` instead: with
a cfg that carries a connection it still joins the caller's
transaction, and only the reads that would have opened their own skip
it. A lock option needs a read-write transaction from the caller,
because PostgreSQL rejects row locks in a read-only one; the docstring
states that.
Tests: the guard test expects the new class; a cfg marked
`::db/read-only` reads a pending migration without persisting it (the
config flag that fills that key is legacy, but the key is set by hand);
and a read-only read is pinned as not opening a transaction while the
same call without the option does. The backend memory documents the
split and the guard.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Fix database pool min-size falling back to max-size
Since 2.16 the pool minimum defaults to the maximum (60), so an idle
backend opens and holds 60 connections at startup. Restore the pre-2.16
default of 0 so the pool grows with load and idle connections expire.
Closes#12622
Signed-off-by: Usama Sadiq <usama7274@gmail.com>
* 📚 Document database connection pool settings
Describe PENPOT_DATABASE_MIN_POOL_SIZE and PENPOT_DATABASE_MAX_POOL_SIZE,
their defaults, and sizing on a PostgreSQL shared with other apps.
Signed-off-by: Usama Sadiq <usama7274@gmail.com>
---------
Signed-off-by: Usama Sadiq <usama7274@gmail.com>