15 Commits

Author SHA1 Message Date
Andrey Antukh
e23ca02aa2
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key

The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON
key. The storage GC parsed it to find live references, and nothing stopped it
from physically deleting an object a live row still pointed at: a failure in
that scan lost the blob.

Migration 0159 moves the pointer to a nullable `storage_object_id` column on
`file_data`, backfills it from the metadata key, removes the key from the
rows it backfills, indexes the column and rebuilds the foreign key to
`storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`.
`INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the
chunk transaction, while a deferred check would fire at commit, outside it,
and the sweep would never move past the blocked chunk.

The column is the only source of truth. The `storage-ref-id` entry is gone
from the file metadata schema, and the single point that serializes the
metadata drops it, so no write can resurrect it. `upsert!` does not query
the row: the pointer of the row being replaced travels in its params, and
the object is touched before the row is written. Fragments and snapshots
pass nil because a modified pointer always gets a new id; the offload passes
its own row; `insert-file!` drops it, since an INSERT never replaces a
persisted object.

`update-file!` takes the file map read from the database plus the fields to
change and merges them, so the pointer comes from the read map and no caller
can lose it by building a partial map. A map without `:data` raises
`:missing-file-data`: it is not `assert`, which is compiled out in the
deployed binary. The params cannot override the pointer or retarget the
write with `:id`/`:file-id`.

`bfc/file->response` strips the pointer from every response (`get-file`,
`create-file`, `duplicate-file`, the view-only bundle, the team access
request, the comments read, the library summary, the snapshot read and the
debug download). A `file-data` object is authentication-only, so a leaked id
would let any logged-in profile fetch the blob from `/assets/by-id`.
`get-file-fragment` resolves through the file data backend, so a fragment on
cold storage no longer hands nil to `blob/decode`, and answers only the keys
of its result schema.

The storage GC reference scan, the restore union and `objects-gc` read the
column. The offload reads `SELECT fd.*`, so its rows go through a
`decode-row` that decodes jsonb metadata and drops a JSON null, which the row
schema rejected; a row without a readable object raises
`:missing-storage-object` with the file and row ids. A chunk blocked by the
foreign key (SQLState 23503) is a real anomaly, since the reference scan
should have frozen the object: it stays an error with its own hint and the
chunk is retried on the next run until the reference is fixed.

The release is a coordinated restart, not a rolling deploy, and not
rollback-safe for the rows it writes: the previous code writes and scans the
metadata pointer, so a key it writes misses the backfill and removing it
loses the reference. `operations/2.20.0-file-data-storage-object-id.md`
builds the column, the index and the foreign key online and runs the
backfill with every instance stopped, then marks the step as applied; a
fresh or small instance does the whole job in the migration. The runbook
keeps the validation scan out of the stopped window and adds a lock timeout
and retry to the online statements.

Tests cover the foreign key shape, the backfill and its corrupt and orphan
guards, the rebuild of a missing index and of a wrong, non-deferrable or
deferred constraint, the deletion refused by a live reference, the metadata
cleanup, the offload row normalizations, the touched objects of the unshare
and the restore, the response stripping on every call site and the chunk
blocked by the foreign key. The backend memories and the developer guide
document the pointer, the GC behavior and the two migration paths.

Closes #12768

AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free

* 🐛 Hold the file row lock across the migration write

`get-file` applies the pending migrations in memory and, when it finds
any, persists the result. That write read the row without a lock and
`update-file!` is an unconditional UPDATE, so a save that committed in
between was overwritten: the saved content, `revn` and `modified-at`
went back, the next save from that client failed with `:revn-conflict`
and the object the save left became an orphan the storage GC cannot
revisit once it is frozen.

The read now runs in a transaction of its own (`get-file` calls
`get-file*` inside `db/tx-run!` and accepts a cfg map only), so the
`file` row lock covers the re-check and the write. `get-file*` reads
the row once (`sql:get-file` plus `resolve-applied-migrations`) and
completes it with `attach-file-data`; only when a migration is pending
it reads the file again under `FOR UPDATE`, re-checks `need-migration?`
on that fresh map and persists it (`migrate-file`). The common case
stays at one row read and one migrations resolution, the migration case
keeps a single decoded copy in the heap, and a row that changed in
between is not overwritten. A migration found on the unlocked read is
applied in memory only, for the next read to persist.

`migrate-file` returns the reattached map when it persists, with the
`:storage-object-id` of the row the write left, and the realized map
when it does not; `:realize?` is what returns the read fully plain.
`fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead
of a boolean, `update-file!` propagates it and the migrating read puts
it back in the map it returns, so a caller that persists again touches
the current object instead of the replaced one. `insert-file!` keeps
nil: an INSERT never replaces a persisted object.

The write also compares the stored pointer with the caller's value
(`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it
affects no row. Every writer holds the `file` row lock, so a pointer
that no longer matches is a protocol violation, not a race to retry.
The expected pointer is a separate argument because the storage branch
replaces `:storage-object-id` before the upsert, and the check lives in
`upsert-in-database`: the `legacy-db` branches do not use that SQL and
their count can legitimately be zero.

The save, snapshot and restore-backup reads take `FOR UPDATE` instead
of `FOR SHARE`: they inherit the lock, and the migration branch asks
for the exclusive one in the same transaction, so a shared lock that
upgrades deadlocks two concurrent readers. A data read rejects
`:lock-for-share?` for the same reason; only row-only reads accept it.
`sql:get-file-without-data` is a plain select now, not a CTE, which
makes locking legal without loading the data. Realization has a single
owner: `attach-file-data` only resolves and decodes, and `get-file*`
realizes once per path. The applied migrations no longer travel in the
`get-file-with-summary` response, cache hit and cache miss alike.

`embed-assets` reads each library without `:realize?`, so its
`:components` is an unloaded pointer map; `absorb-assets` walks it with
`vals`, `PointerMap.seq` calls `load!` and that raised with no loader
bound, so every export with `:merge-libraries` of a library with
components failed. The absorb runs under the library's own loader now.

The in-place binfile import (`import-file-and-overwrite*`,
`::overwrite`, `::bfc/file-id`) had no product caller and cannot carry
the previous `storage_object_id`, so overwriting a storage-backed row
would leave its object unreclaimed. `import-files!` rejects a cfg with
`::bfc/file-id` (`:not-supported`) and the helpers only that path used
go with it (`invalidate-thumbnails`, `*reference-file*`, the
`::overwrite` id branch of `index-object` and of `save-file!`, the
`on-conflict-do-nothing?` flag of the media insert).

Tests: a new `file-migration-read-test` pins the single read, the
locked re-read, the concurrent save interleaving, the double migration
and the read-only path; the upsert tests cover the return shape and the
stale pointer; the SQL recorder they share lives in
`backend-tests.helpers`, and the save, snapshot, restore and summary
sequences follow the new reads. The backend memories document the read
contract and the pointer.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Harden the get-file guard and skip its read-only transaction

The cfg guard of `get-file` was an `assert`, which `app.config` elides
when `:backend-asserts` is off, so a deployed binary accepted a bare
pool or connection while the docstring promised it raises. It throws
`IllegalArgumentException` now, the class the option guards of the same
namespace already use, and the existing test expects it instead of
`AssertionError`.

A read with `:read-only?` applies the pending migrations in memory and
writes nothing, so the transaction the migration write needs only costs
two round trips there. `get-file` runs it with `db/run!` instead: with
a cfg that carries a connection it still joins the caller's
transaction, and only the reads that would have opened their own skip
it. A lock option needs a read-write transaction from the caller,
because PostgreSQL rejects row locks in a read-only one; the docstring
states that.

Tests: the guard test expects the new class; a cfg marked
`::db/read-only` reads a pending migration without persisting it (the
config flag that fills that key is legacy, but the key is set by hand);
and a read-only read is pinned as not opening a transaction while the
same call without the option does. The backend memory documents the
split and the guard.

AI-assisted-by: deepseek-v4.1-flash
2026-11-14 19:27:18 +01:00
Andrey Antukh
1db1487c7f
🐛 Require thread access when marking threads as read (#12894)
* 🐛 Require thread access when marking threads as read

mark-all-threads-as-read took any list of thread ids and wrote the read
status of the caller on each of them, without checking that the caller
can access the file that owns the thread. It was the only comment
mutation that did not check permissions, and it took the ids as an
unbounded array, so a single request could keep a database connection
and a transaction open for a long time.

The rpc now resolves the threads in one query and checks the comment
permissions of every distinct file, answering not-found when a thread
does not exist or is not accessible. Both cases answer the same error
on purpose, so the response cannot be used to tell existing threads
apart.

The status rows are now written with a single insert, keeping the
modified-at date one second in the future that makes the thread count
as read. Repeated ids are deduped, the command returns nil again like
the sibling commands with no result, and the schema limits the number
of threads per call to 500 while the dashboard sends the ids in batches
of that size.

Tests cover access rejection, unknown threads, mixed batches that write
nothing, repeated ids, empty lists, marking twice, and the frontend
batch split.

Relates to GHSA-j4vc-cmf6-jhr7

AI-assisted-by: minimax-m3.1-flash

* 🐛 Refresh the unread list when a batch fails

The dashboard sends the thread ids in batches of 500 and each batch
commits on the server. A failure in a later batch left the unread list
stale: the re-read only ran when every batch succeeded, so the threads
already marked as read stayed visible as unread.

The re-read now runs in both paths. On failure the event is emitted
before the error is propagated, so the error toast still shows and the
success toast stays limited to a full success.

The backend also answers an empty list without opening a transaction,
since there is nothing to check and nothing to write.

Relates to GHSA-j4vc-cmf6-jhr7

AI-assisted-by: deepseek-v4.1-flash
2026-11-10 15:54:11 +01:00
Andrey Antukh
0420db1ca9
🐛 Apply the share-link page scope to the comment queries and writes (#12888)
* 🐛 Apply the share-link page scope to the comment queries and writes

A share link can be restricted to a subset of the pages of a file, and
the comment queries enforced that scope. But the RPC that resolves the
profiles of the commenters authorized the request with
check-comment-permissions! and dropped the returned perms, so its query
filtered only by file id. The holder of a page scoped link got the id,
email, fullname and photo of every profile that commented on any page
of the file, including the pages the owner left out of the link.

The query now takes the pages of the share link and only selects the
authors of the threads on them. The profile of the caller is still
added, so the viewer keeps its own avatar.

The scope was missing as well on the comment writes that take a page id
or a thread id: create-comment-thread accepted a page outside the link,
create-comment accepted a thread of an unshared page, and
update-comment-thread, update-comment-thread-position and
update-comment-thread-frame accepted a thread of an unshared page. They
now raise not-found, the same way the comment queries already did.
Replying to a thread of an unshared page also sent the emails to the
thread participants.

The page scope rule now lives in one place: share-link-pages returns
the pages granted by a share link and check-page-scope raises not-found
when a page is outside of it, so get-comment-threads reuses the helper
instead of checking the pages inline.

The docstring of get-profiles-for-file-comments said that it returns
the participants of all the comment threads of the file, which is not
true anymore for a share link. The docs are public, so the text and
the changes list are updated.

The tests of the share link page scope are hardened: the setup asserts
that the comment threads are created, the membership case has a second
commenter so it can no longer pass for the wrong reason, and the link
without pages, the allowed paths of the thread updates and the
who-comment=team denial are covered.

Relates to GHSA-89cq-3wv7-987r

AI-assisted-by: minimax-m3.1-flash

* 🐛 Enforce the share-link page scope on the remaining comment writes

Check the page scope of the share link before mutating in the four
comment writes that still skipped it: update-comment-thread-status,
update-comment, delete-comment-thread and delete-comment. The check
runs before the ownership checks, so an out-of-scope write fails with
not-found and the response does not reveal that the comment or thread
exists. Load the file in create-comment only after the permissions and
page scope checks, matching create-comment-thread, so a rejected reply
does not load the file data.

Cover the denial and the in-scope paths of the four commands, and the
fail-closed case of a link without pages on the comment reads. Move
the outsider thread creation after the profiles assertion so the
caller-union assertion fails when the union is removed. Document the
page scope invariant in the backend permissions memory.

AI-assisted-by: deepseek-v4.1-flash
2026-11-10 15:53:07 +01:00
Andrey Antukh
a53481c3b7
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids

Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:

- prepare-rpc-event adds :team-id when the event carries a project or
  file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
  per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.

Closes #12219

AI-assisted-by: deepseek-v4.1-flash

* ✨ Emit delete and hard-delete audit events with entity data

The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:

- delete-team keeps returning nil but now carries the deleted row
  (id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
  props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
  with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
  params already carry team-id and member-id.

The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Announce cascade deletes and missing ids in audit events

The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.

Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile delete in audit events

The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile restore in audit events

The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Stop attributing SREPL events to the subject profile

profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Expose file analysis data for the nexus pipeline

The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.

Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename deletion audit events to action names

Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Read worker parallelism knobs with plain config keys

The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix binfile audit test with a real file row

`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix audit event fidelity from review findings

The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.

Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.

Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Nest cascade ids under :cascade in delete event

The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.

AI-assisted-by: muse-spark-1.3-contributor

* 📎 Update copyright name on file header

The company name changed; a single header in this branch still
carried the old one. Align it with the current form.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix nil event-id on webhook trigger audit rows

process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-15 13:01:18 +02:00
Andrey Antukh
f5aad7b1ae Merge remote-tracking branch 'origin/staging' into develop 2026-09-01 12:45:17 +02:00
Andrey Antukh
6d9f411fab
🐛 Enforce share-link comment permissions and page scope (#11371)
Fix two security vulnerabilities in comment RPCs when accessed
via share-links:

- GHSA-4p97-v4wg-jxfx: Share-link holders with who-comment=team
  could bypass the restriction and comment. The check-comment-permissions!
  function treated can-read as sufficient, but share-links always set
  can-read=true.

- GHSA-fwm4-hm9f-rmcp: Comment query RPCs returned threads from all
  pages, ignoring the share-link's :pages restriction.

Changes:
- files.clj: Differentiate :membership vs :share-link in
  check-comment-permissions!. For share-links, require
  has-comment-permissions? only (who-comment=all).
- comments.clj: Filter threads by (:pages perms) for share-link
  access in get-comment-threads, get-comment-thread, and get-comments.

Closes #11370

AI-assisted-by: qwen3.7-plus
2026-09-01 08:48:55 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
alonso.torres
b1dda02b47 ✨ Add mentions to notifications 2025-01-09 11:55:53 +01:00
Andrey Antukh
a31be7e2ff ✨ Use a prefixed dir for storing temp files
And mark them for deletion on JVM exit.
2024-02-14 09:53:54 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrés Moya
dd2321a37b 🐛 Fix weird numeration creating elements in dashboard 2023-02-02 10:19:36 +01:00
Andrey Antukh
27451b9796 ♻️ Refactor comments RPC methods and add tests 2022-12-31 12:00:57 +01:00