From 688c0fc398c3c35e35991607a5ba1269f297f6d3 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Mon, 19 Oct 2026 14:00:13 +0200 Subject: [PATCH] :bug: Keep file_change revisions on their own deletion schedule (#12419) * :bug: Keep file_change revisions on their own deletion schedule Restoring a file from the trash cleared the deletion mark on every file_change row of that file, revisions included. A revision is born already scheduled for deletion one hour later, and objects-gc only looks at rows that carry a date, so unmarking one left it in the table for ever: production had 358,517 of them across 3,614 files. The file deletion cascade had the mirror problem: it rewrote the date of rows that already had one, pushing a revision out to the trash lifetime. Restrict the restore to durable history and date only the rows that have no date yet, so a revision keeps the schedule it was born with. A pinning test covers the snapshot update path, which cannot reach a revision because it resolves its row through a read filter. The backend memories now carry the model of the three file_data classes and this invariant, in place of the two superseded stubs. Closes #12418 AI-assisted-by: deepseek-v4.1-flash * :books: Add autovacuum and Postgres exporter notes to Grafana memory Record what this Grafana instance can and cannot show about autovacuum so the next session does not have to re-derive it: - The Postgres exporter scrapes the primary as well as the read-only replica, so the primary's pg_stat_activity and pg_settings_* are in Prometheus even though the SQL datasource cannot reach them. - It publishes no pg_stat_progress_* collector, and pg_stat_activity_count has no backend_type label: autovacuum workers are the usename/application_name-less bucket with state=active. The launcher shows up as the AutoVacuumMain wait event, and it is absent on the replica. - A metric the exporter does not publish needs a custom query block in its YAML queries file and a restart of the exporter. - The Postgres server ships its own log to Loki (the job is postgresql, primary only), whose streams carry only filename, instance, job and level. The completed vacuum lines live there: a per-table history, not a view of what runs now. - The PostgreSQL section now ends with the vacuum panels, with the Vacuum TXID age table last; an in-flight per-table view would need a pg_stat_progress_vacuum collector in the exporter itself. AI-assisted-by: deepseek-v4.1-flash --- .../auth-permissions-product-domains.md | 4 +- .serena/memories/backend/core.md | 3 +- .../http-storage-filedata-subtleties.md | 31 --- .../backend/rpc-db-worker-subtleties.md | 5 - .serena/memories/backend/subtleties.md | 9 +- .serena/memories/prod-infra/grafana.md | 8 +- backend/src/app/rpc/commands/files.clj | 10 +- backend/src/app/tasks/delete_object.clj | 8 +- .../file_change_retention_test.clj | 209 ++++++++++++++++++ 9 files changed, 242 insertions(+), 45 deletions(-) delete mode 100644 .serena/memories/backend/http-storage-filedata-subtleties.md delete mode 100644 .serena/memories/backend/rpc-db-worker-subtleties.md create mode 100644 backend/test/backend_tests/file_change_retention_test.clj diff --git a/.serena/memories/backend/auth-permissions-product-domains.md b/.serena/memories/backend/auth-permissions-product-domains.md index 130cd62609..7a18794d60 100644 --- a/.serena/memories/backend/auth-permissions-product-domains.md +++ b/.serena/memories/backend/auth-permissions-product-domains.md @@ -9,7 +9,7 @@ - LDAP login validates credentials against the external directory, fetches identity data, then logs in or registers a matching Penpot profile. LDAP registration is not a separate Penpot signup flow. - Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI. - Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation. -- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/http-storage-filedata-subtleties`. +- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`. ## Permission model @@ -38,4 +38,4 @@ - Enable LDAP login locally with frontend flag `enable-login-with-ldap`; the devenv includes a configured test LDAP service. - OIDC testing requires external provider app credentials plus matching backend/frontend config. - Backend domain tests usually live under `backend/test/backend_tests/rpc/commands/*_test.clj` or nearby backend test namespaces. Use focused `clojure -M:dev:test --focus ...` from `backend/` when possible. -- For auth/session or HTTP behavior, combine backend tests with the HTTP/session notes in `mem:backend/http-storage-filedata-subtleties` because RPC-level tests may not exercise cookie/header transforms. +- For auth/session or HTTP behavior, combine backend tests with the HTTP/session notes in `mem:backend/subtleties` because RPC-level tests may not exercise cookie/header transforms. diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index a8756775a8..42e7c7b756 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -4,9 +4,8 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m ## Focused memories -- RPC, DB helpers, workers, cron, CLIMIT: `mem:backend/subtleties` +- RPC, DB helpers, workers, cron, CLIMIT, config and HTTP/session middleware, http self-metrics, media processing, and file data persistence: `mem:backend/subtleties` - Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`. -- HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`. - Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains` - Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:backend/audit-log` - Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`. diff --git a/.serena/memories/backend/http-storage-filedata-subtleties.md b/.serena/memories/backend/http-storage-filedata-subtleties.md deleted file mode 100644 index fb735b6c56..0000000000 --- a/.serena/memories/backend/http-storage-filedata-subtleties.md +++ /dev/null @@ -1,31 +0,0 @@ -# Backend HTTP, Storage, Media, and File Data Subtleties - -## Config and HTTP/session middleware - -- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them. -- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag. -- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags. -- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. -- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling. -- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`; JWTs with `kid=1` and `ver=1` are decoded as v1 session tokens, otherwise they are treated as legacy tokens. -- Shared-key auth requires `x-shared-key` as ` ` and stores the lowercased key id on the request. If no shared keys are configured it always rejects. -- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. DB sessions support both legacy string ids and v2 UUID session ids. -- Session cookies are renewed when using a legacy string id or when `modified-at` is older than the renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config. - -## Storage and media - -- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`. -- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing. -- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8. -- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error. -- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts. - -## File data persistence - -- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data. -- `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob. -- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. -- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor. -- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. -- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again. -- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders. diff --git a/.serena/memories/backend/rpc-db-worker-subtleties.md b/.serena/memories/backend/rpc-db-worker-subtleties.md deleted file mode 100644 index 9ee1d462c1..0000000000 --- a/.serena/memories/backend/rpc-db-worker-subtleties.md +++ /dev/null @@ -1,5 +0,0 @@ -# Backend RPC/DB/Worker Subtleties (superseded) - -This memory was superseded by `mem:backend/subtleties`, which holds everything it had (RPC wrappers, DB helpers, workers and cron) plus the CLIMIT notes, config/HTTP middleware, http self-metrics, storage and file data. Read that one. - -Nothing should point here any more. diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index ed7f390af6..cee764853a 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -74,4 +74,11 @@ - File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data. - `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob. - Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. -- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders. \ No newline at end of file +- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders. +- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data. +- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision. +- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor. +- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. +- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again. +- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter. +- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again. \ No newline at end of file diff --git a/.serena/memories/prod-infra/grafana.md b/.serena/memories/prod-infra/grafana.md index 085308da1b..533b09516f 100644 --- a/.serena/memories/prod-infra/grafana.md +++ b/.serena/memories/prod-infra/grafana.md @@ -7,8 +7,13 @@ Keep this memory free of internal names: no host names, no deployed versions, no ## Datasources - Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage, and the application log events of the backend), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. Tails (ms): requests and waits to 60000, background tasks to 300000, Redis eval to 10000; the sub-7500 buckets are identical everywhere, so old quantiles stay comparable. + - The Postgres exporter scrapes **the primary as well as the read-only replica**, so the primary's `pg_stat_activity`, `pg_settings_*` and `pg_stat_bgwriter_*` are in Prometheus even though the SQL datasource cannot reach them (the replica's views describe the replica, not the primary). + - Per-database data from the exporter is **aggregate only**: `pg_settings_*` (the whole server configuration, so autovacuum, checkpoint and memory parameters are readable from Prometheus), `pg_stat_database_*`, `pg_stat_bgwriter_*`, `pg_stat_activity_*`, and replication/lock counts. It publishes **no per-table series** (`pg_stat_user_tables_*` resolves empty), so dead tuples, per-table reloptions and relation sizes are SQL-only questions. + - It publishes **no `pg_stat_progress_*` collector** and `pg_stat_activity_count` carries no `backend_type` label, so a session is only identifiable by its remaining labels: autovacuum workers are the bucket with `usename` and `application_name` empty plus `state="active"`, and `pg_wait_events_connections{wait_event="AutoVacuumMain"}` is the autovacuum launcher alive (absent on the replica). + - A metric the exporter does not publish itself has to be added as a custom query block in its YAML queries file on the exporter host (that is where `pg_wait_events_connections` comes from), and in this build the file is read at startup: restart the exporter after editing. There is no per-metric flag to turn on. - Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Two things to remember when counting from it: the CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`) and their `hint="rejected"` line goes out at WARN only when something is left queued behind the call (a lone caller that times out is written at TRACE), so counting lines is not counting rejections -- the same goes for `hint="enqueued"` and `hint="acquired"`; and the only signal that exists nowhere else is the profile behind a queue (`logger="app.rpc.climit"`, whose `id` carries the key), chased there on demand. -- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data. + - The Postgres server also ships its own log (`job="postgresql"`, primary only). Those streams carry just `filename`, `instance`, `job` and `level` — no `environment`/`hostname` — so selectors need `instance=~"$host.$env.*"`. `log_autovacuum_min_duration` is low enough that every autovacuum leaves an `automatic vacuum of table "db.schema.table"` line with pages, tuples, index scans and `elapsed`: a per-table history of **finished** vacuums (retrospective, not a view of what runs now). A day of vacuums is only a couple of hundred lines, so pulling them raw over `query_range` and parsing the table, the pages scanned, `tuples: N removed` and the `removable cutoff` XID age in the client works well — less fighting than making LogQL's `pattern`/`regexp` chew a multi-line message. +- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. Catalog data (`pg_class`, `pg_partition_tree`, `pg_stats`, and `reloptions`) describes the primary faithfully, because catalogs replicate; the statistics views (`pg_stat_user_tables`, `pg_stat_all_tables`) read as zero there, since statistics are per-instance and are not replicated. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data. - **Postgres (telemetry)**, uid `9YAUuM_Gz`. - `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs. @@ -16,6 +21,7 @@ Keep this memory free of internal names: no host names, no deployed versions, no - `Production status`, uid `penpot-production-status`. Sections: general state and deploy, `RPC y CLIMIT`, PostgreSQL and pool, Redis/msgbus, tasks, JVM/process. Variables `$env` and `$host` (custom), no auto refresh. Every section opens collapsed except `RPC y CLIMIT`, which is the one read on a normal day. - Per-host investigation dashboards are separate, and the alert rules deeplink into them. +- The PostgreSQL section ends with a block of vacuum panels: two stats (autovacuum workers active on the primary, and the launcher alive), a Loki timeseries of finished vacuums per interval and table, and then the `Vacuum TXID age` table, which is the last panel of the section. The worker stat is an approximation — it counts the `pg_stat_activity` bucket described above, because the exporter publishes no `backend_type` — and the Loki panel is retrospective by construction. A per-table, in-flight view (table, phase, progress) needs a `pg_stat_progress_vacuum` collector added to the exporter itself. ## Conventions diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index c5169643bc..584ba73e35 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1305,8 +1305,16 @@ WHERE file_id = ANY(?::uuid[])") (def ^:private sql:restore-file-changes + ;; Only durable history comes back: a snapshot (carries `label`) or a legacy + ;; payload row (carries `data`, the column the new model replaces with + ;; `file_data`). Revisions have neither and are deliberately left alone: they + ;; are born already scheduled for deletion one hour later, and the collector + ;; only looks at rows that carry a date, so unmarking one here would keep it + ;; in the table for ever. The `data` half is a guard for the rows still + ;; pending migration, not a dependency: drop it with the last one. "UPDATE file_change SET deleted_at = null - WHERE file_id = ANY(?::uuid[])") + WHERE file_id = ANY(?::uuid[]) + AND (label IS NOT NULL OR data IS NOT NULL)") (def ^:private sql:restore-file-data "UPDATE file_data SET deleted_at = null diff --git a/backend/src/app/tasks/delete_object.clj b/backend/src/app/tasks/delete_object.clj index 00d0046f27..b6a30e2092 100644 --- a/backend/src/app/tasks/delete_object.clj +++ b/backend/src/app/tasks/delete_object.clj @@ -60,10 +60,14 @@ :file-id id :cause cause)))) - ;; Mark file change to be deleted + ;; Mark file change to be deleted, but only the rows that do not carry a + ;; date yet. A revision is born already scheduled for deletion one hour + ;; later, and rewriting its date here would push it out to the trash + ;; lifetime; the collector would still take it, just much later than it + ;; should. (db/update! conn :file-change {:deleted-at deleted-at} - {:file-id id} + {:file-id id :deleted-at nil} {::db/return-keys false}) ;; Mark file data fragment to be deleted diff --git a/backend/test/backend_tests/file_change_retention_test.clj b/backend/test/backend_tests/file_change_retention_test.clj new file mode 100644 index 0000000000..7cede5a164 --- /dev/null +++ b/backend/test/backend_tests/file_change_retention_test.clj @@ -0,0 +1,209 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.file-change-retention-test + "Retention of `file_change` rows through the file lifecycle. + + A file has two kinds of row. Durable history: the snapshots, which a restore + brings back. And ephemeral revisions: the xlog of an edit, born already + scheduled for deletion one hour later, which nobody needs once the sessions + that were going to read them are gone. + + The invariant under test is that a deletion date is assigned once and never + moved: restoring a file must not unmark a revision, and deleting a file must + not push a revision's date out to the trash lifetime. When either happens the + revision becomes invisible to `objects-gc`, which only looks at rows that + carry a date, and it stays in the table for ever." + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.db :as db] + [app.rpc :as-alias rpc] + [app.rpc.commands.files :as files] + [app.util.blob :as blob] + [backend-tests.helpers :as th] + [clojure.test :as t])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(def ^:private test-now + "The instant every test pins the clock to. Whole seconds on purpose: a value + with a sub-second part does not survive the round trip through the database, + and the test could not compare it for equality." + (ct/inst "2026-10-01T00:00:00Z")) + +(def ^:private revision-lifetime + "The lifetime the `update-file` flow gives a revision when it inserts it." + (ct/duration {:hours 1})) + +(defn- create-file + [i] + (let [profile (th/create-profile* i {:is-active true})] + [profile (th/create-file* i {:profile-id (:id profile) + :project-id (:default-project-id profile)})])) + +(defn- insert-change + "Inserts a `file_change` row directly, so the test builds the exact shape it + needs. The row class is decided by what the caller passes: a revision carries + only `changes`, a snapshot carries `label`, and a legacy payload row carries + `data` without a label. Returns the new row id." + [file-id {:keys [label data changes created-at deleted-at created-by]}] + (let [created-at (or created-at (ct/now)) + id (uuid/next)] + (th/db-insert! :file-change + {:id id + :file-id file-id + :created-at created-at + :updated-at created-at + :deleted-at deleted-at + :revn 1 + :version 0 + :created-by (or created-by "system") + :label label + :data data + :changes changes} + {::db/return-keys false}) + id)) + +(defn- insert-revision + "A revision exactly as the `update-file` flow writes it: no label, no data and + `changes` filled, already scheduled for deletion one hour after creation." + [file-id created-at] + (insert-change file-id {:created-at created-at + :changes (blob/encode {:foo "bar"}) + :deleted-at (ct/plus created-at revision-lifetime)})) + +(defn- find-change + [id] + (th/db-get :file-change {:id id} {::db/remove-deleted false})) + +(defn- find-file + [id] + (th/db-get :file {:id id} {::db/remove-deleted false})) + +(defn- restore-file + [file-id] + (db/tx-run! th/*system* + (fn [{:keys [::db/conn]}] + (#'files/restore-files conn [file-id])))) + +(t/deftest restoring-a-file-leaves-the-revision-schedule-alone + (binding [ct/*clock* (ct/fixed-clock test-now)] + (let [[_ file] (create-file 1) + file-id (:id file) + trash-at (ct/in-future {:days 7}) + old-at (ct/in-past {:hours 2}) + + revision-id (insert-revision file-id old-at) + revision-at (:deleted-at (find-change revision-id)) + + snapshot-id (insert-change file-id {:created-at old-at + :created-by "user" + :label "a snapshot" + :deleted-at trash-at}) + + legacy-id (insert-change file-id {:created-at old-at + :data (blob/encode {:legacy true}) + :deleted-at trash-at})] + + (restore-file file-id) + + (t/testing "the revision keeps the schedule it was born with" + (t/is (= revision-at (:deleted-at (find-change revision-id))))) + + (t/testing "the snapshot comes back" + (t/is (nil? (:deleted-at (find-change snapshot-id))))) + + (t/testing "the legacy payload row comes back" + (t/is (nil? (:deleted-at (find-change legacy-id)))))))) + +(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out + (binding [ct/*clock* (ct/fixed-clock test-now)] + (let [[_ file] (create-file 1) + file-id (:id file) + trash-at (ct/in-future {:days 7}) + old-at (ct/in-past {:hours 2}) + system-at (ct/in-future {:days 2}) + + revision-id (insert-revision file-id old-at) + revision-at (:deleted-at (find-change revision-id)) + + snapshot-id (insert-change file-id {:created-at old-at + :created-by "user" + :label "a snapshot"}) + + system-id (insert-change file-id {:created-at old-at + :label "internal/snapshot/1" + :deleted-at system-at})] + + (th/run-task! :delete-object {:object :file + :deleted-at trash-at + :id file-id}) + + (t/testing "the revision keeps the schedule it was born with" + (t/is (= revision-at (:deleted-at (find-change revision-id))))) + + (t/testing "the snapshot takes the file's deletion date" + (t/is (= (:deleted-at (find-file file-id)) + (:deleted-at (find-change snapshot-id))))) + + (t/testing "the system snapshot keeps the date it already had" + (t/is (= system-at (:deleted-at (find-change system-id)))))))) + +(t/deftest updating-a-snapshot-cannot-reach-a-revision + ;; The snapshot paths clear the deletion mark of the row they touch, which is + ;; only legitimate for durable history. They reach a row through + ;; `sql:snapshots`, which filters `label IS NOT NULL`, and a revision never + ;; carries one. This test pins that: the row has to keep the date it was born + ;; with, and the RPC has to reject the id. If a future change loosens that + ;; read filter, this goes red instead of the revision losing its date in + ;; silence. + (binding [ct/*clock* (ct/fixed-clock test-now)] + (let [[profile file] (create-file 1) + file-id (:id file) + revision-id (insert-revision file-id (ct/in-past {:hours 2})) + revision-at (:deleted-at (find-change revision-id)) + + out (th/command! {::th/type :update-file-snapshot + ::rpc/profile-id (:id profile) + :id revision-id + :label "renamed"})] + + (t/testing "the RPC rejects the id" + (t/is (th/ex-info? (:error out))) + (t/is (= :not-found (:type (ex-data (:error out)))))) + + (t/testing "and the revision keeps the date it was born with" + (t/is (= revision-at (:deleted-at (find-change revision-id)))))))) + +(t/deftest a-deleted-then-restored-file-does-not-leak-its-revisions + ;; The lifecycle that leaked 358,517 rows in production: a file is edited, it + ;; ends up in the trash, it comes back, and the revisions that were scheduled + ;; to die an hour after they were written have to die anyway. + (binding [ct/*clock* (ct/fixed-clock test-now)] + (let [[_ file] (create-file 1) + file-id (:id file) + + revision-id (insert-revision file-id (ct/in-past {:hours 2})) + revision-at (:deleted-at (find-change revision-id))] + + (th/run-task! :delete-object {:object :file + :deleted-at (ct/in-future {:days 7}) + :id file-id}) + + (t/testing "deleting the file leaves the revision schedule alone" + (t/is (= revision-at (:deleted-at (find-change revision-id))))) + + (restore-file file-id) + + (t/testing "restoring the file leaves it alone too" + (t/is (= revision-at (:deleted-at (find-change revision-id))))) + + (th/run-task! :objects-gc {}) + + (t/testing "so the collector takes it" + (t/is (nil? (find-change revision-id)))))))