mirror of
https://github.com/penpot/penpot.git
synced 2026-11-16 15:11:17 +00:00
* ✨ Reference the file data storage object with a foreign key The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON key. The storage GC parsed it to find live references, and nothing stopped it from physically deleting an object a live row still pointed at: a failure in that scan lost the blob. Migration 0159 moves the pointer to a nullable `storage_object_id` column on `file_data`, backfills it from the metadata key, removes the key from the rows it backfills, indexes the column and rebuilds the foreign key to `storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`. `INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the chunk transaction, while a deferred check would fire at commit, outside it, and the sweep would never move past the blocked chunk. The column is the only source of truth. The `storage-ref-id` entry is gone from the file metadata schema, and the single point that serializes the metadata drops it, so no write can resurrect it. `upsert!` does not query the row: the pointer of the row being replaced travels in its params, and the object is touched before the row is written. Fragments and snapshots pass nil because a modified pointer always gets a new id; the offload passes its own row; `insert-file!` drops it, since an INSERT never replaces a persisted object. `update-file!` takes the file map read from the database plus the fields to change and merges them, so the pointer comes from the read map and no caller can lose it by building a partial map. A map without `:data` raises `:missing-file-data`: it is not `assert`, which is compiled out in the deployed binary. The params cannot override the pointer or retarget the write with `:id`/`:file-id`. `bfc/file->response` strips the pointer from every response (`get-file`, `create-file`, `duplicate-file`, the view-only bundle, the team access request, the comments read, the library summary, the snapshot read and the debug download). A `file-data` object is authentication-only, so a leaked id would let any logged-in profile fetch the blob from `/assets/by-id`. `get-file-fragment` resolves through the file data backend, so a fragment on cold storage no longer hands nil to `blob/decode`, and answers only the keys of its result schema. The storage GC reference scan, the restore union and `objects-gc` read the column. The offload reads `SELECT fd.*`, so its rows go through a `decode-row` that decodes jsonb metadata and drops a JSON null, which the row schema rejected; a row without a readable object raises `:missing-storage-object` with the file and row ids. A chunk blocked by the foreign key (SQLState 23503) is a real anomaly, since the reference scan should have frozen the object: it stays an error with its own hint and the chunk is retried on the next run until the reference is fixed. The release is a coordinated restart, not a rolling deploy, and not rollback-safe for the rows it writes: the previous code writes and scans the metadata pointer, so a key it writes misses the backfill and removing it loses the reference. `operations/2.20.0-file-data-storage-object-id.md` builds the column, the index and the foreign key online and runs the backfill with every instance stopped, then marks the step as applied; a fresh or small instance does the whole job in the migration. The runbook keeps the validation scan out of the stopped window and adds a lock timeout and retry to the online statements. Tests cover the foreign key shape, the backfill and its corrupt and orphan guards, the rebuild of a missing index and of a wrong, non-deferrable or deferred constraint, the deletion refused by a live reference, the metadata cleanup, the offload row normalizations, the touched objects of the unshare and the restore, the response stripping on every call site and the chunk blocked by the foreign key. The backend memories and the developer guide document the pointer, the GC behavior and the two migration paths. Closes #12768 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: inkling-2-small-free * 🐛 Hold the file row lock across the migration write `get-file` applies the pending migrations in memory and, when it finds any, persists the result. That write read the row without a lock and `update-file!` is an unconditional UPDATE, so a save that committed in between was overwritten: the saved content, `revn` and `modified-at` went back, the next save from that client failed with `:revn-conflict` and the object the save left became an orphan the storage GC cannot revisit once it is frozen. The read now runs in a transaction of its own (`get-file` calls `get-file*` inside `db/tx-run!` and accepts a cfg map only), so the `file` row lock covers the re-check and the write. `get-file*` reads the row once (`sql:get-file` plus `resolve-applied-migrations`) and completes it with `attach-file-data`; only when a migration is pending it reads the file again under `FOR UPDATE`, re-checks `need-migration?` on that fresh map and persists it (`migrate-file`). The common case stays at one row read and one migrations resolution, the migration case keeps a single decoded copy in the heap, and a row that changed in between is not overwritten. A migration found on the unlocked read is applied in memory only, for the next read to persist. `migrate-file` returns the reattached map when it persists, with the `:storage-object-id` of the row the write left, and the realized map when it does not; `:realize?` is what returns the read fully plain. `fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead of a boolean, `update-file!` propagates it and the migrating read puts it back in the map it returns, so a caller that persists again touches the current object instead of the replaced one. `insert-file!` keeps nil: an INSERT never replaces a persisted object. The write also compares the stored pointer with the caller's value (`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it affects no row. Every writer holds the `file` row lock, so a pointer that no longer matches is a protocol violation, not a race to retry. The expected pointer is a separate argument because the storage branch replaces `:storage-object-id` before the upsert, and the check lives in `upsert-in-database`: the `legacy-db` branches do not use that SQL and their count can legitimately be zero. The save, snapshot and restore-backup reads take `FOR UPDATE` instead of `FOR SHARE`: they inherit the lock, and the migration branch asks for the exclusive one in the same transaction, so a shared lock that upgrades deadlocks two concurrent readers. A data read rejects `:lock-for-share?` for the same reason; only row-only reads accept it. `sql:get-file-without-data` is a plain select now, not a CTE, which makes locking legal without loading the data. Realization has a single owner: `attach-file-data` only resolves and decodes, and `get-file*` realizes once per path. The applied migrations no longer travel in the `get-file-with-summary` response, cache hit and cache miss alike. `embed-assets` reads each library without `:realize?`, so its `:components` is an unloaded pointer map; `absorb-assets` walks it with `vals`, `PointerMap.seq` calls `load!` and that raised with no loader bound, so every export with `:merge-libraries` of a library with components failed. The absorb runs under the library's own loader now. The in-place binfile import (`import-file-and-overwrite*`, `::overwrite`, `::bfc/file-id`) had no product caller and cannot carry the previous `storage_object_id`, so overwriting a storage-backed row would leave its object unreclaimed. `import-files!` rejects a cfg with `::bfc/file-id` (`:not-supported`) and the helpers only that path used go with it (`invalidate-thumbnails`, `*reference-file*`, the `::overwrite` id branch of `index-object` and of `save-file!`, the `on-conflict-do-nothing?` flag of the media insert). Tests: a new `file-migration-read-test` pins the single read, the locked re-read, the concurrent save interleaving, the double migration and the read-only path; the upsert tests cover the return shape and the stale pointer; the SQL recorder they share lives in `backend-tests.helpers`, and the save, snapshot, restore and summary sequences follow the new reads. The backend memories document the read contract and the pointer. AI-assisted-by: deepseek-v4.1-flash * 🐛 Harden the get-file guard and skip its read-only transaction The cfg guard of `get-file` was an `assert`, which `app.config` elides when `:backend-asserts` is off, so a deployed binary accepted a bare pool or connection while the docstring promised it raises. It throws `IllegalArgumentException` now, the class the option guards of the same namespace already use, and the existing test expects it instead of `AssertionError`. A read with `:read-only?` applies the pending migrations in memory and writes nothing, so the transaction the migration write needs only costs two round trips there. `get-file` runs it with `db/run!` instead: with a cfg that carries a connection it still joins the caller's transaction, and only the reads that would have opened their own skip it. A lock option needs a read-write transaction from the caller, because PostgreSQL rejects row locks in a read-only one; the docstring states that. Tests: the guard test expects the new class; a cfg marked `::db/read-only` reads a pending migration without persisting it (the config flag that fills that key is legacy, but the key is set by hand); and a read-only read is pinned as not opening a transaction while the same call without the option does. The backend memory documents the split and the guard. AI-assisted-by: deepseek-v4.1-flash
792 lines
33 KiB
Clojure
792 lines
33 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
|
|
(ns backend-tests.rpc-management-test
|
|
(:require
|
|
[app.binfile.common :as bfc]
|
|
[app.common.features :as cfeat]
|
|
[app.common.pprint :as pp]
|
|
[app.common.time :as ct]
|
|
[app.common.types.shape :as cts]
|
|
[app.common.uuid :as uuid]
|
|
[app.config :as cf]
|
|
[app.db :as db]
|
|
[app.http :as http]
|
|
[app.rpc :as-alias rpc]
|
|
[app.storage :as sto]
|
|
[backend-tests.helpers :as th]
|
|
[backend-tests.storage-test :refer [configure-storage-backend]]
|
|
[buddy.core.bytes :as b]
|
|
[clojure.test :as t]
|
|
[cuerdas.core :as str]
|
|
[datoteka.fs :as fs]
|
|
[datoteka.io :as io]
|
|
[mockery.core :refer [with-mocks]]))
|
|
|
|
(t/use-fixtures :once th/state-init)
|
|
(t/use-fixtures :each th/database-reset)
|
|
|
|
(defn- update-file!
|
|
[& {:keys [profile-id file-id changes revn] :or {revn 0}}]
|
|
(let [params {::th/type :update-file
|
|
::rpc/profile-id profile-id
|
|
:id file-id
|
|
:session-id (uuid/random)
|
|
:revn revn
|
|
:vern 0
|
|
:features cfeat/supported-features
|
|
:changes changes}
|
|
out (th/command! params)]
|
|
;; (th/print-result! out)
|
|
(t/is (nil? (:error out)))
|
|
(:result out)))
|
|
|
|
(t/deftest upload-tempfile-returns-fresh-object-for-same-content
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
path (fs/create-tempfile :dir "/tmp/penpot" :prefix "test-upload-tempfile-")
|
|
_ (io/write* path "content")
|
|
params {::th/type :upload-tempfile
|
|
::rpc/profile-id (:id profile)
|
|
:content {:filename "export.png"
|
|
:path path
|
|
:mtype "image/png"
|
|
:size 7}}
|
|
config (assoc cf/config :public-uri "https://example.com/penpot")
|
|
out1 (binding [cf/config config]
|
|
(th/multipart-management-command! params))
|
|
out2 (binding [cf/config config]
|
|
(th/multipart-management-command! params))]
|
|
(t/is (nil? (:error out1)))
|
|
(t/is (nil? (:error out2)))
|
|
(t/is (str/starts-with? (str (get-in out1 [:result :uri]))
|
|
"https://example.com/penpot/assets/by-id/"))
|
|
(t/is (not= (get-in out1 [:result :id])
|
|
(get-in out2 [:result :id])))))
|
|
|
|
(t/deftest upload-tempfile-rejects-html-content-type
|
|
;; N2-13: upload-tempfile must reject non-allowed content types
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
path (fs/create-tempfile :dir "/tmp/penpot" :prefix "test-upload-tempfile-")
|
|
_ (io/write* path "<script>alert(1)</script>")
|
|
params {::th/type :upload-tempfile
|
|
::rpc/profile-id (:id profile)
|
|
:content {:filename "evil.html"
|
|
:path path
|
|
:mtype "text/html"
|
|
:size 27}}
|
|
out (th/multipart-management-command! params)]
|
|
(t/is (some? (:error out)))
|
|
(t/is (= :validation (th/ex-type (:error out))))
|
|
(t/is (= :media-type-not-allowed (th/ex-code (:error out))))))
|
|
|
|
(t/deftest upload-tempfile-rejects-a-key-it-does-not-declare
|
|
(let [out (th/management-command! {::th/type :upload-tempfile
|
|
::rpc/auth-key-id :nexus})]
|
|
(t/is (some? (:error out)))
|
|
(t/is (= :authorization (th/ex-type (:error out))))
|
|
(t/is (= :shared-key-not-allowed (th/ex-code (:error out))))))
|
|
|
|
(t/deftest upload-tempfile-accepts-both-declared-keys
|
|
(let [profile (th/create-profile* 1 {:is-active true})]
|
|
(doseq [key-id [:exporter :media-processor]]
|
|
(let [path (fs/create-tempfile :dir "/tmp/penpot"
|
|
:prefix (str "test-upload-" (name key-id) "-"))
|
|
_ (io/write* path "content")
|
|
out (th/multipart-management-command!
|
|
{::th/type :upload-tempfile
|
|
::rpc/auth-key-id key-id
|
|
::rpc/profile-id (:id profile)
|
|
:content {:filename "export.png"
|
|
:path path
|
|
:mtype "image/png"
|
|
:size 7}})]
|
|
(t/is (nil? (:error out)) (name key-id))))))
|
|
|
|
(t/deftest duplicate-file
|
|
(let [storage (-> (:app.storage/storage th/*system*)
|
|
(configure-storage-backend))
|
|
|
|
sobject (sto/put-object! storage {::sto/content (sto/content "content")
|
|
:content-type "text/plain"})
|
|
profile (th/create-profile* 1 {:is-active true})
|
|
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:is-shared true})
|
|
|
|
libl (th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
mobj (th/create-file-media-object* {:file-id (:id file1)
|
|
:is-local false
|
|
:media-id (:id sobject)})]
|
|
(update-file!
|
|
:file-id (:id file1)
|
|
:profile-id (:id profile)
|
|
:revn 0
|
|
:vern 0
|
|
:changes
|
|
[{:type :add-media
|
|
:object mobj}])
|
|
|
|
(let [data {::th/type :duplicate-file
|
|
::rpc/profile-id (:id profile)
|
|
:file-id (:id file1)
|
|
:name "file 1 (copy)"}
|
|
out (th/command! data)]
|
|
|
|
;; (th/print-result! out)
|
|
|
|
;; Check that result is correct
|
|
(t/is (nil? (:error out)))
|
|
(let [result (:result out)]
|
|
;; Check that the returned result is a file but has different id
|
|
;; and different name.
|
|
(t/is (= "file 1 (copy)" (:name result)))
|
|
(t/is (not= (:id file1) (:id result)))
|
|
|
|
;; Check that the new file has a correct file library relation
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id result)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:id file2) (:library-file-id item))))
|
|
|
|
;; Check that the new file has a correct file media objects
|
|
(let [[item :as rows] (th/db-query :file-media-object {:file-id (:id result)})]
|
|
(t/is (= 1 (count rows)))
|
|
|
|
;; Check that both items have different ids
|
|
(t/is (not= (:id item) (:id mobj)))
|
|
|
|
;; check that both file-media-objects points to the same storage object.
|
|
(t/is (= (:media-id item) (:media-id mobj)))
|
|
(t/is (= (:media-id item) (:id sobject)))
|
|
|
|
;; Check if media correctly contains the new file-media-object id
|
|
(t/is (contains? (get-in result [:data :media]) (:id item)))
|
|
|
|
;; And does not contains the old one
|
|
(t/is (not (contains? (get-in result [:data :media]) (:id mobj)))))
|
|
|
|
;; Check the total number of files
|
|
(let [rows (th/db-query :file {:project-id (:id project)})]
|
|
(t/is (= 3 (count rows))))))))
|
|
|
|
(t/deftest duplicate-file-does-not-touch-the-source-storage-object
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project)})]
|
|
|
|
(th/run-task! :offload-file-data {:file-id (:id file1)})
|
|
(th/run-task! :storage-gc-touched {:skip-delay true})
|
|
|
|
(let [sobject (:storage-object-id (th/db-get :file-data {:file-id (:id file1)
|
|
:id (:id file1)}))]
|
|
(t/is (uuid? sobject))
|
|
(t/is (nil? (:touched-at (th/db-get :storage-object {:id sobject}
|
|
{::db/remove-deleted false}))))
|
|
|
|
(let [out (th/command! {::th/type :duplicate-file
|
|
::rpc/profile-id (:id profile)
|
|
:file-id (:id file1)})]
|
|
(t/is (nil? (:error out)))
|
|
(t/is (not (contains? (:result out) :storage-object-id)))
|
|
|
|
(t/testing "duplication does not write to the source object"
|
|
(t/is (nil? (:touched-at (th/db-get :storage-object {:id sobject}
|
|
{::db/remove-deleted false})))))
|
|
|
|
(let [file-id (:id (:result out))
|
|
row (th/db-get :file-data {:file-id file-id :id file-id})]
|
|
(t/testing "the new file does not inherit the source pointer"
|
|
(t/is (= "db" (:backend row)))
|
|
(t/is (nil? (:storage-object-id row)))
|
|
(t/is (bytes? (:data row)))))))))
|
|
|
|
(t/deftest duplicate-file-with-deleted-relations
|
|
(let [storage (-> (:app.storage/storage th/*system*)
|
|
(configure-storage-backend))
|
|
sobject (sto/put-object! storage {::sto/content (sto/content "content")
|
|
:content-type "text/plain"})
|
|
profile (th/create-profile* 1 {:is-active true})
|
|
|
|
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:is-shared true})
|
|
|
|
libl (th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
mobj (th/create-file-media-object* {:file-id (:id file1)
|
|
:is-local false
|
|
:media-id (:id sobject)})]
|
|
|
|
(th/mark-file-deleted* {:id (:id file2)})
|
|
(sto/del-object! storage (:id sobject))
|
|
|
|
(let [data {::th/type :duplicate-file
|
|
::rpc/profile-id (:id profile)
|
|
:file-id (:id file1)
|
|
:name "file 1 (copy)"}
|
|
out (th/command! data)]
|
|
|
|
;; (th/print-result! out)
|
|
|
|
;; Check that result is correct
|
|
(t/is (nil? (:error out)))
|
|
(let [result (:result out)]
|
|
|
|
;; Check that the returned result is a file but has different id
|
|
;; and different name.
|
|
(t/is (= "file 1 (copy)" (:name result)))
|
|
(t/is (not= (:id file1) (:id result)))
|
|
|
|
;; Check that there are no relation to a deleted library
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id result)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; Check that the new file has no media objects
|
|
(let [[item :as rows] (th/db-query :file-media-object {:file-id (:id result)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; Check the total number of files
|
|
(let [rows (th/db-query :file {:project-id (:id project)})]
|
|
(t/is (= 3 (count rows))))))))
|
|
|
|
(t/deftest duplicate-project
|
|
(let [storage (-> (:app.storage/storage th/*system*)
|
|
(configure-storage-backend))
|
|
|
|
sobject (sto/put-object! storage {::sto/content (sto/content "content")
|
|
:content-type "text/plain"})
|
|
|
|
profile (th/create-profile* 1 {:is-active true})
|
|
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:is-shared true})
|
|
|
|
libl (th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
mobj (th/create-file-media-object* {:file-id (:id file1)
|
|
:is-local false
|
|
:media-id (:id sobject)})]
|
|
|
|
(update-file!
|
|
:file-id (:id file1)
|
|
:profile-id (:id profile)
|
|
:revn 0
|
|
:vern 0
|
|
:changes
|
|
[{:type :add-media
|
|
:object mobj}])
|
|
|
|
(let [data {::th/type :duplicate-project
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:name "project 1 (copy)"}
|
|
out (th/command! data)]
|
|
|
|
;; Check that result is correct
|
|
(t/is (nil? (:error out)))
|
|
|
|
(let [result (:result out)]
|
|
;; Check that they are the same project but different id and name
|
|
(t/is (= "project 1 (copy)" (:name result)))
|
|
(t/is (not= (:id project) (:id result)))
|
|
|
|
;; Check the total number of projects (previously is 2, now is 3)
|
|
(let [rows (th/db-query :project {:team-id (:default-team-id profile)})]
|
|
(t/is (= 3 (count rows))))
|
|
|
|
;; Check that the new project has the same files
|
|
(let [p1-files (th/db-query :file
|
|
{:project-id (:id project)}
|
|
{:order-by [:name]})
|
|
p2-files (th/db-query :file
|
|
{:project-id (:id result)}
|
|
{:order-by [:name]})]
|
|
(t/is (= (count p1-files)
|
|
(count p2-files)))
|
|
|
|
;; check that the both files are equivalent
|
|
(doseq [[fa fb] (map vector p1-files p2-files)]
|
|
(t/is (not= (:id fa) (:id fb)))
|
|
(t/is (= (:name fa) (:name fb)))))))))
|
|
|
|
(t/deftest duplicate-project-with-deleted-files
|
|
(let [storage (-> (:app.storage/storage th/*system*)
|
|
(configure-storage-backend))
|
|
sobject (sto/put-object! storage {::sto/content (sto/content "content")
|
|
:content-type "text/plain"})
|
|
profile (th/create-profile* 1 {:is-active true})
|
|
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:is-shared true})
|
|
|
|
libl (th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
mobj (th/create-file-media-object* {:file-id (:id file1)
|
|
:is-local false
|
|
:media-id (:id sobject)})]
|
|
|
|
(th/mark-file-deleted* {:id (:id file1)})
|
|
|
|
(let [data {::th/type :duplicate-project
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:name "project 1 (copy)"}
|
|
out (th/command! data)]
|
|
|
|
;; Check that result is correct
|
|
(t/is (nil? (:error out)))
|
|
|
|
(let [result (:result out)]
|
|
;; Check that they are the same project but different id and name
|
|
(t/is (= "project 1 (copy)" (:name result)))
|
|
(t/is (not= (:id project) (:id result)))
|
|
|
|
;; Check the total number of projects (previously is 2, now is 3)
|
|
(let [rows (th/db-query :project {:team-id (:default-team-id profile)})]
|
|
(t/is (= 3 (count rows))))
|
|
|
|
;; Check that the new project has only the second file
|
|
(let [p1-files (th/db-query :file
|
|
{:project-id (:id project)}
|
|
{:order-by [:name]})
|
|
p2-files (th/db-query :file
|
|
{:project-id (:id result)}
|
|
{:order-by [:name]})]
|
|
(t/is (= (count (rest p1-files))
|
|
(count p2-files)))
|
|
|
|
;; check that the both files are equivalent
|
|
(doseq [[fa fb] (map vector (rest p1-files) p2-files)]
|
|
(t/is (not= (:id fa) (:id fb)))
|
|
(t/is (= (:name fa) (:name fb)))))))))
|
|
|
|
(t/deftest move-file-on-same-team
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 1 {:profile-id (:id profile)})
|
|
|
|
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
project2 (th/create-project* 2 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project1)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:is-shared true})]
|
|
|
|
(th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
;; Try to move to same project
|
|
(let [data {::th/type :move-files
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:ids #{(:id file1)}}
|
|
|
|
out (th/command! data)
|
|
error (:error out)]
|
|
(t/is (th/ex-info? error))
|
|
(t/is (th/ex-of-type? error :validation))
|
|
(t/is (th/ex-of-code? error :cant-move-to-same-project)))
|
|
|
|
;; initially project1 should have 2 files
|
|
(let [rows (th/db-query :file {:project-id (:id project1)})]
|
|
(t/is (= 2 (count rows))))
|
|
|
|
;; initially project2 should be empty
|
|
(let [rows (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; move a file1 to project2 (in the same team)
|
|
(let [data {::th/type :move-files
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project2)
|
|
:ids #{(:id file1)}}
|
|
|
|
out (th/command! data)]
|
|
|
|
(t/is (nil? (:error out)))
|
|
(t/is (nil? (:result out)))
|
|
|
|
;; project1 now should contain 1 file
|
|
(let [rows (th/db-query :file {:project-id (:id project1)})]
|
|
(t/is (= 1 (count rows))))
|
|
|
|
;; project2 now should contain 1 file
|
|
(let [rows (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 1 (count rows))))
|
|
|
|
;; file1 should be still linked to file2
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id file1)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item) (:id file1)))
|
|
(t/is (= (:library-file-id item) (:id file2))))
|
|
|
|
;; should be no libraries on file2
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (= 0 (count rows)))))))
|
|
|
|
|
|
;; TODO: move a library to other team
|
|
(t/deftest move-file-to-other-team
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 1 {:profile-id (:id profile)})
|
|
|
|
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
project2 (th/create-project* 2 {:team-id (:id team)
|
|
:profile-id (:id profile)})
|
|
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project1)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:is-shared true})
|
|
file3 (th/create-file* 3 {:profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:is-shared true})]
|
|
|
|
(th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
(th/link-file-to-library* {:file-id (:id file2)
|
|
:library-id (:id file3)})
|
|
|
|
;; --- initial data checks
|
|
|
|
;; the project1 should have 3 files
|
|
(let [rows (th/db-query :file {:project-id (:id project1)})]
|
|
(t/is (= 3 (count rows))))
|
|
|
|
;; should be no files on project2
|
|
(let [rows (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; the file1 should be linked to file2
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id file1)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item) (:id file1)))
|
|
(t/is (= (:library-file-id item) (:id file2))))
|
|
|
|
;; the file2 should be linked to file3
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item) (:id file2)))
|
|
(t/is (= (:library-file-id item) (:id file3))))
|
|
|
|
;; should be no libraries on file3
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file3)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; move to other project in other team
|
|
(let [data {::th/type :move-files
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project2)
|
|
:ids #{(:id file1)}}
|
|
out (th/command! data)]
|
|
|
|
(t/is (nil? (:error out)))
|
|
(t/is (nil? (:result out)))
|
|
|
|
;; project1 now should have 2 file
|
|
(let [[item1 item2 :as rows] (th/db-query :file {:project-id (:id project1)}
|
|
{:order-by [:created-at]})]
|
|
;; (clojure.pprint/pprint rows)
|
|
(t/is (= 2 (count rows)))
|
|
(t/is (= (:id item1) (:id file2))))
|
|
|
|
;; project2 now should have 1 file
|
|
(let [[item :as rows] (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:id item) (:id file1))))
|
|
|
|
;; the moved file1 should not have any link to libraries
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file1)})]
|
|
(t/is (zero? (count rows))))
|
|
|
|
;; the file2 should still be linked to file3
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item) (:id file2)))
|
|
(t/is (= (:library-file-id item) (:id file3)))))))
|
|
|
|
|
|
(t/deftest move-library-to-other-team
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 1 {:profile-id (:id profile)})
|
|
|
|
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
project2 (th/create-project* 2 {:team-id (:id team)
|
|
:profile-id (:id profile)})
|
|
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project1)})
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:is-shared true})]
|
|
|
|
(th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
;; --- initial data checks
|
|
|
|
;; the project1 should have 2 files
|
|
(let [rows (th/db-query :file {:project-id (:id project1)})]
|
|
(t/is (= 2 (count rows))))
|
|
|
|
;; should be no files on project2
|
|
(let [rows (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; the file1 should be linked to file2
|
|
(let [[item :as rows] (th/db-query :file-library-rel {:file-id (:id file1)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item) (:id file1)))
|
|
(t/is (= (:library-file-id item) (:id file2))))
|
|
|
|
;; should be no libraries on file2
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; move the library to other project
|
|
(let [data {::th/type :move-files
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project2)
|
|
:ids #{(:id file2)}}
|
|
out (th/command! data)]
|
|
|
|
(t/is (nil? (:error out)))
|
|
(t/is (nil? (:result out)))
|
|
|
|
;; project1 now should have 1 file
|
|
(let [[item :as rows] (th/db-query :file {:project-id (:id project1)}
|
|
{:order-by [:created-at]})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:id item) (:id file1))))
|
|
|
|
;; project2 now should have 1 file
|
|
(let [[item :as rows] (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:id item) (:id file2))))
|
|
|
|
;; the file1 should not have any link to libraries
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file1)})]
|
|
(t/is (zero? (count rows))))
|
|
|
|
;; the file2 should not have any link to libraries
|
|
(let [rows (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (zero? (count rows)))))))
|
|
|
|
(t/deftest move-project
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 1 {:profile-id (:id profile)})
|
|
|
|
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
project2 (th/create-project* 2 {:team-id (:default-team-id profile)
|
|
:profile-id (:id profile)})
|
|
|
|
file1 (th/create-file* 1 {:profile-id (:id profile)
|
|
:project-id (:id project1)})
|
|
|
|
file2 (th/create-file* 2 {:profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:is-shared true})
|
|
|
|
file3 (th/create-file* 3 {:profile-id (:id profile)
|
|
:project-id (:id project2)
|
|
:is-shared true})]
|
|
|
|
(th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file2)})
|
|
|
|
(th/link-file-to-library* {:file-id (:id file1)
|
|
:library-id (:id file3)})
|
|
|
|
;; --- initial data checks
|
|
|
|
;; the project1 should have 2 files
|
|
(let [rows (th/db-query :file {:project-id (:id project1)})]
|
|
(t/is (= 2 (count rows))))
|
|
|
|
;; the project2 should have 1 file
|
|
(let [rows (th/db-query :file {:project-id (:id project2)})]
|
|
(t/is (= 1 (count rows))))
|
|
|
|
;; the file1 should be linked to file2 and file3
|
|
(let [[item1 item2 :as rows] (th/db-query :file-library-rel
|
|
{:file-id (:id file1)}
|
|
{:order-by [:created-at]})]
|
|
(t/is (= 2 (count rows)))
|
|
(t/is (= (:file-id item1) (:id file1)))
|
|
(t/is (= (:library-file-id item1) (:id file2)))
|
|
(t/is (= (:file-id item2) (:id file1)))
|
|
(t/is (= (:library-file-id item2) (:id file3))))
|
|
|
|
;; the file2 should not be linked to any file
|
|
(let [[rows] (th/db-query :file-library-rel {:file-id (:id file2)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; the file3 should not be linked to any file
|
|
(let [[rows] (th/db-query :file-library-rel {:file-id (:id file3)})]
|
|
(t/is (= 0 (count rows))))
|
|
|
|
;; move project1 to other team
|
|
(let [data {::th/type :move-project
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project1)
|
|
:team-id (:id team)}
|
|
out (th/command! data)]
|
|
|
|
(t/is (nil? (:error out)))
|
|
(t/is (nil? (:result out)))
|
|
|
|
;; project1 now should still have 2 files
|
|
(let [[item1 item2 :as rows] (th/db-query :file
|
|
{:project-id (:id project1)}
|
|
{:order-by [:created-at]})]
|
|
;; (clojure.pprint/pprint rows)
|
|
(t/is (= 2 (count rows)))
|
|
(t/is (= (:id item1) (:id file1)))
|
|
(t/is (= (:id item2) (:id file2))))
|
|
|
|
;; project2 now should still have 1 file
|
|
(let [[item :as rows] (th/db-query :file {:project-id (:id project2)})]
|
|
;; (pp/pprint rows)
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:id item) (:id file3))))
|
|
|
|
;; the file1 should be linked to file2 but not file3
|
|
(let [[item1 :as rows] (th/db-query :file-library-rel
|
|
{:file-id (:id file1)}
|
|
{:order-by [:created-at]})]
|
|
|
|
(t/is (= 1 (count rows)))
|
|
(t/is (= (:file-id item1) (:id file1)))
|
|
(t/is (= (:library-file-id item1) (:id file2)))))))
|
|
|
|
(t/deftest clone-template
|
|
(let [prof (th/create-profile* 1 {:is-active true})
|
|
data {::th/type :clone-template
|
|
::rpc/profile-id (:id prof)
|
|
:project-id (:default-project-id prof)
|
|
:template-id "test"}
|
|
|
|
out (th/command! data)]
|
|
;; (th/print-result! out)
|
|
|
|
(t/is (nil? (:error out)))
|
|
(let [result (:result out)]
|
|
(t/is (fn? result))
|
|
|
|
(let [events (th/consume-sse result)]
|
|
(t/is (= 5 (count events)))
|
|
(t/is (= :end (first (last events))))))))
|
|
|
|
(t/deftest get-list-of-buitin-templates
|
|
(let [prof (th/create-profile* 1 {:is-active true})
|
|
data {::th/type :get-builtin-templates
|
|
::rpc/profile-id (:id prof)}
|
|
out (th/command! data)]
|
|
;; (th/print-result! out)
|
|
(t/is (nil? (:error out)))
|
|
(let [result (:result out)]
|
|
(t/is (vector? result))
|
|
(t/is (= 1 (count result)))
|
|
(t/is (= "test" (:id (first result)))))))
|
|
|
|
(t/deftest clone-template-audit-events-carry-team-id
|
|
;; The per-file `create-file` events are built by hand, so they do not pass
|
|
;; through the generic resolver: the command must copy the team id itself.
|
|
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
|
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
|
(let [prof (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 1 {:profile-id (:id prof)})
|
|
project (th/create-project* 1 {:profile-id (:id prof)
|
|
:team-id (:id team)})
|
|
out (th/command! {::th/type :clone-template
|
|
::rpc/profile-id (:id prof)
|
|
:project-id (:id project)
|
|
:template-id "test"})]
|
|
(t/is (th/success? out))
|
|
(th/consume-sse (:result out))
|
|
(let [events (->> (:call-args-list @audit-mock)
|
|
(map second)
|
|
(filter #(= "create-file" (:name %))))]
|
|
(t/is (seq events))
|
|
(t/is (every? #(= (:id team) (get-in % [:props :team-id])) events)))))))
|
|
|
|
(t/deftest duplicate-project-audit-events-carry-entity-ids
|
|
;; The duplicated project and files are created by internal calls, so the
|
|
;; command emits one create event per entity for a local projection.
|
|
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
|
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
team (th/create-team* 52 {:profile-id (:id profile)})
|
|
project (th/create-project* 52 {:profile-id (:id profile)
|
|
:team-id (:id team)})
|
|
_file (th/create-file* 52 {:profile-id (:id profile)
|
|
:project-id (:id project)})
|
|
out (th/command! {::th/type :duplicate-project
|
|
::rpc/profile-id (:id profile)
|
|
:project-id (:id project)
|
|
:name "copy"})
|
|
events (->> (:call-args-list @audit-mock)
|
|
(map second)
|
|
(filter #(contains? #{"create-project" "create-file"} (:name %))))]
|
|
(t/is (th/success? out))
|
|
(t/is (some #(and (= "create-project" (:name %))
|
|
(= (get-in out [:result :id]) (get-in % [:props :id]))
|
|
(= (:id team) (get-in % [:props :team-id]))
|
|
(= "copy" (get-in % [:props :name])))
|
|
events))
|
|
(t/is (some #(and (= "create-file" (:name %))
|
|
(string? (get-in % [:props :name]))
|
|
(int? (get-in % [:props :revn]))
|
|
(ct/inst? (get-in % [:props :modified-at])))
|
|
events))
|
|
(t/is (every? #(= (:id team) (get-in % [:props :team-id])) events))))))
|
|
|
|
(t/deftest upload-tempfile-rejects-forged-content
|
|
;; Same transit-forgery vector through the management API: without a
|
|
;; multipart request the wrapper must reject it at params validation.
|
|
(let [profile (th/create-profile* 1 {:is-active true})
|
|
params {::th/type :upload-tempfile
|
|
::rpc/profile-id (:id profile)
|
|
:content {:filename "evil.png"
|
|
:path (fs/path "/etc/passwd")
|
|
:mtype "image/png"
|
|
:size 123}}
|
|
out (th/management-command! params)]
|
|
(t/is (th/ex-of-code? (:error out) :params-validation))))
|