From 9b3436e7a28a287533109839ad2ccae282550d94 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Tejero=20Cantero?= Date: Thu, 13 Aug 2026 07:00:24 +0200 Subject: [PATCH] :books: Describe the op log storage model The technical guide documented the full-copy branch model this change replaces. Update the conceptual model, data model, creation steps, ahead/behind computation, update-from-main (squash), and the limitations section to describe the derived-state model. --- docs/technical-guide/developer/branching.md | 145 ++++++++++++++------ 1 file changed, 100 insertions(+), 45 deletions(-) diff --git a/docs/technical-guide/developer/branching.md b/docs/technical-guide/developer/branching.md index f982683143..6f60d2db8b 100644 --- a/docs/technical-guide/developer/branching.md +++ b/docs/technical-guide/developer/branching.md @@ -19,8 +19,12 @@ the `2.16` API generation (all the RPC methods carry `::doc/added "2.16"`). Branching always reasons about **three** versions of the same file: - * **main** — the source file, the one everybody else keeps editing. - * **branch** — the isolated copy the user works on. + * **branch** — the isolated working state the user edits. A branch is + **not a stored copy**: it is a reference to the merge base plus an + append-only **op log** (`file_branch_change`), one change vector per + branch save. The branch's `:data` is *derived* on every read by + replaying that log over the base. Its file row still exists (the + frontend keys on file ids), but it stores no data payload. * **merge base** — a frozen snapshot of main taken at the exact moment the branch was created. @@ -36,10 +40,13 @@ This is the classic **three-way merge** algorithm, applied at the granularity of individual *entities* (a color, a shape, a typography, a token, …) rather than to the file blob as a whole. -A second, subtler property makes the whole thing possible: a branch is a copy -that **preserves the internal `:data` ids** of every entity. Because "color X" in -the branch and "color X" in main share the same id, the engine can pair them up -and diff them. Without stable ids an id-based merge would be impossible. +A second property makes the whole thing possible: the branch shares the +**internal `:data` ids** of main. Because a branch state starts as +main's state, "color X" in the branch and "color X" in main are the same +entity with the same id, and the engine can pair them up and diff them. +Only the ids created *on* the branch (new shapes, components, media) +carry branch-local ids; `remap-refs` normalizes those before diffing +and merging. Without stable ids an id-based merge would be impossible. ## Where the code lives @@ -49,7 +56,10 @@ The feature spans the three subsystems. The core, shared logic lives in ```text common/src/app/common/files/branch_merge.cljc ; three-way diff + merge->changes engine backend/src/app/rpc/commands/files_branch.clj ; RPC commands (create/list/diff/merge/update) +backend/src/app/binfile/common.clj ; branch :data derivation (get-file* / branch-file-data) +backend/src/app/rpc/commands/files_update.clj ; branch save routing (persist-branch-file!) backend/src/app/migrations/sql/0155-add-file-branch-table.sql ; schema (file_branch table + is_branch) +backend/src/app/migrations/sql/0157-add-file-branch-change-table.sql ; op log (file_branch_change) frontend/src/app/main/data/workspace/branches.cljs ; frontend state + RPC calls (ptk events) frontend/src/app/main/ui/workspace/sidebar/branches.cljs ; panels & dialogs frontend/src/app/main/ui/dashboard/branches_popover.cljs ; dashboard file-card popover @@ -65,12 +75,13 @@ backend/test/backend_tests/rpc_file_branch_test.clj ; integration tests for t The branching commands closely mirror the patterns in `app.rpc.commands.files-snapshot`, and reuse the production change pipeline (`app.common.files.changes/process-changes`), the file snapshot machinery -(`app.features.file-snapshots`) and the file-duplication pipeline -(`app.rpc.commands.management/duplicate-file`). +(`app.features.file-snapshots`), and the file read/save paths +(`app.binfile.common`, `app.rpc.commands.files-update`). ## Data model -The migration adds one column to `file` and one new table. +Two migrations add the branching schema: one column and one table on top of +the existing `file` machinery, plus the op log table. ```sql ALTER TABLE file @@ -105,18 +116,40 @@ CREATE TABLE file_branch ( ); ``` +```sql +CREATE TABLE file_branch_change ( + id uuid PRIMARY KEY, + branch_id uuid NOT NULL REFERENCES file_branch(id) ON DELETE CASCADE DEFERRABLE, + file_id uuid NOT NULL REFERENCES file(id) ON DELETE CASCADE DEFERRABLE, + revn bigint NOT NULL, + changes bytea NOT NULL, + created_at timestamptz NOT NULL DEFAULT clock_timestamp(), + updated_at timestamptz NOT NULL DEFAULT clock_timestamp(), + deleted_at timestamptz NULL, + UNIQUE (file_id, revn) +); +``` + Key points: - * **`is_branch`** on `file` flags the branch copy so it is hidden from the + * **`is_branch`** on `file` flags the branch file so it is hidden from the normal project/team file listings, the dashboard **search**, the team **trash** (and its restore path), and project **duplication/binfile export** — you don't want every branch mixed in with real files, nor restorable zombies without their `file_branch` row. When the *source* file is deleted, the `delete-object` cascade propagates the logical deletion to its branches (they would otherwise be unreachable forever). - * **`branch_file_id`** points to the file row that *is* the branch copy; - **`source_file_id`** points to main. A `UNIQUE` index on `branch_file_id` - guarantees a file can be the branch of at most one source. + * **`branch_file_id`** points to the file row that *is* the branch; the row + carries no data payload, but it anchors permissions, libraries, msgbus + topics and media ownership. **`source_file_id`** points to main. A + `UNIQUE` index on `branch_file_id` guarantees a file can be the branch of + at most one source. + * **`file_branch_change`** is the branch's op log: one row per branch save + (the blob-encoded change vector, `UNIQUE (file_id, revn)`). The branch's + `:data` is derived by replaying its rows in revn order over the base + snapshot. Rows are replaced (squashed) by `update-branch-from-main` and + cascade away when the branch is deleted. + * **`base_snapshot_id`** references the merge-base snapshot. **`base_revn`** records MAIN's revision number and **`base_branch_revn`** the BRANCH file's one at the moment the base was (re)positioned — the two files carry @@ -137,10 +170,10 @@ branch (`:cannot-branch-a-branch`) and runs everything inside a single transaction (with `SET CONSTRAINTS ALL DEFERRED`, because the new file and its branch row reference each other) **under the file's advisory lock** (`db/xact-lock!`, the same one update-file takes), so the base snapshot and -the duplicated copy are guaranteed to capture the SAME state of main — a -concurrent save between the two would otherwise make the base differ from -the branch's initial content and manufacture phantom diffs later: - +the branch's initial state (the empty op log over that base) are guaranteed +to come from the SAME state of main — a concurrent save between the two +would otherwise make the base differ from the branch's initial content and +manufacture phantom diffs later: 1. **Materialize the merge base.** `fsnap/create!` takes a *system* snapshot of main labelled `branch-base/`, with a `deleted-at` 3650 days in the future. The far-future deletion date keeps the snapshot from being pruned by @@ -151,15 +184,17 @@ the branch's initial content and manufacture phantom diffs later: the snapshot can ever not be resolved, merge/update/diff **refuse** with `:base-snapshot-missing` — silently falling back to main would degrade the three-way merge into "branch overwrites main" without conflicts. - 2. **Duplicate main into the branch file.** `mgmt/duplicate-file` is run with - `bfc/*state*` bound to `{:index {file-id branch-id}}` and - `:reset-shared-flag true`. The duplicate pipeline remaps the - file/media/library ids but **preserves the internal `:data` ids** — the - precondition for the id-based merge. - 3. **Mark the new file as a branch** (`is_branch = true`) so it stays out of the - listings. - 4. **Insert the `file_branch` row** with `status = 'open'`, the base snapshot id - and `base_revn = (:revn file)`. + 2. **Create the branch file row.** An empty file row (`ctf/make-file` with + no pages) carries main's project, features (minus the `fdata/*` ones, the + derived data is plain maps) and data version. No data, no media, no + library copy: the read path derives the state, and the op log starts + empty, so the branch's initial content equals the base by construction. + 3. **Mark the new file as a branch** (`is_branch = true`) so it stays out of + the listings, grant the creator ownership (`file_profile_rel`, the same + grant the old duplicate path made), and copy main's `file_library_rel` + rows so shared-library components keep resolving on the derived state. + 4. **Insert the `file_branch` row** with `status = 'open'`, the base snapshot + id and `base_revn = base_branch_revn = (:revn file)`. Before any of this it enforces edition permissions on main and checks two quotas: `::quotes/branches-per-file` and `::quotes/branches-per-team` (configurable via @@ -172,9 +207,11 @@ from a fully-saved state of main. ### Working on a branch -Once created, main and the branch are two live, independent files. When a branch -file is opened, the frontend fetches its context via `:get-file-branch-info` -(events `fetch-branches` / `fetch-branch-context`) and shows a +Once created, main and the branch evolve independently: main keeps its +stored data, the branch keeps an op log. When a branch file is opened, the +backend derives its state (base + replayed ops) before serving it, and the +frontend fetches its context via `:get-file-branch-info` (events +`fetch-branches` / `fetch-branch-context`) and shows a `branch-context-banner*` at the top of the workspace indicating "you are on a branch", together with its *ahead* / *behind* counts and the merge / update actions. @@ -195,13 +232,15 @@ there is a two-level gate (`branch-diff-counts`): the base. * If **both deltas are zero**, the branch is in sync and the diff is skipped entirely (`[0 0 0]`). - * Otherwise main's `:data` is realized **once** and shared across all branches - of the same source. `ahead` is the count of clean changes in the - `:branch->main` direction, `behind` the count in the reverse direction, and - `conflicts` is only non-zero when *both* sides diverged. Only **open** - branches are diffed (merged/archived ones report `[0 0 0]`), and a listing - never fails because of one corrupted branch: count errors degrade to zeros - with a warning (the merge itself still refuses loudly). + * Otherwise main's `:data` is realized **once** and shared across all + branches of the same source, while each diverged branch's state is + **derived** (base snapshot + op log replay) on demand. `ahead` is the + count of clean changes in the `:branch->main` direction, `behind` the + count in the reverse direction, and `conflicts` is only non-zero when + *both* sides diverged. Only **open** branches are diffed (merged/archived + ones report `[0 0 0]`), and a listing never fails because of one corrupted + branch: count errors degrade to zeros with a warning (the merge itself + still refuses loudly). ## The three-way diff engine @@ -425,14 +464,25 @@ changes (plus any conflicts resolved to `:main`) into the branch. It works like a merge with `compute-merge`/`compute-changes` run with the sides swapped, the same `:conflicts` / `:unsupported` gates, and the same safety -snapshot. The one extra step at the end is that it **repositions the merge base**: -the base snapshot is moved forward to the current state of main (releasing the -previous one), and both `base_revn`/`base_branch_revn` are refreshed. When the -branch carried its own changes, `base_branch_revn` is stored one revn short on -purpose: the new base equals MAIN, so the branch still differs from it even if -its revn never moves again — the trick keeps the cheap gate open and the diff -reports the real counts. UI resolutions arrive in main/branch terms and are -inverted server-side (per-attr maps included) before translation. +snapshot. Two extra steps at the end keep the op log sound: + + * it **squashes the op log**: the repositioned base already carries + everything main contributed, so the log is replaced with the net + branch-only changes (`compute-changes` with base = main's current state + and branch = the updated branch state). Replaying those over the new base + reproduces the updated branch exactly, and main-side ops never accumulate + in the log. If that net cannot be translated into replayable ops the + update refuses (`:unsupported-update-squash`), so nothing is persisted; + * it **repositions the merge base**: the base snapshot is moved forward to + the current state of main (releasing the previous one), and both + `base_revn`/`base_branch_revn` are refreshed. When the branch carried its + own changes, `base_branch_revn` is stored one revn short on purpose: the + new base equals MAIN, so the branch still differs from it even if its + revn never moves again — the trick keeps the cheap gate open and the diff + reports the real counts. + +UI resolutions arrive in main/branch terms and are inverted server-side +(per-attr maps included) before translation. ## RPC API summary @@ -441,7 +491,7 @@ All commands live in `app.rpc.commands.files-branch`, are gated by | Command | Kind | Purpose | | ----------------------------- | -------- | --------------------------------------------------- | -| `::create-file-branch` | mutation | Snapshot main + duplicate it into a branch | +| `::create-file-branch` | mutation | Snapshot main + create the branch file and op log | | `::get-file-branches` | query | List branches with ahead/behind/conflicts counts | | `::get-branch-diff` | query | Read-only three-way diff (either direction) | | `::merge-file-branch` | mutation | Integrate a branch into main (resolutions, `expected-main-revn`, `keep-branch`) | @@ -595,3 +645,8 @@ reproduce faithfully rather than risk losing work: merge (the same choice Figma makes). * The merge base snapshot is kept alive for the whole life of the branch; the far-future `deleted-at` is what protects it from the snapshot GC. + * Branch state is **derived** (base snapshot + op log replay) on every read, + so a long-lived branch replays one blob per save; `update-branch-from-main` + squashes the log, but there is no standalone compaction yet. The derived + data also inherits the base's data version, so a branch reads + `:base-snapshot-missing` loudly if its pin was released.