github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-29 14:00:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

📚 Describe the op log storage model

Browse Source 
The technical guide documented the full-copy branch model this change
replaces. Update the conceptual model, data model, creation steps,
ahead/behind computation, update-from-main (squash), and the limitations
section to describe the derived-state model.
This commit is contained in:
Álvaro Tejero Cantero 2026-08-13 07:00:24 +02:00
parent b043fcc76a
commit 9b3436e7a2
No known key found for this signature in database
1 changed files with 100 additions and 45 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

145
docs/technical-guide/developer/branching.md
View File

@ -19,8 +19,12 @@ the `2.16` API generation (all the RPC methods carry `::doc/added "2.16"`).
Branching always reasons about **three** versions of the same file:
* **main** — the source file, the one everybody else keeps editing.
* **branch** — the isolated copy the user works on.
* **branch** — the isolated working state the user edits. A branch is
**not a stored copy**: it is a reference to the merge base plus an
append-only **op log** (`file_branch_change`), one change vector per
branch save. The branch's `:data` is *derived* on every read by
replaying that log over the base. Its file row still exists (the
frontend keys on file ids), but it stores no data payload.
* **merge base** — a frozen snapshot of main taken at the exact moment the
branch was created.
@ -36,10 +40,13 @@ This is the classic **three-way merge** algorithm, applied at the granularity of
individual *entities* (a color, a shape, a typography, a token, …) rather than to
the file blob as a whole.
A second, subtler property makes the whole thing possible: a branch is a copy
that **preserves the internal `:data` ids** of every entity. Because "color X" in
the branch and "color X" in main share the same id, the engine can pair them up
and diff them. Without stable ids an id-based merge would be impossible.
A second property makes the whole thing possible: the branch shares the
**internal `:data` ids** of main. Because a branch state starts as
main's state, "color X" in the branch and "color X" in main are the same
entity with the same id, and the engine can pair them up and diff them.
Only the ids created *on* the branch (new shapes, components, media)
carry branch-local ids; `remap-refs` normalizes those before diffing
and merging. Without stable ids an id-based merge would be impossible.
## Where the code lives
@ -49,7 +56,10 @@ The feature spans the three subsystems. The core, shared logic lives in
```text
common/src/app/common/files/branch_merge.cljc ; three-way diff + merge->changes engine
backend/src/app/rpc/commands/files_branch.clj ; RPC commands (create/list/diff/merge/update)
backend/src/app/binfile/common.clj ; branch :data derivation (get-file* / branch-file-data)
backend/src/app/rpc/commands/files_update.clj ; branch save routing (persist-branch-file!)
backend/src/app/migrations/sql/0155-add-file-branch-table.sql ; schema (file_branch table + is_branch)
backend/src/app/migrations/sql/0157-add-file-branch-change-table.sql ; op log (file_branch_change)
frontend/src/app/main/data/workspace/branches.cljs ; frontend state + RPC calls (ptk events)
frontend/src/app/main/ui/workspace/sidebar/branches.cljs ; panels & dialogs
frontend/src/app/main/ui/dashboard/branches_popover.cljs ; dashboard file-card popover
@ -65,12 +75,13 @@ backend/test/backend_tests/rpc_file_branch_test.clj ; integration tests for t
The branching commands closely mirror the patterns in
`app.rpc.commands.files-snapshot`, and reuse the production change pipeline
(`app.common.files.changes/process-changes`), the file snapshot machinery
(`app.features.file-snapshots`) and the file-duplication pipeline
(`app.rpc.commands.management/duplicate-file`).
(`app.features.file-snapshots`), and the file read/save paths
(`app.binfile.common`, `app.rpc.commands.files-update`).
## Data model
The migration adds one column to `file` and one new table.
Two migrations add the branching schema: one column and one table on top of
the existing `file` machinery, plus the op log table.
```sql
ALTER TABLE file
@ -105,18 +116,40 @@ CREATE TABLE file_branch (
);
```
```sql
CREATE TABLE file_branch_change (
id uuid PRIMARY KEY,
branch_id uuid NOT NULL REFERENCES file_branch(id) ON DELETE CASCADE DEFERRABLE,
file_id uuid NOT NULL REFERENCES file(id) ON DELETE CASCADE DEFERRABLE,
revn bigint NOT NULL,
changes bytea NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
deleted_at timestamptz NULL,
UNIQUE (file_id, revn)
);
```
Key points:
* **`is_branch`** on `file` flags the branch copy so it is hidden from the
* **`is_branch`** on `file` flags the branch file so it is hidden from the
normal project/team file listings, the dashboard **search**, the team
**trash** (and its restore path), and project **duplication/binfile
export** — you don't want every branch mixed in with real files, nor
restorable zombies without their `file_branch` row. When the *source*
file is deleted, the `delete-object` cascade propagates the logical
deletion to its branches (they would otherwise be unreachable forever).
* **`branch_file_id`** points to the file row that *is* the branch copy;
**`source_file_id`** points to main. A `UNIQUE` index on `branch_file_id`
guarantees a file can be the branch of at most one source.
* **`branch_file_id`** points to the file row that *is* the branch; the row
carries no data payload, but it anchors permissions, libraries, msgbus
topics and media ownership. **`source_file_id`** points to main. A
`UNIQUE` index on `branch_file_id` guarantees a file can be the branch of
at most one source.
* **`file_branch_change`** is the branch's op log: one row per branch save
(the blob-encoded change vector, `UNIQUE (file_id, revn)`). The branch's
`:data` is derived by replaying its rows in revn order over the base
snapshot. Rows are replaced (squashed) by `update-branch-from-main` and
cascade away when the branch is deleted.
* **`base_snapshot_id`** references the merge-base snapshot. **`base_revn`**
records MAIN's revision number and **`base_branch_revn`** the BRANCH file's
one at the moment the base was (re)positioned — the two files carry
@ -137,10 +170,10 @@ branch (`:cannot-branch-a-branch`) and runs everything inside a single
transaction (with `SET CONSTRAINTS ALL DEFERRED`, because the new file and
its branch row reference each other) **under the file's advisory lock**
(`db/xact-lock!`, the same one update-file takes), so the base snapshot and
the duplicated copy are guaranteed to capture the SAME state of main — a
concurrent save between the two would otherwise make the base differ from
the branch's initial content and manufacture phantom diffs later:
the branch's initial state (the empty op log over that base) are guaranteed
to come from the SAME state of main — a concurrent save between the two
would otherwise make the base differ from the branch's initial content and
manufacture phantom diffs later:
1. **Materialize the merge base.** `fsnap/create!` takes a *system* snapshot of
main labelled `branch-base/<name>`, with a `deleted-at` 3650 days in the
future. The far-future deletion date keeps the snapshot from being pruned by
@ -151,15 +184,17 @@ the branch's initial content and manufacture phantom diffs later:
the snapshot can ever not be resolved, merge/update/diff **refuse** with
`:base-snapshot-missing` — silently falling back to main would degrade the
three-way merge into "branch overwrites main" without conflicts.
2. **Duplicate main into the branch file.** `mgmt/duplicate-file` is run with
`bfc/*state*` bound to `{:index {file-id branch-id}}` and
`:reset-shared-flag true`. The duplicate pipeline remaps the
file/media/library ids but **preserves the internal `:data` ids** — the
precondition for the id-based merge.
3. **Mark the new file as a branch** (`is_branch = true`) so it stays out of the
listings.
4. **Insert the `file_branch` row** with `status = 'open'`, the base snapshot id
and `base_revn = (:revn file)`.
2. **Create the branch file row.** An empty file row (`ctf/make-file` with
no pages) carries main's project, features (minus the `fdata/*` ones, the
derived data is plain maps) and data version. No data, no media, no
library copy: the read path derives the state, and the op log starts
empty, so the branch's initial content equals the base by construction.
3. **Mark the new file as a branch** (`is_branch = true`) so it stays out of
the listings, grant the creator ownership (`file_profile_rel`, the same
grant the old duplicate path made), and copy main's `file_library_rel`
rows so shared-library components keep resolving on the derived state.
4. **Insert the `file_branch` row** with `status = 'open'`, the base snapshot
id and `base_revn = base_branch_revn = (:revn file)`.
Before any of this it enforces edition permissions on main and checks two quotas:
`::quotes/branches-per-file` and `::quotes/branches-per-team` (configurable via
@ -172,9 +207,11 @@ from a fully-saved state of main.
### Working on a branch
Once created, main and the branch are two live, independent files. When a branch
file is opened, the frontend fetches its context via `:get-file-branch-info`
(events `fetch-branches` / `fetch-branch-context`) and shows a
Once created, main and the branch evolve independently: main keeps its
stored data, the branch keeps an op log. When a branch file is opened, the
backend derives its state (base + replayed ops) before serving it, and the
frontend fetches its context via `:get-file-branch-info` (events
`fetch-branches` / `fetch-branch-context`) and shows a
`branch-context-banner*` at the top of the workspace indicating "you are on a
branch", together with its *ahead* / *behind* counts and the merge / update
actions.
@ -195,13 +232,15 @@ there is a two-level gate (`branch-diff-counts`):
the base.
* If **both deltas are zero**, the branch is in sync and the diff is skipped
entirely (`[0 0 0]`).
* Otherwise main's `:data` is realized **once** and shared across all branches
of the same source. `ahead` is the count of clean changes in the
`:branch->main` direction, `behind` the count in the reverse direction, and
`conflicts` is only non-zero when *both* sides diverged. Only **open**
branches are diffed (merged/archived ones report `[0 0 0]`), and a listing
never fails because of one corrupted branch: count errors degrade to zeros
with a warning (the merge itself still refuses loudly).
* Otherwise main's `:data` is realized **once** and shared across all
branches of the same source, while each diverged branch's state is
**derived** (base snapshot + op log replay) on demand. `ahead` is the
count of clean changes in the `:branch->main` direction, `behind` the
count in the reverse direction, and `conflicts` is only non-zero when
*both* sides diverged. Only **open** branches are diffed (merged/archived
ones report `[0 0 0]`), and a listing never fails because of one corrupted
branch: count errors degrade to zeros with a warning (the merge itself
still refuses loudly).
## The three-way diff engine
@ -425,14 +464,25 @@ changes (plus any conflicts resolved to `:main`) into the branch.
It works like a merge with `compute-merge`/`compute-changes` run with the sides
swapped, the same `:conflicts` / `:unsupported` gates, and the same safety
snapshot. The one extra step at the end is that it **repositions the merge base**:
the base snapshot is moved forward to the current state of main (releasing the
previous one), and both `base_revn`/`base_branch_revn` are refreshed. When the
branch carried its own changes, `base_branch_revn` is stored one revn short on
purpose: the new base equals MAIN, so the branch still differs from it even if
its revn never moves again — the trick keeps the cheap gate open and the diff
reports the real counts. UI resolutions arrive in main/branch terms and are
inverted server-side (per-attr maps included) before translation.
snapshot. Two extra steps at the end keep the op log sound:
* it **squashes the op log**: the repositioned base already carries
everything main contributed, so the log is replaced with the net
branch-only changes (`compute-changes` with base = main's current state
and branch = the updated branch state). Replaying those over the new base
reproduces the updated branch exactly, and main-side ops never accumulate
in the log. If that net cannot be translated into replayable ops the
update refuses (`:unsupported-update-squash`), so nothing is persisted;
* it **repositions the merge base**: the base snapshot is moved forward to
the current state of main (releasing the previous one), and both
`base_revn`/`base_branch_revn` are refreshed. When the branch carried its
own changes, `base_branch_revn` is stored one revn short on purpose: the
new base equals MAIN, so the branch still differs from it even if its
revn never moves again — the trick keeps the cheap gate open and the diff
reports the real counts.
UI resolutions arrive in main/branch terms and are inverted server-side
(per-attr maps included) before translation.
## RPC API summary
@ -441,7 +491,7 @@ All commands live in `app.rpc.commands.files-branch`, are gated by
| Command | Kind | Purpose |
| ----------------------------- | -------- | --------------------------------------------------- |
| `::create-file-branch` | mutation | Snapshot main + duplicate it into a branch |
| `::create-file-branch` | mutation | Snapshot main + create the branch file and op log |
| `::get-file-branches` | query | List branches with ahead/behind/conflicts counts |
| `::get-branch-diff` | query | Read-only three-way diff (either direction) |
| `::merge-file-branch` | mutation | Integrate a branch into main (resolutions, `expected-main-revn`, `keep-branch`) |
@ -595,3 +645,8 @@ reproduce faithfully rather than risk losing work:
merge (the same choice Figma makes).
* The merge base snapshot is kept alive for the whole life of the branch; the
far-future `deleted-at` is what protects it from the snapshot GC.
* Branch state is **derived** (base snapshot + op log replay) on every read,
so a long-lived branch replays one blob per save; `update-branch-from-main`
squashes the log, but there is no standalone compaction yet. The derived
data also inherits the base's data version, so a branch reads
`:base-snapshot-missing` loudly if its pin was released.