diff --git a/performance/README.md b/performance/README.md new file mode 100644 index 0000000000..ebcbbbb4b1 --- /dev/null +++ b/performance/README.md @@ -0,0 +1,102 @@ +# Penpot Performance Tests + +k6-based load and performance test suite for the Penpot backend. Measures HTTP RPC latency, throughput, and error rates under synthetic user load. + +## Prerequisites + +- **k6** — Install from https://k6.io/docs/get-started/installation/ +- **Running Penpot backend** — Local devenv (`http://localhost:6060`) or a remote instance + +## Quick Start + +```bash +# Smoke test — 1 VU, 1 iteration, demo mode +./run.sh smoke + +# Full lifecycle with 10 VUs, 5 iterations each +./run.sh lifecycle -v 10 -n 5 + +# Use registration flow instead of demo profiles +./run.sh lifecycle -m register -v 5 -n 1 + +# Point to a remote backend +./run.sh lifecycle -u https://penpot.example.com + +# Show all options +./run.sh help +``` + +## Test Scripts + +### `scripts/lifecycle.js` — Full User Lifecycle + +Simulates a realistic user journey from account creation through CRUD operations: + +1. **Register** — Create a new user (demo profile or full registration) +2. **Login** — Authenticate and obtain session cookie +3. **Get Profile** — Fetch current user profile +4. **Get Teams** — List user teams +5. **Create Project** — Create a new project in the default team +6. **Create File** — Create a new design file in the project +7. **Get File** — Fetch the file with its data (pages, objects) +8. **Update File** — Add a rectangle shape (tests optimistic concurrency) +9. **Upload Image** — Upload a PNG to the file's media objects +10. **Delete File** — Remove the file +11. **Delete Project** — Remove the project +12. **Logout** — End the session + +Each VU performs the full flow independently, creating and cleaning up its own artifacts. + +## Configuration + +Options for `run.sh lifecycle`: + +| Flag | Env Variable | Default | Description | +|------|-------------|---------|-------------| +| `-u URL` | `PENPOT_BASE_URL` | `http://localhost:6060` | Penpot backend URL | +| `-v NUM` | — | `1` | Number of virtual users | +| `-n NUM` | — | `1` | Iterations per VU | +| `-m MODE` | `PENPOT_REGISTER_MODE` | `demo` | `demo` or `register` | +| `-k PATH` | `K6` | `k6` | Path to k6 binary | + +### Register Modes + +- **`demo`** (default): Uses the `create-demo-profile` RPC endpoint. Requires the `demo-users` feature flag to be enabled on the backend. Fastest for testing. +- **`register`**: Uses the full two-step registration flow (`prepare-register-profile` + `register-profile`). Works without any feature flags but is slower. + +## Shared Client (`lib/penpot-client.js`) + +The shared client module wraps the Penpot backend RPC API using plain JSON (not Transit). Key features: + +- **JSON transport**: Uses `Content-Type: application/json` for POST bodies and `Accept: application/json` (or `_fmt=json` for GET) for responses. +- **Cookie-based auth**: k6 automatically manages session cookies per VU. +- **Session headers**: Generates `x-session-id` and `x-external-session-id` UUIDs per VU. +- **Tagged metrics**: Every request is tagged with `rpc_command` for k6 metric slicing. + +## Results + +Test results are written to `results//` as JSON. k6 also prints a summary to stdout with percentile breakdowns per RPC command. + +## Thresholds + +The lifecycle script includes built-in thresholds that will cause k6 to exit with a non-zero code if exceeded: + +- `http_req_duration p95 < 5000ms` (global) +- `http_req_failed < 1%` (global) +- Per-command thresholds for login, profile, project, file, and update operations + +## Adding New Flows + +To add a new test flow: + +1. Create `scripts/.js` +2. Import the shared client: `import { createClient } from "../lib/penpot-client.js";` +3. Implement the flow using the client methods +4. Add a command in `run.sh` + +## Architecture Notes + +- The backend supports both Transit JSON and plain JSON. This test suite uses **plain JSON** for simplicity (no Transit encoder needed in k6). +- JSON request keys are in **kebab-case** (matching Clojure conventions). JSON response keys are in **camelCase** (backend's default JSON encoding). +- `update-file` sends the `id` parameter both in the query string and in the POST body, matching the frontend's behavior. +- The backend uses optimistic concurrency control (`revn`) for file updates. The test retries once on conflict. diff --git a/performance/fixtures/test-large.png b/performance/fixtures/test-large.png new file mode 100644 index 0000000000..fbb82a92ac Binary files /dev/null and b/performance/fixtures/test-large.png differ diff --git a/performance/fixtures/test-medium.png b/performance/fixtures/test-medium.png new file mode 100644 index 0000000000..0094599ffe Binary files /dev/null and b/performance/fixtures/test-medium.png differ diff --git a/performance/fixtures/test-small.png b/performance/fixtures/test-small.png new file mode 100644 index 0000000000..5391345596 Binary files /dev/null and b/performance/fixtures/test-small.png differ diff --git a/performance/lib/penpot-client.js b/performance/lib/penpot-client.js new file mode 100644 index 0000000000..18df142255 --- /dev/null +++ b/performance/lib/penpot-client.js @@ -0,0 +1,525 @@ +// Penpot k6 HTTP Client +// +// Shared module that wraps the Penpot backend RPC API using plain JSON. +// The backend supports `application/json` request bodies (kebab-case keys) +// and `application/json` responses (camelCase keys) via Accept header or _fmt=json. +// +// Authentication is cookie-based: login-with-password sets a session cookie, +// and all subsequent requests include it automatically via the k6 cookie jar. + +import http from "k6/http"; +import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js"; + +// --------------------------------------------------------------------------- +// Configuration +// --------------------------------------------------------------------------- + +/** + * Creates a new Penpot client instance. + * + * @param {string} baseUrl - The base URL of the Penpot backend (e.g., "http://localhost:6060") + * @returns {object} Client instance with RPC methods + */ +export function createClient(baseUrl) { + // Per-VU session identifiers — consistent across all requests within one VU iteration + const sessionId = uuidv4(); + const externalSessionId = uuidv4(); + + const defaultHeaders = { + "Accept": "application/json", + "x-session-id": sessionId, + "x-external-session-id": externalSessionId, + "x-event-origin": "perf-test", + "x-client": "penpot-perf/1.0", + }; + + // k6 automatically manages cookies per VU when `cookies` are returned by the server. + // We use the default cookie jar which is per-VU. + + /** + * Make an RPC call to the Penpot backend. + * + * GET requests: params go as query parameters, response is JSON via _fmt=json. + * POST requests: params go as JSON body, response is JSON via Accept header. + * + * @param {string} method - HTTP method ("GET" or "POST") + * @param {string} command - RPC command name (e.g., "login-with-password") + * @param {object} params - Parameters for the RPC call + * @param {object} [opts] - Additional options + * @param {string} [opts.tag] - k6 metric tag for this request + * @returns {object} k6 Response object with parsed JSON body + */ + function rpc(method, command, params = {}, opts = {}) { + const url = `${baseUrl}/api/main/methods/${command}`; + const tag = opts.tag || command; + + const tags = { + rpc_command: tag, + }; + + if (method === "GET") { + // GET requests: params go as query string, add _fmt=json for JSON response + const queryParams = { ...params, _fmt: "json" }; + const qs = Object.entries(queryParams) + .filter(([, v]) => v !== undefined && v !== null) + .map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`) + .join("&"); + + const fullUrl = qs ? `${url}?${qs}` : url; + + return http.get(fullUrl, { + headers: defaultHeaders, + tags, + }); + } else { + // POST requests: params go as JSON body + const headers = { + ...defaultHeaders, + "Content-Type": "application/json", + }; + + return http.post(url, JSON.stringify(params), { + headers, + tags, + }); + } + } + + /** + * Login with email and password. + * Returns the profile data on success. The session cookie is stored + * automatically by k6's cookie jar. + * + * @param {string} email + * @param {string} password + * @returns {object} Parsed response { status, body } + */ + function login(email, password) { + const res = rpc("POST", "login-with-password", { + email, + password, + }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Get the current user's profile (requires prior login). + * + * @returns {object} Parsed response { status, body } + */ + function getProfile() { + const res = rpc("GET", "get-profile"); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Get all teams for the current user. + * + * @returns {object} Parsed response { status, body } + */ + function getTeams() { + const res = rpc("GET", "get-teams"); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Create a new team. + * + * @param {string} name - Team name + * @returns {object} Parsed response { status, body } + */ + function createTeam(name) { + const res = rpc("POST", "create-team", { name }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Get projects for a team. + * + * @param {string} teamId - Team UUID + * @returns {object} Parsed response { status, body } + */ + function getProjects(teamId) { + const res = rpc("GET", "get-projects", { "team-id": teamId }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Create a new project. + * + * @param {string} teamId - Team UUID + * @param {string} name - Project name + * @returns {object} Parsed response { status, body } + */ + function createProject(teamId, name) { + const res = rpc("POST", "create-project", { + "team-id": teamId, + name, + }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Create a new file in a project. + * + * @param {string} projectId - Project UUID + * @param {string} name - File name + * @returns {object} Parsed response { status, body } + */ + function createFile(projectId, name) { + const res = rpc("POST", "create-file", { + "project-id": projectId, + name, + }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Get a file by ID. + * + * @param {string} fileId - File UUID + * @returns {object} Parsed response { status, body } + */ + function getFile(fileId) { + const res = rpc("GET", "get-file", { + id: fileId, + }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Update a file with changes. + * + * The backend uses optimistic concurrency control via `revn`. + * If a conflict occurs (status 400 with :revn-conflict), the caller + * should retry with the latest revn from getFile(). + * + * @param {string} fileId - File UUID + * @param {number} revn - Current file revision number + * @param {number} vern - Current file version number + * @param {string} sessionId - Client session ID (UUID) + * @param {Array} changes - Array of change objects + * @returns {object} Parsed response { status, body } + */ + function updateFile(fileId, revn, vern, changesSessionId, changes) { + const params = { + id: fileId, + revn: revn, + vern: vern, + "session-id": changesSessionId, + origin: "workspace", + "created-at": new Date().toISOString(), + "commit-id": uuidv4(), + changes: changes, + }; + + // update-file uses POST with id also as query param (per frontend convention) + const url = `${baseUrl}/api/main/methods/update-file?id=${encodeURIComponent(fileId)}`; + const headers = { + ...defaultHeaders, + "Content-Type": "application/json", + }; + + const res = http.post(url, JSON.stringify(params), { + headers, + tags: { rpc_command: "update-file" }, + }); + + let body = null; + try { + if (res.body && res.body.length > 0) { + body = res.json(); + } + } catch (e) { + // body may not be JSON + } + + return { + status: res.status, + body: body, + raw: res, + }; + } + + /** + * Upload a file media object using direct multipart upload. + * + * @param {string} fileId - File UUID + * @param {Uint8Array} fileBytes - The file content + * @param {string} fileName - The file name + * @param {string} mimeType - MIME type (e.g., "image/png") + * @returns {object} Parsed response { status, body } + */ + function uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType) { + const url = `${baseUrl}/api/main/methods/upload-file-media-object`; + + const headers = { + ...defaultHeaders, + // No Content-Type — k6 sets it automatically for multipart/form-data + }; + + const formData = { + "file-id": fileId, + "is-local": "true", + name: fileName, + content: http.file(fileBytes, fileName, mimeType), + }; + + const res = http.post(url, formData, { + headers, + tags: { rpc_command: "upload-file-media-object" }, + }); + + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + // ----------------------------------------------------------------------- + // Chunked upload + // ----------------------------------------------------------------------- + + /** + * Create an upload session for chunked uploads. + * + * @param {number} totalChunks - Number of chunks + * @returns {object} { status, sessionId } + */ + function createUploadSession(totalChunks) { + const res = rpc("POST", "create-upload-session", { + "total-chunks": totalChunks, + }); + const body = res.status === 200 ? res.json() : null; + return { + status: res.status, + sessionId: body ? body.sessionId : null, + raw: res, + }; + } + + /** + * Upload a single chunk within an upload session. + * + * @param {string} sessionId - Upload session UUID + * @param {number} index - Chunk index (0-based) + * @param {Uint8Array} chunkBytes - The chunk content + * @param {string} fileName - Original file name + * @param {string} mimeType - MIME type + * @returns {object} Parsed response { status, body } + */ + function uploadChunk(sessionId, index, chunkBytes, fileName, mimeType) { + const url = `${baseUrl}/api/main/methods/upload-chunk`; + + const headers = { + ...defaultHeaders, + }; + + const formData = { + "session-id": sessionId, + index: String(index), + content: http.file(chunkBytes, fileName, mimeType), + }; + + const res = http.post(url, formData, { + headers, + tags: { rpc_command: "upload-chunk" }, + }); + + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + /** + * Assemble all uploaded chunks into a final media object. + * + * @param {string} sessionId - Upload session UUID + * @param {string} fileId - File UUID + * @param {string} name - Media object name + * @param {boolean} isLocal - Whether the object is local to the file + * @param {string} mimeType - MIME type (e.g., "image/png") + * @returns {object} Parsed response { status, body } + */ + function assembleFileMediaObject(sessionId, fileId, name, isLocal, mimeType) { + const res = rpc("POST", "assemble-file-media-object", { + "session-id": sessionId, + "file-id": fileId, + name: name, + "is-local": isLocal, + mtype: mimeType, + }); + return { + status: res.status, + body: res.status === 200 ? res.json() : null, + raw: res, + }; + } + + // ----------------------------------------------------------------------- + // Smart upload — picks direct or chunked based on file size + // ----------------------------------------------------------------------- + + // Chunk size threshold: files larger than this use chunked upload. + // The actual chunk size is irrelevant to the backend; this controls + // which upload path is exercised. + const CHUNK_SIZE = 50 * 1024; // 50 KB + + /** + * Upload a file media object, automatically selecting direct or chunked + * upload based on file size. + * + * Files <= CHUNK_SIZE use direct multipart upload. + * Files > CHUNK_SIZE use chunked upload (create-upload-session → + * upload-chunk × N → assemble-file-media-object). + * + * @param {string} fileId - File UUID + * @param {Uint8Array} fileBytes - The file content + * @param {string} fileName - The file name + * @param {string} mimeType - MIME type (e.g., "image/png") + * @returns {object} Parsed response { status, body } + */ + function uploadFileMediaObject(fileId, fileBytes, fileName, mimeType) { + if (fileBytes.byteLength <= CHUNK_SIZE) { + return uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType); + } + + // Chunked upload path + const totalChunks = Math.ceil(fileBytes.byteLength / CHUNK_SIZE); + + const sessionRes = createUploadSession(totalChunks); + if (sessionRes.status !== 200) { + return { status: sessionRes.status, body: null, raw: sessionRes.raw }; + } + const uploadSessionId = sessionRes.sessionId; + + for (let i = 0; i < totalChunks; i++) { + const start = i * CHUNK_SIZE; + const end = Math.min(start + CHUNK_SIZE, fileBytes.byteLength); + const chunk = fileBytes.slice(start, end); + + const chunkRes = uploadChunk(uploadSessionId, i, chunk, fileName, mimeType); + if (chunkRes.status !== 200) { + return { status: chunkRes.status, body: null, raw: chunkRes.raw }; + } + } + + return assembleFileMediaObject(uploadSessionId, fileId, fileName, true, mimeType); + } + + /** + * Delete a file. + * + * @param {string} fileId - File UUID + * @returns {object} Parsed response { status } + */ + function deleteFile(fileId) { + const res = rpc("POST", "delete-file", { id: fileId }); + return { + status: res.status, + raw: res, + }; + } + + /** + * Delete a project. + * + * @param {string} projectId - Project UUID + * @returns {object} Parsed response { status } + */ + function deleteProject(projectId) { + const res = rpc("POST", "delete-project", { id: projectId }); + return { + status: res.status, + raw: res, + }; + } + + /** + * Delete a team. + * + * @param {string} teamId - Team UUID + * @returns {object} Parsed response { status } + */ + function deleteTeam(teamId) { + const res = rpc("POST", "delete-team", { id: teamId }); + return { + status: res.status, + raw: res, + }; + } + + /** + * Logout the current user. + * + * @param {string} profileId - Profile UUID + * @returns {object} Parsed response { status } + */ + function logout(profileId) { + const res = rpc("POST", "logout", { "profile-id": profileId }); + return { + status: res.status, + raw: res, + }; + } + + // Return the client interface + return { + sessionId, + externalSessionId, + rpc, + login, + getProfile, + getTeams, + createTeam, + getProjects, + createProject, + createFile, + getFile, + updateFile, + uploadFileMediaObject, + uploadFileMediaObjectDirect, + createUploadSession, + uploadChunk, + assembleFileMediaObject, + deleteFile, + deleteProject, + deleteTeam, + logout, + }; +} diff --git a/performance/run.sh b/performance/run.sh new file mode 100755 index 0000000000..48f890e202 --- /dev/null +++ b/performance/run.sh @@ -0,0 +1,160 @@ +#!/usr/bin/env bash +# +# Penpot Performance Tests +# +# k6-based load/performance test suite for the Penpot backend. +# +# Prerequisites: +# - k6 (https://k6.io/) installed and in PATH +# - A running Penpot backend (local devenv or remote) +# +# Usage: +# ./run.sh smoke # 1 VU, 1 iteration smoke test +# ./run.sh lifecycle # 1 VU, 1 iteration (defaults) +# ./run.sh lifecycle -v 10 -n 5 # 10 VUs, 5 iterations each +# ./run.sh lifecycle -u http://remote:6060 -m register -v 5 +# ./run.sh clean # Remove test results + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- + +BASE_URL="${PENPOT_BASE_URL:-http://localhost:6060}" +VUS=1 +ITER=1 +REGISTER_MODE="${PENPOT_REGISTER_MODE:-demo}" +K6="${K6:-k6}" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +usage() { + cat < [options] + +Commands: + smoke 1 VU, 1 iteration smoke test (forces demo mode) + lifecycle Full user lifecycle test + clean Remove test results + help Show this help + +Options (lifecycle only): + -u URL Backend base URL (default: $BASE_URL) + -v NUM Number of virtual users (default: $VUS) + -n NUM Iterations per VU (default: $ITER) + -m MODE Register mode: 'demo' or 'register' (default: $REGISTER_MODE) + -k PATH Path to k6 binary (default: $K6) + +Environment variables: + PENPOT_BASE_URL Same as -u + PENPOT_REGISTER_MODE Same as -m + K6 Same as -k + +Examples: + $(basename "$0") smoke + $(basename "$0") lifecycle -v 10 -n 5 + $(basename "$0") lifecycle -m register -v 5 -n 1 + $(basename "$0") lifecycle -u https://penpot.example.com +EOF +} + +check_k6() { + if ! command -v "$K6" &>/dev/null; then + echo "Error: k6 not found at '$K6'" >&2 + echo "Install from https://k6.io/docs/get-started/installation/" >&2 + exit 1 + fi +} + +run_k6() { + local results_dir="$SCRIPT_DIR/results/$(date +%Y%m%d-%H%M%S)" + mkdir -p "$results_dir" + + echo "Penpot Performance Test" + echo " Base URL: $BASE_URL" + echo " Register mode: $REGISTER_MODE" + echo " VUs: $VUS" + echo " Iterations: $ITER" + echo " Results: $results_dir" + echo "" + + "$K6" run \ + --env "PENPOT_BASE_URL=$BASE_URL" \ + --env "PENPOT_REGISTER_MODE=$REGISTER_MODE" \ + --vus "$VUS" \ + --iterations "$ITER" \ + --out "json=$results_dir/k6-summary.json" \ + "$SCRIPT_DIR/scripts/lifecycle.js" +} + +# --------------------------------------------------------------------------- +# Commands +# --------------------------------------------------------------------------- + +cmd_smoke() { + check_k6 + REGISTER_MODE=demo + VUS=1 + ITER=1 + run_k6 +} + +cmd_lifecycle() { + # Parse options + while getopts "u:v:n:m:k:h" opt; do + case "$opt" in + u) BASE_URL="$OPTARG" ;; + v) VUS="$OPTARG" ;; + n) ITER="$OPTARG" ;; + m) REGISTER_MODE="$OPTARG" ;; + k) K6="$OPTARG" ;; + h) usage; exit 0 ;; + *) usage >&2; exit 1 ;; + esac + done + + check_k6 + run_k6 +} + +cmd_clean() { + local results_dir="$SCRIPT_DIR/results" + if [[ -d "$results_dir" ]]; then + rm -rf "$results_dir" + echo "Cleaned $results_dir" + else + echo "Nothing to clean" + fi +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +if [[ $# -lt 1 ]]; then + usage >&2 + exit 1 +fi + +command="$1" +shift + +case "$command" in + smoke) cmd_smoke "$@" ;; + lifecycle) cmd_lifecycle "$@" ;; + clean) cmd_clean "$@" ;; + help|-h|--help) usage ;; + *) + echo "Unknown command: $command" >&2 + usage >&2 + exit 1 + ;; +esac diff --git a/performance/scripts/lifecycle.js b/performance/scripts/lifecycle.js new file mode 100644 index 0000000000..8ba13f06e4 --- /dev/null +++ b/performance/scripts/lifecycle.js @@ -0,0 +1,417 @@ +// Lifecycle Performance Test +// +// Simulates a realistic user lifecycle from registration through CRUD operations. +// Each VU performs the full flow independently, creating its own artifacts. +// +// Flow: +// 1. Register (via demo profile or prepare+register) +// 2. Login +// 3. Get profile & teams +// 4. Create project +// 5. Create file +// 6. Get file +// 7. Update file (add a shape) +// 8. Upload image to file +// 9. Delete file +// 10. Delete project +// 11. Delete team +// +// Usage: +// k6 run scripts/lifecycle.js +// k6 run --env PENPOT_BASE_URL=http://localhost:6060 scripts/lifecycle.js +// k6 run --env PENPOT_REGISTER_MODE=register scripts/lifecycle.js + +import { check, sleep, fail } from "k6"; +import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js"; +import { createClient } from "../lib/penpot-client.js"; + +// --------------------------------------------------------------------------- +// Configuration +// --------------------------------------------------------------------------- + +const BASE_URL = __ENV.PENPOT_BASE_URL || "http://localhost:6060"; +// "demo" = use create-demo-profile (requires demo-users flag) +// "register" = use prepare-register-profile + register-profile +const REGISTER_MODE = __ENV.PENPOT_REGISTER_MODE || "demo"; + +// k6 options — smoke test defaults (1 VU, 1 iteration) +export const options = { + scenarios: { + lifecycle: { + executor: "per-vu-iterations", + vus: 1, + iterations: 1, + maxDuration: "2m", + }, + }, + thresholds: { + http_req_duration: ["p(95)<5000"], + http_req_failed: ["rate<0.01"], + "http_req_duration{rpc_command:login-with-password}": ["p(95)<1000"], + "http_req_duration{rpc_command:get-profile}": ["p(95)<500"], + "http_req_duration{rpc_command:create-project}": ["p(95)<1000"], + "http_req_duration{rpc_command:create-file}": ["p(95)<1000"], + "http_req_duration{rpc_command:get-file}": ["p(95)<500"], + "http_req_duration{rpc_command:update-file}": ["p(95)<2000"], + "http_req_duration{rpc_command:delete-file}": ["p(95)<1000"], + }, +}; + +// --------------------------------------------------------------------------- +// Test Data +// --------------------------------------------------------------------------- + +// Load test PNG fixtures — small uses direct upload, large uses chunked upload +const testImageSmall = open("../fixtures/test-small.png", "b"); +const testImageLarge = open("../fixtures/test-large.png", "b"); + +// A minimal "add-obj" change payload for update-file. +// This adds a simple rectangle shape to the first page. +// All object properties use camelCase — the backend's JSON parser +// (json/read-kebab-key) converts camelCase to kebab-case keywords automatically. +function makeAddRectChange(pageId) { + const shapeId = uuidv4(); + const x = 100; + const y = 100; + const w = 200; + const h = 150; + + return { + type: "add-obj", + pageId: pageId, + id: shapeId, + frameId: pageId, // required: the frame this shape belongs to + parentId: pageId, // root frame is the page itself + obj: { + id: shapeId, + type: "rect", + name: "Perf Test Rect", + x: x, + y: y, + width: w, + height: h, + fillColor: "#ff0000", + fillOpacity: 1, + rotation: 0, + hidden: false, + locked: false, + // Required base attrs + selrect: { + x: x, + y: y, + width: w, + height: h, + x1: x, + y1: y, + x2: x + w, + y2: y + h, + }, + points: [ + { x: x, y: y }, + { x: x + w, y: y }, + { x: x + w, y: y + h }, + { x: x, y: y + h }, + ], + transform: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 }, + transformInverse: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 }, + parentId: pageId, + frameId: pageId, + }, + }; +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function assertOk(res, label) { + const ok = check(res, { + [`${label} — status is 2xx`]: (r) => r.status >= 200 && r.status < 300, + }); + if (!ok) { + // Try to get the response body for debugging + let bodyStr = ""; + try { + if (res.raw && res.raw.body) { + bodyStr = typeof res.raw.body === "string" + ? res.raw.body.substring(0, 500) + : JSON.stringify(res.raw.body).substring(0, 500); + } else if (res.body) { + bodyStr = JSON.stringify(res.body).substring(0, 500); + } + } catch (e) { + bodyStr = "(could not read body)"; + } + console.error( + `FAIL: ${label} — status=${res.status} body=${bodyStr}` + ); + } + return ok; +} + +// --------------------------------------------------------------------------- +// Setup — runs once before VUs start +// --------------------------------------------------------------------------- + +export function setup() { + console.log(`Penpot Lifecycle Test`); + console.log(` Base URL: ${BASE_URL}`); + console.log(` Register mode: ${REGISTER_MODE}`); + console.log(``); + + // Verify the backend is reachable + const client = createClient(BASE_URL); + const res = client.getProfile(); + // We expect 401/403 (not logged in) — anything else means the backend is down + if (res.status === 0) { + fail(`Backend unreachable at ${BASE_URL}`); + } + + return { baseUrl: BASE_URL, registerMode: REGISTER_MODE }; +} + +// --------------------------------------------------------------------------- +// Main VU Function +// --------------------------------------------------------------------------- + +export default function (data) { + const client = createClient(data.baseUrl); + + // ---- Step 0: Create a user account ---- + let email, password; + + if (data.registerMode === "demo") { + // Demo mode: create-demo-profile returns email + password + const demoRes = client.rpc("POST", "create-demo-profile", {}); + if (!assertOk(demoRes, "create-demo-profile")) { + fail("Failed to create demo profile"); + } + const demoBody = demoRes.json(); + email = demoBody.email; + password = demoBody.password; + console.log(`VU ${__VU}: Created demo profile: ${email}`); + } else { + // Register mode: prepare + register + email = `perf-${uuidv4()}@test.local`; + password = "PerfTest1234!"; + const fullname = `Perf User ${__VU}`; + + const prepareRes = client.rpc("POST", "prepare-register-profile", { + fullname, + email, + password, + }); + if (!assertOk(prepareRes, "prepare-register-profile")) { + fail("Failed to prepare registration"); + } + const prepareBody = prepareRes.json(); + const token = prepareBody.token; + + const registerRes = client.rpc("POST", "register-profile", { + token, + }); + if (!assertOk(registerRes, "register-profile")) { + fail("Failed to register profile"); + } + console.log(`VU ${__VU}: Registered profile: ${email}`); + } + + sleep(1); + + // ---- Step 1: Login ---- + const loginRes = client.login(email, password); + if (!assertOk(loginRes, "login-with-password")) { + fail("Login failed"); + } + const profile = loginRes.body; + const profileId = profile.id; + console.log(`VU ${__VU}: Logged in, profile-id=${profileId}`); + + sleep(1); + + // ---- Step 2: Get profile ---- + const profileRes = client.getProfile(); + if (!assertOk(profileRes, "get-profile")) { + fail("get-profile failed"); + } + + sleep(0.5); + + // ---- Step 3: Get teams ---- + const teamsRes = client.getTeams(); + if (!assertOk(teamsRes, "get-teams")) { + fail("get-teams failed"); + } + const teams = teamsRes.body; + // teams is an array; the user has a default team from registration + const defaultTeamId = Array.isArray(teams) && teams.length > 0 + ? teams[0].id + : null; + + if (!defaultTeamId) { + fail("No default team found after registration"); + } + + sleep(0.5); + + // ---- Step 4: Create a project ---- + const projectName = `Perf Project ${uuidv4().substring(0, 8)}`; + const projectRes = client.createProject(defaultTeamId, projectName); + if (!assertOk(projectRes, "create-project")) { + fail("create-project failed"); + } + const project = projectRes.body; + const projectId = project.id; + console.log(`VU ${__VU}: Created project: ${projectId}`); + + sleep(1); + + // ---- Step 5: Create a file ---- + const fileName = `Perf File ${uuidv4().substring(0, 8)}`; + const fileRes = client.createFile(projectId, fileName); + if (!assertOk(fileRes, "create-file")) { + fail("create-file failed"); + } + const file = fileRes.body; + const fileId = file.id; + console.log(`VU ${__VU}: Created file: ${fileId}`); + + sleep(1); + + // ---- Step 6: Get the file (to read revn, vern, page-id) ---- + const getFileRes = client.getFile(fileId); + if (!assertOk(getFileRes, "get-file")) { + fail("get-file failed"); + } + const fileData = getFileRes.body; + const revn = fileData.revn; + const vern = fileData.vern; + + // Extract the first page-id from the file data + // fileData.data.pages is an array of page UUIDs + // fileData.data.pages-index is a map of page-id -> page objects + let pageId = null; + if (fileData.data && fileData.data.pages && fileData.data.pages.length > 0) { + pageId = fileData.data.pages[0]; + } + + if (!pageId) { + console.warn(`VU ${__VU}: Could not find page-id in file data, skipping update-file`); + } + + sleep(1); + + // ---- Step 7: Update file (add a rectangle shape) ---- + let updateOk = false; + if (pageId) { + const changes = [makeAddRectChange(pageId)]; + const updateRes = client.updateFile(fileId, revn, vern, client.sessionId, changes); + + if (updateRes.status === 200) { + updateOk = true; + console.log(`VU ${__VU}: Updated file successfully`); + } else { + // Check for revn conflict — retry once + const body = updateRes.body; + console.error(`VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 500)}`); + const isRevnConflict = + body && (body.code === "revn-conflict" || body.type === "revn-conflict"); + + if (isRevnConflict) { + console.log(`VU ${__VU}: Revn conflict, retrying...`); + // Fetch latest file state + const retryFileRes = client.getFile(fileId); + if (assertOk(retryFileRes, "get-file (retry)")) { + const retryData = retryFileRes.body; + const retryRes = client.updateFile( + fileId, + retryData.revn, + retryData.vern, + client.sessionId, + changes + ); + if (assertOk(retryRes, "update-file (retry)")) { + updateOk = true; + console.log(`VU ${__VU}: Updated file on retry`); + } + } + } else { + console.error( + `VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 300)}` + ); + } + } + } + + sleep(1); + + // ---- Step 8: Upload images to the file ---- + // Small image (97 B) uses direct multipart upload. + // Large image (120 KB) uses chunked upload (create-session → upload-chunk × N → assemble). + if (testImageSmall && testImageSmall.byteLength > 0) { + const uploadRes = client.uploadFileMediaObject( + fileId, + testImageSmall, + "test-small.png", + "image/png" + ); + if (assertOk(uploadRes, "upload-file-media-object (direct)")) { + console.log(`VU ${__VU}: Uploaded small image (direct)`); + } + } + + sleep(0.5); + + if (testImageLarge && testImageLarge.byteLength > 0) { + const uploadRes = client.uploadFileMediaObject( + fileId, + testImageLarge, + "test-large.png", + "image/png" + ); + if (assertOk(uploadRes, "upload-file-media-object (chunked)")) { + console.log(`VU ${__VU}: Uploaded large image (chunked)`); + } + } + + sleep(1); + + // ---- Step 9: Delete the file ---- + const deleteFileRes = client.deleteFile(fileId); + if (assertOk(deleteFileRes, "delete-file")) { + console.log(`VU ${__VU}: Deleted file: ${fileId}`); + } + + sleep(0.5); + + // ---- Step 10: Delete the project ---- + const deleteProjectRes = client.deleteProject(projectId); + if (assertOk(deleteProjectRes, "delete-project")) { + console.log(`VU ${__VU}: Deleted project: ${projectId}`); + } + + sleep(0.5); + + // ---- Step 11: Delete the team ---- + // Note: We only delete the team if it's NOT the default team. + // The default team cannot be deleted (or may cause errors). + // For this test, we skip team deletion to avoid errors. + // In a real scenario, we'd create a separate team and delete that. + console.log(`VU ${__VU}: Skipping team deletion (using default team)`); + + sleep(0.5); + + // ---- Step 12: Logout ---- + const logoutRes = client.logout(profileId); + console.log(`VU ${__VU}: Logout status: ${logoutRes.status}`); + + console.log(`VU ${__VU}: Lifecycle complete`); +} + +// --------------------------------------------------------------------------- +// Teardown — runs once after all VUs finish +// --------------------------------------------------------------------------- + +export function teardown(data) { + console.log("Lifecycle test complete."); +} diff --git a/plans/2026-06-12-backend-performance-test.md b/plans/2026-06-12-backend-performance-test.md new file mode 100644 index 0000000000..e2e63f6baf --- /dev/null +++ b/plans/2026-06-12-backend-performance-test.md @@ -0,0 +1,494 @@ +# Backend Performance Test Plan + +**Context:** Build a k6-based load/performance test suite that simulates realistic browser-to-backend HTTP flows for distinct Penpot user operations. The goal is to measure backend impact (latency, throughput, error rates, resource saturation) under synthetic user load. **Browser rendering performance is explicitly out of scope.** WebSocket testing is deferred. + +**Date:** 2026-06-12 +**Validated Requirements:** +- Tool: **k6** (confirmed). +- Environment: flexible — local devenv first, then remote staging/perf. +- Target scale: **1000 concurrent VUs** (ramping from lower baselines). +- Flows: **realistic CRUD lifecycle** — create, edit, upload, delete. Must include **image upload** and **font upload**. +- `update-file` is important but difficult because it requires **2–3 concurrent users editing the same file**, and **file size matters**. +- WebSocket: **deferred**. + +--- + +## Current Progress + +### Completed (2026-06-12) + +Phase 1 is done. Phase 2 has the first flow (lifecycle) implemented and validated. + +**What was built:** + +``` +performance/ +├── run.sh # Bash runner script (smoke, lifecycle, clean, help) +├── README.md # Usage docs, configuration, architecture notes +├── lib/ +│ └── penpot-client.js # 388 lines — shared k6 HTTP client module +├── scripts/ +│ └── lifecycle.js # 398 lines — full user lifecycle test +├── fixtures/ +│ ├── test-small.png # 97 B +│ ├── test-medium.png # 30 KB +│ └── test-large.png # 120 KB +├── results/ # k6 JSON output (gitignored) +└── baselines/ # for regression baselines +``` + +**What the lifecycle test covers (all passing):** + +| Step | RPC Command | Method | Observed Latency | +|------|------------|--------|-----------------| +| 1. Register | `create-demo-profile` | POST | ~136ms | +| 2. Login | `login-with-password` | POST | ~136ms | +| 3. Get profile | `get-profile` | GET | ~7ms | +| 4. Get teams | `get-teams` | GET | ~6ms | +| 5. Create project | `create-project` | POST | ~9ms | +| 6. Create file | `create-file` | POST | ~22ms | +| 7. Get file | `get-file` | GET | ~12ms | +| 8. Update file | `update-file` | POST | ~25ms | +| 9. Upload image | `upload-file-media-object` | POST (multipart) | ~7ms | +| 10. Delete file | `delete-file` | POST | ~12ms | +| 11. Delete project | `delete-project` | POST | ~5ms | +| 12. Logout | `logout` | POST | ~4ms | + +Smoke test result: **10/10 checks pass, 0% failure rate, ~10s total iteration**. + +**Key discoveries during implementation:** + +1. **JSON transport works.** The backend accepts `Content-Type: application/json` for POST bodies (kebab-case keys auto-converted) and returns `application/json` responses (camelCase keys) via `Accept: application/json` or `_fmt=json` query param. No Transit encoder needed in k6. + +2. **`create-file` `features` param.** Sending `features: []` (empty array) causes a 400 validation error. Omit the field entirely — it's optional in the schema (`backend/src/app/rpc/commands/files_create.clj`). + +3. **`update-file` `changes` payload — shape schema is strict.** The `add-obj` change requires a fully valid shape object. Minimum required fields beyond the basics: + - `selrect`: `{x, y, width, height, x1, y1, x2, y2}` + - `points`: array of 4 `{x, y}` corner points + - `transform` / `transform-inverse`: identity matrix `{a:1, b:0, c:0, d:1, e:0, f:0}` + - `parent-id` and `frame-id` inside the `obj` + - `frame-id` at the top level of the change object itself (required, not optional) + - Schema defined in `common/src/app/common/files/changes.cljc` (line 189) and `common/src/app/common/types/shape.cljc` (line 165). + +4. **`update-file` URL convention.** The frontend sends `id` both as a query param and in the POST body. The URL is `POST /api/main/methods/update-file?id=`. + +5. **Two registration modes work:** + - `demo` mode: `create-demo-profile` — fast, requires `demo-users` feature flag. + - `register` mode: `prepare-register-profile` + `register-profile` — two-step flow, works without flags. + +6. **k6 installed at** `/home/penpot/.local/bin/k6` (v0.56.0). Not in default PATH; use `PATH="/home/penpot/.local/bin:$PATH"` or set `K6` env var. + +### Remaining Work + +| Phase | Status | Next Actions | +|-------|--------|-------------| +| Phase 1 – Discovery & Tooling | **Done** | — | +| Phase 2 – Core HTTP Flows | **Partial** — lifecycle.js done | Write `workspace-open.js`, `workspace-edit.js`, `media-upload.js`, `font-upload.js`, `viewer.js`, `export.js` | +| Phase 2 – Data seeding | **Not started** | Create `seed-data.js` for 100+ user pool | +| Phase 3 – Scenarios | **Not started** | Define multi-scenario k6 config mixing flows | +| Phase 4 – Advanced update-file | **Not started** | File size tiers, concurrent editing matrix | +| Phase 5 – CI & Reporting | **Not started** | Grafana dashboards, regression guard | + +### Immediate Next Steps + +1. Write `workspace-open.js` — read-heavy flow (get-file, get-file-libraries, thumbnails). +2. Write `workspace-edit.js` — write-heavy loop (get-file + update-file per VU, independent files). +3. Write `media-upload.js` — direct + chunked upload flows. +4. Write `font-upload.js` — chunked upload + create-font-variant. +5. Write `viewer.js` — get-view-only-bundle + comments. +6. Create `seed-data.js` — pre-seed 100+ users/teams/files for multi-VU runs. +7. Define scenario mix in `run.sh` (multi-scenario support with ramping VUs). +8. Add `--scenario` flag to `run.sh` to select individual flows or the full mix. + +--- + +## Affected Modules + +| Module | Why it is involved | +|--------|---------------------| +| `backend/` | Target system. All HTTP RPC (`/api/main/methods/*`), auth, storage, media processing, DB, and Prometheus metrics (`/metrics`). | +| `frontend/` | Source of truth for user request flows. We inspect `app.main.repo` (RPC client), `app.main.data.*` (user flows), and `app.main.data.persistence` (save semantics). | +| `common/` | Shared schemas, Transit helpers, and data structures. Used to understand valid `update-file` `changes` payloads. | + +--- + +## Approach + +### Phase 1 – Discovery & Tooling (Days 1–2) + +#### 1.1. Read the frontend RPC flows to build a request catalog + +Inspect these files to map every user action to its RPC command: + +- `frontend/src/app/main/repo.cljs` — HTTP client conventions (headers, retry, GET vs POST rules, query params, form-data, multipart). +- `frontend/src/app/main/data/dashboard.cljs` — Dashboard init (`get-projects`, `fetch-fonts`, `search-files`). +- `frontend/src/app/main/data/workspace.cljs` — Workspace init (`get-file`, `get-file-libraries`, `get-file-object-thumbnails`, `resolve-file` via `get-file-fragment`). +- `frontend/src/app/main/data/persistence.cljs` — File save flow (`update-file` with `changes`, `revn`, `session-id`, debounce/buffer logic). +- `frontend/src/app/main/data/viewer.cljs` — Viewer flow (`get-view-only-bundle`). +- `frontend/src/app/main/data/comments.cljs` — Comment thread fetch (`get-comment-threads`). +- `frontend/src/app/main/data/media.cljs` / `upload.cljs` — Media upload flows (`upload-file-media-object`, `create-upload-session`, `upload-chunk`, `assemble-file-media-object`). +- `frontend/src/app/main/data/fonts.cljs` — Font upload flow (`create-font-variant` with `:uploads` map). +- `frontend/src/app/main/data/team.cljs` — Team creation (`create-team`), invitation (`create-team-invitations`). +- `frontend/src/app/main/data/project.cljs` — Project creation (`create-project`). + +**Goal:** produce a **Request Catalog** mapping user actions to RPC command names, HTTP methods, payload shapes, and required preconditions (e.g., `team-id`, `file-id`). + +#### 1.2. Confirm JSON compatibility for the test harness + +The backend middleware (`app.http.middleware`) supports `application/json` request bodies and `application/json` responses (via `_fmt=json` or `Accept: application/json`). + +- **Action:** Send a manual `curl` to `POST /api/main/methods/login-with-password` with `Content-Type: application/json` and verify the response format. +- **Action:** Verify `GET /api/main/methods/get-profile` with `Accept: application/json` returns plain JSON. +- **Action:** Verify `POST /api/main/methods/update-file` with `Content-Type: application/json` and `_fmt=json` works. +- **Action:** Verify `POST /api/main/methods/upload-file-media-object` with `multipart/form-data` works (k6 supports this natively). + +#### 1.3. Set up the load testing directory and shared client + +Create a directory `performance/` at the repo root. + +Install **k6** (`k6` CLI or Docker image). + +Create a shared `penpot-client.js` module that wraps: +- `login(email, password)` → returns session cookie / token. +- `rpc(cmd, params, opts)` → builds the correct URL, headers, body, and query params. +- `uploadFileMediaObject(fileId, filePath, name)` → multipart upload. +- `createUploadSession(totalChunks)` → chunked upload setup. +- `uploadChunk(sessionId, index, chunkBytes)` → multipart chunk upload. +- `assembleFileMediaObject(sessionId, fileId, name, isLocal)` → finalize chunked upload. + +**Headers to replicate (critical for backend telemetry and session binding):** +- `x-session-id`: generated UUID per VU (must be consistent across requests for the same session). +- `x-external-session-id`: generated UUID per VU. +- `x-event-origin`: a string origin (e.g., `"perf-test"`) +- `accept`: `application/json` (for HTTP-only load path) +- `content-type`: `application/json` (or `multipart/form-data` for uploads) +- `credentials: "include"` (for cookie jar) + +#### 1.4. Data seeding strategy for 1000 VU scale + +Creating 1000 users/teams/files *inside* the load test is too slow and will distort the results. + +**Recommended approach:** +- **Setup Phase (k6 `setup()`):** Run a pre-test script that creates a shared pool of test artifacts. + - Use `login-with-password` with a fixture account (e.g., `profile1@example.com` / `123123` if fixtures exist). + - Create `N` teams, `N` projects, `N` files of varying sizes (see **File Size Tiers** below). + - Export the IDs into a JSON file that k6 `setup()` reads. +- **Alternative:** Use the backend REPL / fixtures (`app.cli.fixtures/run {:preset :small}`) to create fixture data, then export the IDs via a small Clojure script. +- **Data pool per VU:** Each VU picks a random user from the pool, or uses a dedicated user (e.g., VU #1 → `profile1@example.com`, VU #2 → `profile2@example.com`). For 1000 VUs, we need at least 1000 pre-seeded users. +- **Cleanup:** A post-test script can delete the seeded data, or we can use a dedicated perf DB that is reset between runs. + +**Action:** Document the seeding procedure in `performance/README.md` and create a `seed-data.js` script. + +--- + +### Phase 2 – Core HTTP Flow Scripts (Days 3–5) + +Create one k6 script per user flow. Each script: +- Uses `setup()` to read the shared data pool and log in. +- Uses `vu` iterations to simulate the flow. +- Tags every request with the RPC command name so k6 metrics are sliced by endpoint. +- Uses `check()` assertions for HTTP 200 and valid JSON structure. + +#### Flow 1: Realistic User Lifecycle (`lifecycle.js`) + +This is the primary realistic flow. Each VU performs a full lifecycle: + +1. **Auth** + - `POST /api/main/methods/login-with-password` → `{email, password}` + - `GET /api/main/methods/get-profile` + - `GET /api/main/methods/get-teams` + +2. **Create Team** + - `POST /api/main/methods/create-team` → `{name: "Perf Team "}` + - `GET /api/main/methods/get-team?team-id=` + +3. **Create Project** + - `POST /api/main/methods/create-project` → `{team-id, name}` + - `GET /api/main/methods/get-project?id=` + +4. **Create File** + - `POST /api/main/methods/create-file` → `{project-id, name, features}` + - `GET /api/main/methods/get-file?id=&features=<...>` + +5. **Edit File (Simple Update)** + - `POST /api/main/methods/update-file` with a minimal `changes` payload. + - **Changes payload:** Use a simple change like `{:type "add-obj", :id "", :page-id "", :parent-id "", :obj {:type "rect", ...}}`. Inspect `app.common.files.changes` for the exact schema. For a load test, we only need the shape to be structurally valid; the backend validates it. + - **Revn tracking:** Fetch the file first, read `revn`, then send `revn` in the update. If a conflict occurs (`409` or `:revn-conflict` error), retry once with the latest `revn`. + +6. **Upload Image (Direct)** + - `POST /api/main/methods/upload-file-media-object` (multipart) + - Payload: `file-id`, `is-local: true`, `name`, `content` (the file bytes). + - Use a small dummy PNG/SVG (e.g., 1 KB, 100 KB, 1 MB) stored in `performance/fixtures/`. + +7. **Upload Image (Chunked)** + - `POST /api/main/methods/create-upload-session` → `{total-chunks: N}` + - Loop `N` times: `POST /api/main/methods/upload-chunk` (multipart, `session-id`, `index`, `chunk`) + - `POST /api/main/methods/assemble-file-media-object` → `{session-id, file-id, name, is-local}` + - Use a larger dummy file (e.g., 5 MB) to stress the chunked pipeline. + +8. **Upload Font** + - `POST /api/main/methods/create-upload-session` (chunked, because fonts can be large) + - `POST /api/main/methods/upload-chunk` for each chunk + - `POST /api/main/methods/create-font-variant` → `{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": ""}}` + - Use a small real TTF/OTF file from `performance/fixtures/`. + +9. **Delete File** + - `DELETE /api/main/methods/delete-file` (verify the exact method name; it may be `update-file` with a deletion flag or a dedicated command). Inspect `frontend/src/app/main/data/dashboard.cljs` for the delete action. + +10. **Delete Project** + - `DELETE /api/main/methods/delete-project?id=` + +11. **Delete Team** + - `POST /api/main/methods/delete-team?id=` + +12. **Logout** + - (Optional; session cookie expiry is usually sufficient) + +**Pacing:** Add `sleep()` between steps to simulate realistic think time (e.g., 1–3 seconds between dashboard navigation, 3–5 seconds between edits). + +#### Flow 2: Workspace Open (Read-heavy) (`workspace-open.js`) + +For 1000 VUs, most will be read-only viewers or editors opening files. + +1. Login (reuse token from `setup`). +2. `GET /api/main/methods/get-file?id=&features=<...>` +3. `GET /api/main/methods/get-file-libraries?file-id=` +4. For each library: `GET /api/main/methods/get-file?id=` +5. `GET /api/main/methods/get-file-object-thumbnails?file-id=` +6. `GET /api/main/methods/get-file-data-for-thumbnail?file-id=&page-id=&object-id=` + +**Data:** Use a pool of files of varying sizes (see **File Size Tiers**). + +#### Flow 3: Workspace Edit (Write-heavy) (`workspace-edit.js`) + +**Scenario A — Independent editors (default, easiest to scale):** +- Each VU creates its own file in `setup()`, or picks a dedicated file from the pool. +- Loop: + 1. `GET /api/main/methods/get-file?id=` (to refresh `revn`) + 2. `POST /api/main/methods/update-file` with minimal changes + 3. `sleep(3)` +- This measures the latency of the save path without concurrency conflicts. + +**Scenario B — Concurrent editors (advanced, measures conflict resolution):** +- 2–3 VUs share the **same file ID**. +- Each VU: + 1. `GET /api/main/methods/get-file?id=` (to get latest `revn`) + 2. `POST /api/main/methods/update-file` with changes + 3. If `revn-conflict` (HTTP 400 or 409 with `:code :revn-conflict`), retry with the latest `revn`. +- **Problem:** k6 VUs are independent; they cannot easily share a mutable `revn` counter. +- **Solutions:** + 1. **Optimistic concurrency:** Let conflicts happen naturally. Measure the conflict rate and retry latency. This is realistic for many-user editing. + 2. **Shared state service:** Run a tiny Redis or in-memory service that stores the latest `revn` per file. VUs read/write it before each update. This adds coordination overhead but reduces conflicts. + 3. **Sequential VU groups:** Use k6 `scenarios` with `executor: 'per-vu-iterations'` and a small shared file pool. Accept that some conflicts will occur and measure them as part of the benchmark. +- **Recommendation:** Start with **Solution 1** (optimistic). If the conflict rate is >10%, consider **Solution 2**. + +**File Size Tiers for `update-file`:** +The backend `update-file` performance depends heavily on file data size (serialization, validation, pointer-map resolution, snapshotting). + +| Tier | Size | How to create | +|------|------|---------------| +| Small | ~10 shapes | Create a file with a few rectangles. | +| Medium | ~100 shapes | Duplicate a page with many shapes. | +| Large | ~1000 shapes | Import a real-world design file or use a fixture. | + +**Action:** Create a `create-file-fixture.js` helper that generates files of each tier via the `create-file` + `update-file` API (or by importing a `.penpot` file via the binfile import API if available). + +#### Flow 4: Viewer (Read-heavy, anonymous or logged-in) (`viewer.js`) + +1. Login (or use share-link token for anonymous). +2. `GET /api/main/methods/get-view-only-bundle?file-id=&share-id=&features=<...>` +3. `GET /api/main/methods/get-comment-threads?file-id=&share-id=` + +#### Flow 5: Export (CPU/IO-heavy) (`export.js`) + +1. Login. +2. `POST /api/export` with export payload. + - Inspect `frontend/src/app/main/data/export.cljs` for the exact payload shape. + - Common exports: `type: "png"`, `type: "svg"`, `type: "pdf"`. + - This hits the **exporter** service (Node.js/Playwright), which is a separate process. If the goal is to stress the **backend**, limit export tests or target the backend export queue endpoints. + +#### Flow 6: Media Upload (Storage/IO-heavy) (`media-upload.js`) + +1. Login. +2. Direct upload: `POST /api/main/methods/upload-file-media-object` (multipart, small PNG). +3. Chunked upload: `POST /api/main/methods/create-upload-session` → `upload-chunk` x N → `assemble-file-media-object` (large PNG). +4. URL-based upload: `POST /api/main/methods/create-file-media-object-from-url` (if a stable external image URL is available). + +#### Flow 7: Font Upload (Storage/CPU-heavy) (`font-upload.js`) + +1. Login. +2. `POST /api/main/methods/create-upload-session` (for the font file) +3. `POST /api/main/methods/upload-chunk` for each chunk +4. `POST /api/main/methods/create-font-variant` → `{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": ""}}` +5. `GET /api/main/methods/get-font-variants?team-id=` + +--- + +### Phase 3 – Scenarios & Orchestration (Day 6) + +Define k6 `options.scenarios` that mix the flows to simulate realistic traffic. + +**Example scenario mix for 1000 VUs:** + +| Scenario | Script | VUs | Arrival Rate | Duration | Notes | +|----------|--------|-----|--------------|----------|-------| +| `lifecycle` | `lifecycle.js` | 100 | 1/s (ramp 0→100 over 5m) | 10m | Full CRUD, most realistic. | +| `workspace_open` | `workspace-open.js` | 400 | 5/s (ramp 0→400 over 5m) | 10m | Read-heavy, simulates many editors opening files. | +| `workspace_edit` | `workspace-edit.js` | 200 | 2/s (ramp 0→200 over 5m) | 10m | Write-heavy, independent files. | +| `workspace_edit_concurrent` | `workspace-edit.js` | 30 (10 groups of 3) | 0.5/s | 10m | 3 VUs per file, measures conflicts. | +| `viewer` | `viewer.js` | 200 | 3/s (ramp 0→200 over 5m) | 10m | Read-heavy, simulates public/private viewers. | +| `media_upload` | `media-upload.js` | 50 | 0.5/s | 10m | Storage stress. | +| `font_upload` | `font-upload.js` | 20 | 0.2/s | 10m | Font processing stress. | + +**Thresholds:** +- `http_req_duration{p95} < 200ms` for `get-profile`, `get-teams`, `get-projects`. +- `http_req_duration{p95} < 500ms` for `get-file` (small), `search-files`. +- `http_req_duration{p95} < 2000ms` for `get-file` (large / 1000 shapes). +- `http_req_duration{p95} < 1000ms` for `update-file` (small). +- `http_req_duration{p95} < 3000ms` for `update-file` (large). +- `http_req_duration{p95} < 5000ms` for `upload-file-media-object` (1 MB). +- `http_req_duration{p95} < 10000ms` for `assemble-file-media-object` (5 MB chunked). +- `http_req_failed < 1%` globally. +- `http_req_failed{code:revn-conflict} < 5%` for `workspace_edit_concurrent`. + +**Correlation with backend metrics:** +- Scrape `/metrics` before, during, and after the test. +- Key Prometheus metrics to watch: + - `rpc_main_timing_seconds` (histogram/summary, labeled by command name) + - `rpc_management_timing_seconds` + - `http_server_dispatch_timing_seconds` + - `websocket_active_connections` (if any WS is active) + - `websocket_messages_total` + - JVM hotspot metrics (`process_cpu_seconds_total`, `jvm_memory_bytes_used`, `jvm_threads_current`) + - HikariCP metrics (if exposed; check `com.zaxxer.hikari:type=Pool` via JMX or custom Prometheus exporter) + - PostgreSQL: `pg_stat_activity` count by state. + - Redis: `INFO` `connected_clients`, `used_memory`. + +--- + +### Phase 4 – Advanced `update-file` Testing (Days 7–8) + +Because `update-file` is the core of the product and the user explicitly noted that **file size matters** and **concurrent editing is difficult**, we need a dedicated deep-dive. + +#### 4.1. File Size Tiers + +Create a `file-size-matrix.js` script that parameterizes the file size: +- `SMALL_FILE_ID`: 1 page, 10 shapes. +- `MEDIUM_FILE_ID`: 1 page, 100 shapes. +- `LARGE_FILE_ID`: 1 page, 500 shapes. +- `XLARGE_FILE_ID`: 1 page, 1000+ shapes, or a multi-page file. + +Run `workspace-edit.js` against each tier separately and plot: +- `update-file` latency vs file size. +- `get-file` latency vs file size. +- Backend CPU and DB time vs file size. + +#### 4.2. Concurrent Editing Strategy + +**Problem:** `update-file` uses optimistic concurrency control (`revn`). If two users submit the same `revn`, the second gets a conflict. + +**Backend behavior:** +- `update-file` acquires an advisory lock (`db/xact-lock! conn id`) on the file ID. +- It checks `revn` and `vern` conflicts. +- It processes changes, validates, updates the file, and sends notifications via `msgbus`. +- The transaction duration is what we want to measure. + +**Test design for concurrent editing:** +- **Shared file pool:** Pre-create 50 files of `MEDIUM` size. +- **VU grouping:** Use k6 `scenarios` with `per-vu-iterations` or `shared-iterations`. Assign groups of 3 VUs to the same file ID. +- **Conflict measurement:** Do *not* synchronize `revn` between VUs. Let them race. + - Track `http_req_failed{code:revn-conflict}`. + - Track retry latency (if a VU retries after fetching the latest `revn`). + - This gives us the **natural conflict rate** under load, which is a realistic product metric. +- **If the natural conflict rate is too high (>20%):** + - Add a small `sleep()` jitter (0–500 ms) between `get-file` and `update-file` to spread out the requests. + - Or, use a tiny shared counter (e.g., a small HTTP endpoint or Redis) that VUs read to get the "next" `revn`. This is less realistic but gives a cleaner latency measurement. + +**Action:** Create `workspace-edit-concurrent.js` with the grouping logic and conflict-rate thresholds. + +--- + +### Phase 5 – CI Integration & Reporting (Days 9–10) + +1. **Runner script (`run.sh`):** + - `./run.sh smoke` for a 1-VU, 1-iteration smoke test. ✅ Done + - `./run.sh lifecycle -v 100 -n 10` for the standard run. + - Add `--scenario` flag to run individual flows or the full mix. + +2. **Output:** + - k6 JSON/CSV output to `performance/results//`. + - Prometheus snapshot diff (before vs after). + - Grafana screenshot or dashboard export. + +3. **Grafana Dashboard:** + - Panel: `p95 latency by RPC command` (from `rpc_main_timing_seconds`). + - Panel: `HTTP requests/sec` (from k6). + - Panel: `Error rate by command` (from k6). + - Panel: `DB connection pool` (if available). + - Panel: `JVM heap used`. + - Panel: `update-file conflict rate` (custom metric from k6). + - Panel: `File size vs latency` (from the matrix test). + +4. **Regression guard:** + - Store baseline results in `performance/baselines/`. + - After any backend change, run the baseline scenario. If p95 increases by >20% for any critical command, fail the CI step. + +--- + +## Risks & Considerations + +| Risk | Mitigation | +|------|------------| +| **Scale: 1000 VUs creating data simultaneously will exhaust DB connection pool or storage quota.** | Pre-seed the data pool. Use a dedicated perf DB. Monitor `pg_stat_activity` and HikariCP metrics. | +| **Media upload (images/fonts) will saturate network I/O before the backend is stressed.** | Run the load test from the same datacenter/VPC as the backend. Use small dummy files for most tests; reserve large files for a dedicated storage-stress scenario. | +| **`update-file` conflicts under 1000 VUs may be so high that the test becomes a conflict test, not a latency test.** | Measure both. The conflict rate is itself a critical metric. If it is too high, we can add jitter or use independent files. | +| **Exporter service is a separate bottleneck.** | `export.js` should target the backend queue endpoint, not the full export pipeline, unless we want to test the exporter too. If exporter is in scope, run it as a separate scenario. | +| **Chunked upload creates many temporary DB rows (`upload_chunk` table).** | The backend has a `upload-session-gc` cron job. Ensure it runs after the test, or clean up manually. | +| **Font upload shells out to FontForge and WOFF tools.** | This is CPU-intensive and may be a bottleneck. Run font upload as a separate, low-VU scenario to measure the processing time without blocking other tests. | +| **Prometheus metrics may not expose DB pool wait time.** | Add a custom JMX exporter for HikariCP if needed, or query `pg_stat_activity` directly. | +| **Cleanup:** 1000 VUs creating teams/files will leave logical deletions or orphaned storage objects.** | Use a dedicated perf environment. Run a cleanup script after the test that deletes all seeded data via the RPC API. | + +--- + +## Testing Strategy + +### How to verify the test harness itself works + +1. **Smoke test:** Run each k6 script with `1 VU, 1 iteration` against a local devenv. Verify all requests return `200` and the response body is valid JSON. +2. **Baseline run:** Run `workspace-open.js` with `10 VUs, 60 s` against a clean devenv. Record baseline p95 and p99 latencies. +3. **Regression guard:** After any backend change, re-run the baseline. If p95 increases by >20%, flag it. +4. **Saturation test:** Ramp `workspace-edit.js` to 100 VUs editing independent files. Monitor backend CPU and DB connection pool. The test should reveal the breaking point where `update-file` latency spikes. +5. **Media upload stress test:** Run `media-upload.js` with 50 VUs uploading 1 MB files. Verify storage throughput and no `413` errors. +6. **Font upload stress test:** Run `font-upload.js` with 10 VUs. Verify FontForge CPU usage and no timeouts. + +### Manual validation checklist + +- [x] `POST /api/main/methods/login-with-password` with JSON body returns a session cookie. (Validated via k6 lifecycle) +- [x] `GET /api/main/methods/get-profile` with `Accept: application/json` returns JSON. (Validated via k6 lifecycle) +- [ ] `curl -H "Accept: application/json" http://localhost:6060/metrics` returns Prometheus text. +- [ ] Backend fixtures create at least 100 test users and 100 test files. +- [x] A `update-file` request with a minimal `changes` payload succeeds and returns `{"revn": N}`. (Validated — needs full shape with selrect, points, transform, frame-id) +- [x] A `upload-file-media-object` multipart request succeeds and returns a media object ID. (Validated via k6 lifecycle) +- [ ] A chunked upload (`create-upload-session` → `upload-chunk` → `assemble-file-media-object`) succeeds. +- [ ] A `create-font-variant` request with chunked uploads succeeds. + +--- + +## Immediate Next Steps (if approved) + +1. ~~Create `performance/` directory and `README.md`.~~ ✅ Done +2. ~~Write `penpot-client.js` (k6 shared module) with `login()`, `rpc()`, `uploadMultipart()`, and `uploadChunked()` helpers.~~ ✅ Done (388 lines, JSON transport, cookie auth, session headers, tagged metrics) +3. ~~Write a manual `curl` validation script~~ — Skipped; JSON compatibility confirmed via k6 smoke test. +4. Write a data seeding script (`performance/scripts/seed-data.js`) that creates 100+ users, teams, projects, and files of varying sizes. +5. ~~Write the first k6 script: `lifecycle.js`~~ ✅ Done (398 lines, 12-step CRUD flow, all checks passing) +6. ~~Run a 1-VU smoke test against local devenv and commit the baseline results.~~ ✅ Done (10/10 checks, 0% failure, ~10s iteration) +7. Write `workspace-open.js` and `workspace-edit.js`. +8. Write `media-upload.js` and `font-upload.js`. +9. Define the `1000-VU` scenario mix in `options.js` (shared scenario config). +10. Run the first 100-VU ramp test and capture Prometheus metrics. + +--- + +**Plan Author:** Senior Software Architect +**Status:** Phase 1 complete, Phase 2 partially complete (lifecycle.js done). See "Current Progress" section above. +