Add initial structure for performance tests

This commit is contained in:
Andrey Antukh 2026-06-12 11:42:15 +02:00
parent b2439694af
commit 4416d9380e
8 changed files with 1698 additions and 0 deletions

102
performance/README.md Normal file
View File

@ -0,0 +1,102 @@
# Penpot Performance Tests
k6-based load and performance test suite for the Penpot backend. Measures HTTP RPC latency, throughput, and error rates under synthetic user load.
## Prerequisites
- **k6** — Install from https://k6.io/docs/get-started/installation/
- **Running Penpot backend** — Local devenv (`http://localhost:6060`) or a remote instance
## Quick Start
```bash
# Smoke test — 1 VU, 1 iteration, demo mode
./run.sh smoke
# Full lifecycle with 10 VUs, 5 iterations each
./run.sh lifecycle -v 10 -n 5
# Use registration flow instead of demo profiles
./run.sh lifecycle -m register -v 5 -n 1
# Point to a remote backend
./run.sh lifecycle -u https://penpot.example.com
# Show all options
./run.sh help
```
## Test Scripts
### `scripts/lifecycle.js` — Full User Lifecycle
Simulates a realistic user journey from account creation through CRUD operations:
1. **Register** — Create a new user (demo profile or full registration)
2. **Login** — Authenticate and obtain session cookie
3. **Get Profile** — Fetch current user profile
4. **Get Teams** — List user teams
5. **Create Project** — Create a new project in the default team
6. **Create File** — Create a new design file in the project
7. **Get File** — Fetch the file with its data (pages, objects)
8. **Update File** — Add a rectangle shape (tests optimistic concurrency)
9. **Upload Image** — Upload a PNG to the file's media objects
10. **Delete File** — Remove the file
11. **Delete Project** — Remove the project
12. **Logout** — End the session
Each VU performs the full flow independently, creating and cleaning up its own artifacts.
## Configuration
Options for `run.sh lifecycle`:
| Flag | Env Variable | Default | Description |
|------|-------------|---------|-------------|
| `-u URL` | `PENPOT_BASE_URL` | `http://localhost:6060` | Penpot backend URL |
| `-v NUM` | — | `1` | Number of virtual users |
| `-n NUM` | — | `1` | Iterations per VU |
| `-m MODE` | `PENPOT_REGISTER_MODE` | `demo` | `demo` or `register` |
| `-k PATH` | `K6` | `k6` | Path to k6 binary |
### Register Modes
- **`demo`** (default): Uses the `create-demo-profile` RPC endpoint. Requires the `demo-users` feature flag to be enabled on the backend. Fastest for testing.
- **`register`**: Uses the full two-step registration flow (`prepare-register-profile` + `register-profile`). Works without any feature flags but is slower.
## Shared Client (`lib/penpot-client.js`)
The shared client module wraps the Penpot backend RPC API using plain JSON (not Transit). Key features:
- **JSON transport**: Uses `Content-Type: application/json` for POST bodies and `Accept: application/json` (or `_fmt=json` for GET) for responses.
- **Cookie-based auth**: k6 automatically manages session cookies per VU.
- **Session headers**: Generates `x-session-id` and `x-external-session-id` UUIDs per VU.
- **Tagged metrics**: Every request is tagged with `rpc_command` for k6 metric slicing.
## Results
Test results are written to `results/<timestamp>/` as JSON. k6 also prints a summary to stdout with percentile breakdowns per RPC command.
## Thresholds
The lifecycle script includes built-in thresholds that will cause k6 to exit with a non-zero code if exceeded:
- `http_req_duration p95 < 5000ms` (global)
- `http_req_failed < 1%` (global)
- Per-command thresholds for login, profile, project, file, and update operations
## Adding New Flows
To add a new test flow:
1. Create `scripts/<flow-name>.js`
2. Import the shared client: `import { createClient } from "../lib/penpot-client.js";`
3. Implement the flow using the client methods
4. Add a command in `run.sh`
## Architecture Notes
- The backend supports both Transit JSON and plain JSON. This test suite uses **plain JSON** for simplicity (no Transit encoder needed in k6).
- JSON request keys are in **kebab-case** (matching Clojure conventions). JSON response keys are in **camelCase** (backend's default JSON encoding).
- `update-file` sends the `id` parameter both in the query string and in the POST body, matching the frontend's behavior.
- The backend uses optimistic concurrency control (`revn`) for file updates. The test retries once on conflict.

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 97 B

View File

@ -0,0 +1,525 @@
// Penpot k6 HTTP Client
//
// Shared module that wraps the Penpot backend RPC API using plain JSON.
// The backend supports `application/json` request bodies (kebab-case keys)
// and `application/json` responses (camelCase keys) via Accept header or _fmt=json.
//
// Authentication is cookie-based: login-with-password sets a session cookie,
// and all subsequent requests include it automatically via the k6 cookie jar.
import http from "k6/http";
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js";
// ---------------------------------------------------------------------------
// Configuration
// ---------------------------------------------------------------------------
/**
* Creates a new Penpot client instance.
*
* @param {string} baseUrl - The base URL of the Penpot backend (e.g., "http://localhost:6060")
* @returns {object} Client instance with RPC methods
*/
export function createClient(baseUrl) {
// Per-VU session identifiers — consistent across all requests within one VU iteration
const sessionId = uuidv4();
const externalSessionId = uuidv4();
const defaultHeaders = {
"Accept": "application/json",
"x-session-id": sessionId,
"x-external-session-id": externalSessionId,
"x-event-origin": "perf-test",
"x-client": "penpot-perf/1.0",
};
// k6 automatically manages cookies per VU when `cookies` are returned by the server.
// We use the default cookie jar which is per-VU.
/**
* Make an RPC call to the Penpot backend.
*
* GET requests: params go as query parameters, response is JSON via _fmt=json.
* POST requests: params go as JSON body, response is JSON via Accept header.
*
* @param {string} method - HTTP method ("GET" or "POST")
* @param {string} command - RPC command name (e.g., "login-with-password")
* @param {object} params - Parameters for the RPC call
* @param {object} [opts] - Additional options
* @param {string} [opts.tag] - k6 metric tag for this request
* @returns {object} k6 Response object with parsed JSON body
*/
function rpc(method, command, params = {}, opts = {}) {
const url = `${baseUrl}/api/main/methods/${command}`;
const tag = opts.tag || command;
const tags = {
rpc_command: tag,
};
if (method === "GET") {
// GET requests: params go as query string, add _fmt=json for JSON response
const queryParams = { ...params, _fmt: "json" };
const qs = Object.entries(queryParams)
.filter(([, v]) => v !== undefined && v !== null)
.map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`)
.join("&");
const fullUrl = qs ? `${url}?${qs}` : url;
return http.get(fullUrl, {
headers: defaultHeaders,
tags,
});
} else {
// POST requests: params go as JSON body
const headers = {
...defaultHeaders,
"Content-Type": "application/json",
};
return http.post(url, JSON.stringify(params), {
headers,
tags,
});
}
}
/**
* Login with email and password.
* Returns the profile data on success. The session cookie is stored
* automatically by k6's cookie jar.
*
* @param {string} email
* @param {string} password
* @returns {object} Parsed response { status, body }
*/
function login(email, password) {
const res = rpc("POST", "login-with-password", {
email,
password,
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Get the current user's profile (requires prior login).
*
* @returns {object} Parsed response { status, body }
*/
function getProfile() {
const res = rpc("GET", "get-profile");
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Get all teams for the current user.
*
* @returns {object} Parsed response { status, body }
*/
function getTeams() {
const res = rpc("GET", "get-teams");
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Create a new team.
*
* @param {string} name - Team name
* @returns {object} Parsed response { status, body }
*/
function createTeam(name) {
const res = rpc("POST", "create-team", { name });
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Get projects for a team.
*
* @param {string} teamId - Team UUID
* @returns {object} Parsed response { status, body }
*/
function getProjects(teamId) {
const res = rpc("GET", "get-projects", { "team-id": teamId });
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Create a new project.
*
* @param {string} teamId - Team UUID
* @param {string} name - Project name
* @returns {object} Parsed response { status, body }
*/
function createProject(teamId, name) {
const res = rpc("POST", "create-project", {
"team-id": teamId,
name,
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Create a new file in a project.
*
* @param {string} projectId - Project UUID
* @param {string} name - File name
* @returns {object} Parsed response { status, body }
*/
function createFile(projectId, name) {
const res = rpc("POST", "create-file", {
"project-id": projectId,
name,
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Get a file by ID.
*
* @param {string} fileId - File UUID
* @returns {object} Parsed response { status, body }
*/
function getFile(fileId) {
const res = rpc("GET", "get-file", {
id: fileId,
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Update a file with changes.
*
* The backend uses optimistic concurrency control via `revn`.
* If a conflict occurs (status 400 with :revn-conflict), the caller
* should retry with the latest revn from getFile().
*
* @param {string} fileId - File UUID
* @param {number} revn - Current file revision number
* @param {number} vern - Current file version number
* @param {string} sessionId - Client session ID (UUID)
* @param {Array} changes - Array of change objects
* @returns {object} Parsed response { status, body }
*/
function updateFile(fileId, revn, vern, changesSessionId, changes) {
const params = {
id: fileId,
revn: revn,
vern: vern,
"session-id": changesSessionId,
origin: "workspace",
"created-at": new Date().toISOString(),
"commit-id": uuidv4(),
changes: changes,
};
// update-file uses POST with id also as query param (per frontend convention)
const url = `${baseUrl}/api/main/methods/update-file?id=${encodeURIComponent(fileId)}`;
const headers = {
...defaultHeaders,
"Content-Type": "application/json",
};
const res = http.post(url, JSON.stringify(params), {
headers,
tags: { rpc_command: "update-file" },
});
let body = null;
try {
if (res.body && res.body.length > 0) {
body = res.json();
}
} catch (e) {
// body may not be JSON
}
return {
status: res.status,
body: body,
raw: res,
};
}
/**
* Upload a file media object using direct multipart upload.
*
* @param {string} fileId - File UUID
* @param {Uint8Array} fileBytes - The file content
* @param {string} fileName - The file name
* @param {string} mimeType - MIME type (e.g., "image/png")
* @returns {object} Parsed response { status, body }
*/
function uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType) {
const url = `${baseUrl}/api/main/methods/upload-file-media-object`;
const headers = {
...defaultHeaders,
// No Content-Type — k6 sets it automatically for multipart/form-data
};
const formData = {
"file-id": fileId,
"is-local": "true",
name: fileName,
content: http.file(fileBytes, fileName, mimeType),
};
const res = http.post(url, formData, {
headers,
tags: { rpc_command: "upload-file-media-object" },
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
// -----------------------------------------------------------------------
// Chunked upload
// -----------------------------------------------------------------------
/**
* Create an upload session for chunked uploads.
*
* @param {number} totalChunks - Number of chunks
* @returns {object} { status, sessionId }
*/
function createUploadSession(totalChunks) {
const res = rpc("POST", "create-upload-session", {
"total-chunks": totalChunks,
});
const body = res.status === 200 ? res.json() : null;
return {
status: res.status,
sessionId: body ? body.sessionId : null,
raw: res,
};
}
/**
* Upload a single chunk within an upload session.
*
* @param {string} sessionId - Upload session UUID
* @param {number} index - Chunk index (0-based)
* @param {Uint8Array} chunkBytes - The chunk content
* @param {string} fileName - Original file name
* @param {string} mimeType - MIME type
* @returns {object} Parsed response { status, body }
*/
function uploadChunk(sessionId, index, chunkBytes, fileName, mimeType) {
const url = `${baseUrl}/api/main/methods/upload-chunk`;
const headers = {
...defaultHeaders,
};
const formData = {
"session-id": sessionId,
index: String(index),
content: http.file(chunkBytes, fileName, mimeType),
};
const res = http.post(url, formData, {
headers,
tags: { rpc_command: "upload-chunk" },
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
/**
* Assemble all uploaded chunks into a final media object.
*
* @param {string} sessionId - Upload session UUID
* @param {string} fileId - File UUID
* @param {string} name - Media object name
* @param {boolean} isLocal - Whether the object is local to the file
* @param {string} mimeType - MIME type (e.g., "image/png")
* @returns {object} Parsed response { status, body }
*/
function assembleFileMediaObject(sessionId, fileId, name, isLocal, mimeType) {
const res = rpc("POST", "assemble-file-media-object", {
"session-id": sessionId,
"file-id": fileId,
name: name,
"is-local": isLocal,
mtype: mimeType,
});
return {
status: res.status,
body: res.status === 200 ? res.json() : null,
raw: res,
};
}
// -----------------------------------------------------------------------
// Smart upload — picks direct or chunked based on file size
// -----------------------------------------------------------------------
// Chunk size threshold: files larger than this use chunked upload.
// The actual chunk size is irrelevant to the backend; this controls
// which upload path is exercised.
const CHUNK_SIZE = 50 * 1024; // 50 KB
/**
* Upload a file media object, automatically selecting direct or chunked
* upload based on file size.
*
* Files <= CHUNK_SIZE use direct multipart upload.
* Files > CHUNK_SIZE use chunked upload (create-upload-session
* upload-chunk × N assemble-file-media-object).
*
* @param {string} fileId - File UUID
* @param {Uint8Array} fileBytes - The file content
* @param {string} fileName - The file name
* @param {string} mimeType - MIME type (e.g., "image/png")
* @returns {object} Parsed response { status, body }
*/
function uploadFileMediaObject(fileId, fileBytes, fileName, mimeType) {
if (fileBytes.byteLength <= CHUNK_SIZE) {
return uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType);
}
// Chunked upload path
const totalChunks = Math.ceil(fileBytes.byteLength / CHUNK_SIZE);
const sessionRes = createUploadSession(totalChunks);
if (sessionRes.status !== 200) {
return { status: sessionRes.status, body: null, raw: sessionRes.raw };
}
const uploadSessionId = sessionRes.sessionId;
for (let i = 0; i < totalChunks; i++) {
const start = i * CHUNK_SIZE;
const end = Math.min(start + CHUNK_SIZE, fileBytes.byteLength);
const chunk = fileBytes.slice(start, end);
const chunkRes = uploadChunk(uploadSessionId, i, chunk, fileName, mimeType);
if (chunkRes.status !== 200) {
return { status: chunkRes.status, body: null, raw: chunkRes.raw };
}
}
return assembleFileMediaObject(uploadSessionId, fileId, fileName, true, mimeType);
}
/**
* Delete a file.
*
* @param {string} fileId - File UUID
* @returns {object} Parsed response { status }
*/
function deleteFile(fileId) {
const res = rpc("POST", "delete-file", { id: fileId });
return {
status: res.status,
raw: res,
};
}
/**
* Delete a project.
*
* @param {string} projectId - Project UUID
* @returns {object} Parsed response { status }
*/
function deleteProject(projectId) {
const res = rpc("POST", "delete-project", { id: projectId });
return {
status: res.status,
raw: res,
};
}
/**
* Delete a team.
*
* @param {string} teamId - Team UUID
* @returns {object} Parsed response { status }
*/
function deleteTeam(teamId) {
const res = rpc("POST", "delete-team", { id: teamId });
return {
status: res.status,
raw: res,
};
}
/**
* Logout the current user.
*
* @param {string} profileId - Profile UUID
* @returns {object} Parsed response { status }
*/
function logout(profileId) {
const res = rpc("POST", "logout", { "profile-id": profileId });
return {
status: res.status,
raw: res,
};
}
// Return the client interface
return {
sessionId,
externalSessionId,
rpc,
login,
getProfile,
getTeams,
createTeam,
getProjects,
createProject,
createFile,
getFile,
updateFile,
uploadFileMediaObject,
uploadFileMediaObjectDirect,
createUploadSession,
uploadChunk,
assembleFileMediaObject,
deleteFile,
deleteProject,
deleteTeam,
logout,
};
}

160
performance/run.sh Executable file
View File

@ -0,0 +1,160 @@
#!/usr/bin/env bash
#
# Penpot Performance Tests
#
# k6-based load/performance test suite for the Penpot backend.
#
# Prerequisites:
# - k6 (https://k6.io/) installed and in PATH
# - A running Penpot backend (local devenv or remote)
#
# Usage:
# ./run.sh smoke # 1 VU, 1 iteration smoke test
# ./run.sh lifecycle # 1 VU, 1 iteration (defaults)
# ./run.sh lifecycle -v 10 -n 5 # 10 VUs, 5 iterations each
# ./run.sh lifecycle -u http://remote:6060 -m register -v 5
# ./run.sh clean # Remove test results
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ---------------------------------------------------------------------------
# Defaults
# ---------------------------------------------------------------------------
BASE_URL="${PENPOT_BASE_URL:-http://localhost:6060}"
VUS=1
ITER=1
REGISTER_MODE="${PENPOT_REGISTER_MODE:-demo}"
K6="${K6:-k6}"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
usage() {
cat <<EOF
Penpot Performance Tests
Usage:
$(basename "$0") <command> [options]
Commands:
smoke 1 VU, 1 iteration smoke test (forces demo mode)
lifecycle Full user lifecycle test
clean Remove test results
help Show this help
Options (lifecycle only):
-u URL Backend base URL (default: $BASE_URL)
-v NUM Number of virtual users (default: $VUS)
-n NUM Iterations per VU (default: $ITER)
-m MODE Register mode: 'demo' or 'register' (default: $REGISTER_MODE)
-k PATH Path to k6 binary (default: $K6)
Environment variables:
PENPOT_BASE_URL Same as -u
PENPOT_REGISTER_MODE Same as -m
K6 Same as -k
Examples:
$(basename "$0") smoke
$(basename "$0") lifecycle -v 10 -n 5
$(basename "$0") lifecycle -m register -v 5 -n 1
$(basename "$0") lifecycle -u https://penpot.example.com
EOF
}
check_k6() {
if ! command -v "$K6" &>/dev/null; then
echo "Error: k6 not found at '$K6'" >&2
echo "Install from https://k6.io/docs/get-started/installation/" >&2
exit 1
fi
}
run_k6() {
local results_dir="$SCRIPT_DIR/results/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$results_dir"
echo "Penpot Performance Test"
echo " Base URL: $BASE_URL"
echo " Register mode: $REGISTER_MODE"
echo " VUs: $VUS"
echo " Iterations: $ITER"
echo " Results: $results_dir"
echo ""
"$K6" run \
--env "PENPOT_BASE_URL=$BASE_URL" \
--env "PENPOT_REGISTER_MODE=$REGISTER_MODE" \
--vus "$VUS" \
--iterations "$ITER" \
--out "json=$results_dir/k6-summary.json" \
"$SCRIPT_DIR/scripts/lifecycle.js"
}
# ---------------------------------------------------------------------------
# Commands
# ---------------------------------------------------------------------------
cmd_smoke() {
check_k6
REGISTER_MODE=demo
VUS=1
ITER=1
run_k6
}
cmd_lifecycle() {
# Parse options
while getopts "u:v:n:m:k:h" opt; do
case "$opt" in
u) BASE_URL="$OPTARG" ;;
v) VUS="$OPTARG" ;;
n) ITER="$OPTARG" ;;
m) REGISTER_MODE="$OPTARG" ;;
k) K6="$OPTARG" ;;
h) usage; exit 0 ;;
*) usage >&2; exit 1 ;;
esac
done
check_k6
run_k6
}
cmd_clean() {
local results_dir="$SCRIPT_DIR/results"
if [[ -d "$results_dir" ]]; then
rm -rf "$results_dir"
echo "Cleaned $results_dir"
else
echo "Nothing to clean"
fi
}
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
if [[ $# -lt 1 ]]; then
usage >&2
exit 1
fi
command="$1"
shift
case "$command" in
smoke) cmd_smoke "$@" ;;
lifecycle) cmd_lifecycle "$@" ;;
clean) cmd_clean "$@" ;;
help|-h|--help) usage ;;
*)
echo "Unknown command: $command" >&2
usage >&2
exit 1
;;
esac

View File

@ -0,0 +1,417 @@
// Lifecycle Performance Test
//
// Simulates a realistic user lifecycle from registration through CRUD operations.
// Each VU performs the full flow independently, creating its own artifacts.
//
// Flow:
// 1. Register (via demo profile or prepare+register)
// 2. Login
// 3. Get profile & teams
// 4. Create project
// 5. Create file
// 6. Get file
// 7. Update file (add a shape)
// 8. Upload image to file
// 9. Delete file
// 10. Delete project
// 11. Delete team
//
// Usage:
// k6 run scripts/lifecycle.js
// k6 run --env PENPOT_BASE_URL=http://localhost:6060 scripts/lifecycle.js
// k6 run --env PENPOT_REGISTER_MODE=register scripts/lifecycle.js
import { check, sleep, fail } from "k6";
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js";
import { createClient } from "../lib/penpot-client.js";
// ---------------------------------------------------------------------------
// Configuration
// ---------------------------------------------------------------------------
const BASE_URL = __ENV.PENPOT_BASE_URL || "http://localhost:6060";
// "demo" = use create-demo-profile (requires demo-users flag)
// "register" = use prepare-register-profile + register-profile
const REGISTER_MODE = __ENV.PENPOT_REGISTER_MODE || "demo";
// k6 options — smoke test defaults (1 VU, 1 iteration)
export const options = {
scenarios: {
lifecycle: {
executor: "per-vu-iterations",
vus: 1,
iterations: 1,
maxDuration: "2m",
},
},
thresholds: {
http_req_duration: ["p(95)<5000"],
http_req_failed: ["rate<0.01"],
"http_req_duration{rpc_command:login-with-password}": ["p(95)<1000"],
"http_req_duration{rpc_command:get-profile}": ["p(95)<500"],
"http_req_duration{rpc_command:create-project}": ["p(95)<1000"],
"http_req_duration{rpc_command:create-file}": ["p(95)<1000"],
"http_req_duration{rpc_command:get-file}": ["p(95)<500"],
"http_req_duration{rpc_command:update-file}": ["p(95)<2000"],
"http_req_duration{rpc_command:delete-file}": ["p(95)<1000"],
},
};
// ---------------------------------------------------------------------------
// Test Data
// ---------------------------------------------------------------------------
// Load test PNG fixtures — small uses direct upload, large uses chunked upload
const testImageSmall = open("../fixtures/test-small.png", "b");
const testImageLarge = open("../fixtures/test-large.png", "b");
// A minimal "add-obj" change payload for update-file.
// This adds a simple rectangle shape to the first page.
// All object properties use camelCase — the backend's JSON parser
// (json/read-kebab-key) converts camelCase to kebab-case keywords automatically.
function makeAddRectChange(pageId) {
const shapeId = uuidv4();
const x = 100;
const y = 100;
const w = 200;
const h = 150;
return {
type: "add-obj",
pageId: pageId,
id: shapeId,
frameId: pageId, // required: the frame this shape belongs to
parentId: pageId, // root frame is the page itself
obj: {
id: shapeId,
type: "rect",
name: "Perf Test Rect",
x: x,
y: y,
width: w,
height: h,
fillColor: "#ff0000",
fillOpacity: 1,
rotation: 0,
hidden: false,
locked: false,
// Required base attrs
selrect: {
x: x,
y: y,
width: w,
height: h,
x1: x,
y1: y,
x2: x + w,
y2: y + h,
},
points: [
{ x: x, y: y },
{ x: x + w, y: y },
{ x: x + w, y: y + h },
{ x: x, y: y + h },
],
transform: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 },
transformInverse: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 },
parentId: pageId,
frameId: pageId,
},
};
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function assertOk(res, label) {
const ok = check(res, {
[`${label} — status is 2xx`]: (r) => r.status >= 200 && r.status < 300,
});
if (!ok) {
// Try to get the response body for debugging
let bodyStr = "";
try {
if (res.raw && res.raw.body) {
bodyStr = typeof res.raw.body === "string"
? res.raw.body.substring(0, 500)
: JSON.stringify(res.raw.body).substring(0, 500);
} else if (res.body) {
bodyStr = JSON.stringify(res.body).substring(0, 500);
}
} catch (e) {
bodyStr = "(could not read body)";
}
console.error(
`FAIL: ${label} — status=${res.status} body=${bodyStr}`
);
}
return ok;
}
// ---------------------------------------------------------------------------
// Setup — runs once before VUs start
// ---------------------------------------------------------------------------
export function setup() {
console.log(`Penpot Lifecycle Test`);
console.log(` Base URL: ${BASE_URL}`);
console.log(` Register mode: ${REGISTER_MODE}`);
console.log(``);
// Verify the backend is reachable
const client = createClient(BASE_URL);
const res = client.getProfile();
// We expect 401/403 (not logged in) — anything else means the backend is down
if (res.status === 0) {
fail(`Backend unreachable at ${BASE_URL}`);
}
return { baseUrl: BASE_URL, registerMode: REGISTER_MODE };
}
// ---------------------------------------------------------------------------
// Main VU Function
// ---------------------------------------------------------------------------
export default function (data) {
const client = createClient(data.baseUrl);
// ---- Step 0: Create a user account ----
let email, password;
if (data.registerMode === "demo") {
// Demo mode: create-demo-profile returns email + password
const demoRes = client.rpc("POST", "create-demo-profile", {});
if (!assertOk(demoRes, "create-demo-profile")) {
fail("Failed to create demo profile");
}
const demoBody = demoRes.json();
email = demoBody.email;
password = demoBody.password;
console.log(`VU ${__VU}: Created demo profile: ${email}`);
} else {
// Register mode: prepare + register
email = `perf-${uuidv4()}@test.local`;
password = "PerfTest1234!";
const fullname = `Perf User ${__VU}`;
const prepareRes = client.rpc("POST", "prepare-register-profile", {
fullname,
email,
password,
});
if (!assertOk(prepareRes, "prepare-register-profile")) {
fail("Failed to prepare registration");
}
const prepareBody = prepareRes.json();
const token = prepareBody.token;
const registerRes = client.rpc("POST", "register-profile", {
token,
});
if (!assertOk(registerRes, "register-profile")) {
fail("Failed to register profile");
}
console.log(`VU ${__VU}: Registered profile: ${email}`);
}
sleep(1);
// ---- Step 1: Login ----
const loginRes = client.login(email, password);
if (!assertOk(loginRes, "login-with-password")) {
fail("Login failed");
}
const profile = loginRes.body;
const profileId = profile.id;
console.log(`VU ${__VU}: Logged in, profile-id=${profileId}`);
sleep(1);
// ---- Step 2: Get profile ----
const profileRes = client.getProfile();
if (!assertOk(profileRes, "get-profile")) {
fail("get-profile failed");
}
sleep(0.5);
// ---- Step 3: Get teams ----
const teamsRes = client.getTeams();
if (!assertOk(teamsRes, "get-teams")) {
fail("get-teams failed");
}
const teams = teamsRes.body;
// teams is an array; the user has a default team from registration
const defaultTeamId = Array.isArray(teams) && teams.length > 0
? teams[0].id
: null;
if (!defaultTeamId) {
fail("No default team found after registration");
}
sleep(0.5);
// ---- Step 4: Create a project ----
const projectName = `Perf Project ${uuidv4().substring(0, 8)}`;
const projectRes = client.createProject(defaultTeamId, projectName);
if (!assertOk(projectRes, "create-project")) {
fail("create-project failed");
}
const project = projectRes.body;
const projectId = project.id;
console.log(`VU ${__VU}: Created project: ${projectId}`);
sleep(1);
// ---- Step 5: Create a file ----
const fileName = `Perf File ${uuidv4().substring(0, 8)}`;
const fileRes = client.createFile(projectId, fileName);
if (!assertOk(fileRes, "create-file")) {
fail("create-file failed");
}
const file = fileRes.body;
const fileId = file.id;
console.log(`VU ${__VU}: Created file: ${fileId}`);
sleep(1);
// ---- Step 6: Get the file (to read revn, vern, page-id) ----
const getFileRes = client.getFile(fileId);
if (!assertOk(getFileRes, "get-file")) {
fail("get-file failed");
}
const fileData = getFileRes.body;
const revn = fileData.revn;
const vern = fileData.vern;
// Extract the first page-id from the file data
// fileData.data.pages is an array of page UUIDs
// fileData.data.pages-index is a map of page-id -> page objects
let pageId = null;
if (fileData.data && fileData.data.pages && fileData.data.pages.length > 0) {
pageId = fileData.data.pages[0];
}
if (!pageId) {
console.warn(`VU ${__VU}: Could not find page-id in file data, skipping update-file`);
}
sleep(1);
// ---- Step 7: Update file (add a rectangle shape) ----
let updateOk = false;
if (pageId) {
const changes = [makeAddRectChange(pageId)];
const updateRes = client.updateFile(fileId, revn, vern, client.sessionId, changes);
if (updateRes.status === 200) {
updateOk = true;
console.log(`VU ${__VU}: Updated file successfully`);
} else {
// Check for revn conflict — retry once
const body = updateRes.body;
console.error(`VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 500)}`);
const isRevnConflict =
body && (body.code === "revn-conflict" || body.type === "revn-conflict");
if (isRevnConflict) {
console.log(`VU ${__VU}: Revn conflict, retrying...`);
// Fetch latest file state
const retryFileRes = client.getFile(fileId);
if (assertOk(retryFileRes, "get-file (retry)")) {
const retryData = retryFileRes.body;
const retryRes = client.updateFile(
fileId,
retryData.revn,
retryData.vern,
client.sessionId,
changes
);
if (assertOk(retryRes, "update-file (retry)")) {
updateOk = true;
console.log(`VU ${__VU}: Updated file on retry`);
}
}
} else {
console.error(
`VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 300)}`
);
}
}
}
sleep(1);
// ---- Step 8: Upload images to the file ----
// Small image (97 B) uses direct multipart upload.
// Large image (120 KB) uses chunked upload (create-session → upload-chunk × N → assemble).
if (testImageSmall && testImageSmall.byteLength > 0) {
const uploadRes = client.uploadFileMediaObject(
fileId,
testImageSmall,
"test-small.png",
"image/png"
);
if (assertOk(uploadRes, "upload-file-media-object (direct)")) {
console.log(`VU ${__VU}: Uploaded small image (direct)`);
}
}
sleep(0.5);
if (testImageLarge && testImageLarge.byteLength > 0) {
const uploadRes = client.uploadFileMediaObject(
fileId,
testImageLarge,
"test-large.png",
"image/png"
);
if (assertOk(uploadRes, "upload-file-media-object (chunked)")) {
console.log(`VU ${__VU}: Uploaded large image (chunked)`);
}
}
sleep(1);
// ---- Step 9: Delete the file ----
const deleteFileRes = client.deleteFile(fileId);
if (assertOk(deleteFileRes, "delete-file")) {
console.log(`VU ${__VU}: Deleted file: ${fileId}`);
}
sleep(0.5);
// ---- Step 10: Delete the project ----
const deleteProjectRes = client.deleteProject(projectId);
if (assertOk(deleteProjectRes, "delete-project")) {
console.log(`VU ${__VU}: Deleted project: ${projectId}`);
}
sleep(0.5);
// ---- Step 11: Delete the team ----
// Note: We only delete the team if it's NOT the default team.
// The default team cannot be deleted (or may cause errors).
// For this test, we skip team deletion to avoid errors.
// In a real scenario, we'd create a separate team and delete that.
console.log(`VU ${__VU}: Skipping team deletion (using default team)`);
sleep(0.5);
// ---- Step 12: Logout ----
const logoutRes = client.logout(profileId);
console.log(`VU ${__VU}: Logout status: ${logoutRes.status}`);
console.log(`VU ${__VU}: Lifecycle complete`);
}
// ---------------------------------------------------------------------------
// Teardown — runs once after all VUs finish
// ---------------------------------------------------------------------------
export function teardown(data) {
console.log("Lifecycle test complete.");
}

View File

@ -0,0 +1,494 @@
# Backend Performance Test Plan
**Context:** Build a k6-based load/performance test suite that simulates realistic browser-to-backend HTTP flows for distinct Penpot user operations. The goal is to measure backend impact (latency, throughput, error rates, resource saturation) under synthetic user load. **Browser rendering performance is explicitly out of scope.** WebSocket testing is deferred.
**Date:** 2026-06-12
**Validated Requirements:**
- Tool: **k6** (confirmed).
- Environment: flexible — local devenv first, then remote staging/perf.
- Target scale: **1000 concurrent VUs** (ramping from lower baselines).
- Flows: **realistic CRUD lifecycle** — create, edit, upload, delete. Must include **image upload** and **font upload**.
- `update-file` is important but difficult because it requires **23 concurrent users editing the same file**, and **file size matters**.
- WebSocket: **deferred**.
---
## Current Progress
### Completed (2026-06-12)
Phase 1 is done. Phase 2 has the first flow (lifecycle) implemented and validated.
**What was built:**
```
performance/
├── run.sh # Bash runner script (smoke, lifecycle, clean, help)
├── README.md # Usage docs, configuration, architecture notes
├── lib/
│ └── penpot-client.js # 388 lines — shared k6 HTTP client module
├── scripts/
│ └── lifecycle.js # 398 lines — full user lifecycle test
├── fixtures/
│ ├── test-small.png # 97 B
│ ├── test-medium.png # 30 KB
│ └── test-large.png # 120 KB
├── results/ # k6 JSON output (gitignored)
└── baselines/ # for regression baselines
```
**What the lifecycle test covers (all passing):**
| Step | RPC Command | Method | Observed Latency |
|------|------------|--------|-----------------|
| 1. Register | `create-demo-profile` | POST | ~136ms |
| 2. Login | `login-with-password` | POST | ~136ms |
| 3. Get profile | `get-profile` | GET | ~7ms |
| 4. Get teams | `get-teams` | GET | ~6ms |
| 5. Create project | `create-project` | POST | ~9ms |
| 6. Create file | `create-file` | POST | ~22ms |
| 7. Get file | `get-file` | GET | ~12ms |
| 8. Update file | `update-file` | POST | ~25ms |
| 9. Upload image | `upload-file-media-object` | POST (multipart) | ~7ms |
| 10. Delete file | `delete-file` | POST | ~12ms |
| 11. Delete project | `delete-project` | POST | ~5ms |
| 12. Logout | `logout` | POST | ~4ms |
Smoke test result: **10/10 checks pass, 0% failure rate, ~10s total iteration**.
**Key discoveries during implementation:**
1. **JSON transport works.** The backend accepts `Content-Type: application/json` for POST bodies (kebab-case keys auto-converted) and returns `application/json` responses (camelCase keys) via `Accept: application/json` or `_fmt=json` query param. No Transit encoder needed in k6.
2. **`create-file` `features` param.** Sending `features: []` (empty array) causes a 400 validation error. Omit the field entirely — it's optional in the schema (`backend/src/app/rpc/commands/files_create.clj`).
3. **`update-file` `changes` payload — shape schema is strict.** The `add-obj` change requires a fully valid shape object. Minimum required fields beyond the basics:
- `selrect`: `{x, y, width, height, x1, y1, x2, y2}`
- `points`: array of 4 `{x, y}` corner points
- `transform` / `transform-inverse`: identity matrix `{a:1, b:0, c:0, d:1, e:0, f:0}`
- `parent-id` and `frame-id` inside the `obj`
- `frame-id` at the top level of the change object itself (required, not optional)
- Schema defined in `common/src/app/common/files/changes.cljc` (line 189) and `common/src/app/common/types/shape.cljc` (line 165).
4. **`update-file` URL convention.** The frontend sends `id` both as a query param and in the POST body. The URL is `POST /api/main/methods/update-file?id=<uuid>`.
5. **Two registration modes work:**
- `demo` mode: `create-demo-profile` — fast, requires `demo-users` feature flag.
- `register` mode: `prepare-register-profile` + `register-profile` — two-step flow, works without flags.
6. **k6 installed at** `/home/penpot/.local/bin/k6` (v0.56.0). Not in default PATH; use `PATH="/home/penpot/.local/bin:$PATH"` or set `K6` env var.
### Remaining Work
| Phase | Status | Next Actions |
|-------|--------|-------------|
| Phase 1 Discovery & Tooling | **Done** | — |
| Phase 2 Core HTTP Flows | **Partial** — lifecycle.js done | Write `workspace-open.js`, `workspace-edit.js`, `media-upload.js`, `font-upload.js`, `viewer.js`, `export.js` |
| Phase 2 Data seeding | **Not started** | Create `seed-data.js` for 100+ user pool |
| Phase 3 Scenarios | **Not started** | Define multi-scenario k6 config mixing flows |
| Phase 4 Advanced update-file | **Not started** | File size tiers, concurrent editing matrix |
| Phase 5 CI & Reporting | **Not started** | Grafana dashboards, regression guard |
### Immediate Next Steps
1. Write `workspace-open.js` — read-heavy flow (get-file, get-file-libraries, thumbnails).
2. Write `workspace-edit.js` — write-heavy loop (get-file + update-file per VU, independent files).
3. Write `media-upload.js` — direct + chunked upload flows.
4. Write `font-upload.js` — chunked upload + create-font-variant.
5. Write `viewer.js` — get-view-only-bundle + comments.
6. Create `seed-data.js` — pre-seed 100+ users/teams/files for multi-VU runs.
7. Define scenario mix in `run.sh` (multi-scenario support with ramping VUs).
8. Add `--scenario` flag to `run.sh` to select individual flows or the full mix.
---
## Affected Modules
| Module | Why it is involved |
|--------|---------------------|
| `backend/` | Target system. All HTTP RPC (`/api/main/methods/*`), auth, storage, media processing, DB, and Prometheus metrics (`/metrics`). |
| `frontend/` | Source of truth for user request flows. We inspect `app.main.repo` (RPC client), `app.main.data.*` (user flows), and `app.main.data.persistence` (save semantics). |
| `common/` | Shared schemas, Transit helpers, and data structures. Used to understand valid `update-file` `changes` payloads. |
---
## Approach
### Phase 1 Discovery & Tooling (Days 12)
#### 1.1. Read the frontend RPC flows to build a request catalog
Inspect these files to map every user action to its RPC command:
- `frontend/src/app/main/repo.cljs` — HTTP client conventions (headers, retry, GET vs POST rules, query params, form-data, multipart).
- `frontend/src/app/main/data/dashboard.cljs` — Dashboard init (`get-projects`, `fetch-fonts`, `search-files`).
- `frontend/src/app/main/data/workspace.cljs` — Workspace init (`get-file`, `get-file-libraries`, `get-file-object-thumbnails`, `resolve-file` via `get-file-fragment`).
- `frontend/src/app/main/data/persistence.cljs` — File save flow (`update-file` with `changes`, `revn`, `session-id`, debounce/buffer logic).
- `frontend/src/app/main/data/viewer.cljs` — Viewer flow (`get-view-only-bundle`).
- `frontend/src/app/main/data/comments.cljs` — Comment thread fetch (`get-comment-threads`).
- `frontend/src/app/main/data/media.cljs` / `upload.cljs` — Media upload flows (`upload-file-media-object`, `create-upload-session`, `upload-chunk`, `assemble-file-media-object`).
- `frontend/src/app/main/data/fonts.cljs` — Font upload flow (`create-font-variant` with `:uploads` map).
- `frontend/src/app/main/data/team.cljs` — Team creation (`create-team`), invitation (`create-team-invitations`).
- `frontend/src/app/main/data/project.cljs` — Project creation (`create-project`).
**Goal:** produce a **Request Catalog** mapping user actions to RPC command names, HTTP methods, payload shapes, and required preconditions (e.g., `team-id`, `file-id`).
#### 1.2. Confirm JSON compatibility for the test harness
The backend middleware (`app.http.middleware`) supports `application/json` request bodies and `application/json` responses (via `_fmt=json` or `Accept: application/json`).
- **Action:** Send a manual `curl` to `POST /api/main/methods/login-with-password` with `Content-Type: application/json` and verify the response format.
- **Action:** Verify `GET /api/main/methods/get-profile` with `Accept: application/json` returns plain JSON.
- **Action:** Verify `POST /api/main/methods/update-file` with `Content-Type: application/json` and `_fmt=json` works.
- **Action:** Verify `POST /api/main/methods/upload-file-media-object` with `multipart/form-data` works (k6 supports this natively).
#### 1.3. Set up the load testing directory and shared client
Create a directory `performance/` at the repo root.
Install **k6** (`k6` CLI or Docker image).
Create a shared `penpot-client.js` module that wraps:
- `login(email, password)` → returns session cookie / token.
- `rpc(cmd, params, opts)` → builds the correct URL, headers, body, and query params.
- `uploadFileMediaObject(fileId, filePath, name)` → multipart upload.
- `createUploadSession(totalChunks)` → chunked upload setup.
- `uploadChunk(sessionId, index, chunkBytes)` → multipart chunk upload.
- `assembleFileMediaObject(sessionId, fileId, name, isLocal)` → finalize chunked upload.
**Headers to replicate (critical for backend telemetry and session binding):**
- `x-session-id`: generated UUID per VU (must be consistent across requests for the same session).
- `x-external-session-id`: generated UUID per VU.
- `x-event-origin`: a string origin (e.g., `"perf-test"`)
- `accept`: `application/json` (for HTTP-only load path)
- `content-type`: `application/json` (or `multipart/form-data` for uploads)
- `credentials: "include"` (for cookie jar)
#### 1.4. Data seeding strategy for 1000 VU scale
Creating 1000 users/teams/files *inside* the load test is too slow and will distort the results.
**Recommended approach:**
- **Setup Phase (k6 `setup()`):** Run a pre-test script that creates a shared pool of test artifacts.
- Use `login-with-password` with a fixture account (e.g., `profile1@example.com` / `123123` if fixtures exist).
- Create `N` teams, `N` projects, `N` files of varying sizes (see **File Size Tiers** below).
- Export the IDs into a JSON file that k6 `setup()` reads.
- **Alternative:** Use the backend REPL / fixtures (`app.cli.fixtures/run {:preset :small}`) to create fixture data, then export the IDs via a small Clojure script.
- **Data pool per VU:** Each VU picks a random user from the pool, or uses a dedicated user (e.g., VU #1`profile1@example.com`, VU #2`profile2@example.com`). For 1000 VUs, we need at least 1000 pre-seeded users.
- **Cleanup:** A post-test script can delete the seeded data, or we can use a dedicated perf DB that is reset between runs.
**Action:** Document the seeding procedure in `performance/README.md` and create a `seed-data.js` script.
---
### Phase 2 Core HTTP Flow Scripts (Days 35)
Create one k6 script per user flow. Each script:
- Uses `setup()` to read the shared data pool and log in.
- Uses `vu` iterations to simulate the flow.
- Tags every request with the RPC command name so k6 metrics are sliced by endpoint.
- Uses `check()` assertions for HTTP 200 and valid JSON structure.
#### Flow 1: Realistic User Lifecycle (`lifecycle.js`)
This is the primary realistic flow. Each VU performs a full lifecycle:
1. **Auth**
- `POST /api/main/methods/login-with-password``{email, password}`
- `GET /api/main/methods/get-profile`
- `GET /api/main/methods/get-teams`
2. **Create Team**
- `POST /api/main/methods/create-team``{name: "Perf Team <uuid>"}`
- `GET /api/main/methods/get-team?team-id=<id>`
3. **Create Project**
- `POST /api/main/methods/create-project``{team-id, name}`
- `GET /api/main/methods/get-project?id=<id>`
4. **Create File**
- `POST /api/main/methods/create-file``{project-id, name, features}`
- `GET /api/main/methods/get-file?id=<file-id>&features=<...>`
5. **Edit File (Simple Update)**
- `POST /api/main/methods/update-file` with a minimal `changes` payload.
- **Changes payload:** Use a simple change like `{:type "add-obj", :id "<uuid>", :page-id "<page-id>", :parent-id "<parent-id>", :obj {:type "rect", ...}}`. Inspect `app.common.files.changes` for the exact schema. For a load test, we only need the shape to be structurally valid; the backend validates it.
- **Revn tracking:** Fetch the file first, read `revn`, then send `revn` in the update. If a conflict occurs (`409` or `:revn-conflict` error), retry once with the latest `revn`.
6. **Upload Image (Direct)**
- `POST /api/main/methods/upload-file-media-object` (multipart)
- Payload: `file-id`, `is-local: true`, `name`, `content` (the file bytes).
- Use a small dummy PNG/SVG (e.g., 1 KB, 100 KB, 1 MB) stored in `performance/fixtures/`.
7. **Upload Image (Chunked)**
- `POST /api/main/methods/create-upload-session``{total-chunks: N}`
- Loop `N` times: `POST /api/main/methods/upload-chunk` (multipart, `session-id`, `index`, `chunk`)
- `POST /api/main/methods/assemble-file-media-object``{session-id, file-id, name, is-local}`
- Use a larger dummy file (e.g., 5 MB) to stress the chunked pipeline.
8. **Upload Font**
- `POST /api/main/methods/create-upload-session` (chunked, because fonts can be large)
- `POST /api/main/methods/upload-chunk` for each chunk
- `POST /api/main/methods/create-font-variant``{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": "<session-id>"}}`
- Use a small real TTF/OTF file from `performance/fixtures/`.
9. **Delete File**
- `DELETE /api/main/methods/delete-file` (verify the exact method name; it may be `update-file` with a deletion flag or a dedicated command). Inspect `frontend/src/app/main/data/dashboard.cljs` for the delete action.
10. **Delete Project**
- `DELETE /api/main/methods/delete-project?id=<id>`
11. **Delete Team**
- `POST /api/main/methods/delete-team?id=<id>`
12. **Logout**
- (Optional; session cookie expiry is usually sufficient)
**Pacing:** Add `sleep()` between steps to simulate realistic think time (e.g., 13 seconds between dashboard navigation, 35 seconds between edits).
#### Flow 2: Workspace Open (Read-heavy) (`workspace-open.js`)
For 1000 VUs, most will be read-only viewers or editors opening files.
1. Login (reuse token from `setup`).
2. `GET /api/main/methods/get-file?id=<file-id>&features=<...>`
3. `GET /api/main/methods/get-file-libraries?file-id=<file-id>`
4. For each library: `GET /api/main/methods/get-file?id=<lib-id>`
5. `GET /api/main/methods/get-file-object-thumbnails?file-id=<file-id>`
6. `GET /api/main/methods/get-file-data-for-thumbnail?file-id=<file-id>&page-id=<page-id>&object-id=<frame-id>`
**Data:** Use a pool of files of varying sizes (see **File Size Tiers**).
#### Flow 3: Workspace Edit (Write-heavy) (`workspace-edit.js`)
**Scenario A — Independent editors (default, easiest to scale):**
- Each VU creates its own file in `setup()`, or picks a dedicated file from the pool.
- Loop:
1. `GET /api/main/methods/get-file?id=<file-id>` (to refresh `revn`)
2. `POST /api/main/methods/update-file` with minimal changes
3. `sleep(3)`
- This measures the latency of the save path without concurrency conflicts.
**Scenario B — Concurrent editors (advanced, measures conflict resolution):**
- 23 VUs share the **same file ID**.
- Each VU:
1. `GET /api/main/methods/get-file?id=<file-id>` (to get latest `revn`)
2. `POST /api/main/methods/update-file` with changes
3. If `revn-conflict` (HTTP 400 or 409 with `:code :revn-conflict`), retry with the latest `revn`.
- **Problem:** k6 VUs are independent; they cannot easily share a mutable `revn` counter.
- **Solutions:**
1. **Optimistic concurrency:** Let conflicts happen naturally. Measure the conflict rate and retry latency. This is realistic for many-user editing.
2. **Shared state service:** Run a tiny Redis or in-memory service that stores the latest `revn` per file. VUs read/write it before each update. This adds coordination overhead but reduces conflicts.
3. **Sequential VU groups:** Use k6 `scenarios` with `executor: 'per-vu-iterations'` and a small shared file pool. Accept that some conflicts will occur and measure them as part of the benchmark.
- **Recommendation:** Start with **Solution 1** (optimistic). If the conflict rate is >10%, consider **Solution 2**.
**File Size Tiers for `update-file`:**
The backend `update-file` performance depends heavily on file data size (serialization, validation, pointer-map resolution, snapshotting).
| Tier | Size | How to create |
|------|------|---------------|
| Small | ~10 shapes | Create a file with a few rectangles. |
| Medium | ~100 shapes | Duplicate a page with many shapes. |
| Large | ~1000 shapes | Import a real-world design file or use a fixture. |
**Action:** Create a `create-file-fixture.js` helper that generates files of each tier via the `create-file` + `update-file` API (or by importing a `.penpot` file via the binfile import API if available).
#### Flow 4: Viewer (Read-heavy, anonymous or logged-in) (`viewer.js`)
1. Login (or use share-link token for anonymous).
2. `GET /api/main/methods/get-view-only-bundle?file-id=<id>&share-id=<id>&features=<...>`
3. `GET /api/main/methods/get-comment-threads?file-id=<id>&share-id=<id>`
#### Flow 5: Export (CPU/IO-heavy) (`export.js`)
1. Login.
2. `POST /api/export` with export payload.
- Inspect `frontend/src/app/main/data/export.cljs` for the exact payload shape.
- Common exports: `type: "png"`, `type: "svg"`, `type: "pdf"`.
- This hits the **exporter** service (Node.js/Playwright), which is a separate process. If the goal is to stress the **backend**, limit export tests or target the backend export queue endpoints.
#### Flow 6: Media Upload (Storage/IO-heavy) (`media-upload.js`)
1. Login.
2. Direct upload: `POST /api/main/methods/upload-file-media-object` (multipart, small PNG).
3. Chunked upload: `POST /api/main/methods/create-upload-session``upload-chunk` x N → `assemble-file-media-object` (large PNG).
4. URL-based upload: `POST /api/main/methods/create-file-media-object-from-url` (if a stable external image URL is available).
#### Flow 7: Font Upload (Storage/CPU-heavy) (`font-upload.js`)
1. Login.
2. `POST /api/main/methods/create-upload-session` (for the font file)
3. `POST /api/main/methods/upload-chunk` for each chunk
4. `POST /api/main/methods/create-font-variant``{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": "<session-id>"}}`
5. `GET /api/main/methods/get-font-variants?team-id=<id>`
---
### Phase 3 Scenarios & Orchestration (Day 6)
Define k6 `options.scenarios` that mix the flows to simulate realistic traffic.
**Example scenario mix for 1000 VUs:**
| Scenario | Script | VUs | Arrival Rate | Duration | Notes |
|----------|--------|-----|--------------|----------|-------|
| `lifecycle` | `lifecycle.js` | 100 | 1/s (ramp 0→100 over 5m) | 10m | Full CRUD, most realistic. |
| `workspace_open` | `workspace-open.js` | 400 | 5/s (ramp 0→400 over 5m) | 10m | Read-heavy, simulates many editors opening files. |
| `workspace_edit` | `workspace-edit.js` | 200 | 2/s (ramp 0→200 over 5m) | 10m | Write-heavy, independent files. |
| `workspace_edit_concurrent` | `workspace-edit.js` | 30 (10 groups of 3) | 0.5/s | 10m | 3 VUs per file, measures conflicts. |
| `viewer` | `viewer.js` | 200 | 3/s (ramp 0→200 over 5m) | 10m | Read-heavy, simulates public/private viewers. |
| `media_upload` | `media-upload.js` | 50 | 0.5/s | 10m | Storage stress. |
| `font_upload` | `font-upload.js` | 20 | 0.2/s | 10m | Font processing stress. |
**Thresholds:**
- `http_req_duration{p95} < 200ms` for `get-profile`, `get-teams`, `get-projects`.
- `http_req_duration{p95} < 500ms` for `get-file` (small), `search-files`.
- `http_req_duration{p95} < 2000ms` for `get-file` (large / 1000 shapes).
- `http_req_duration{p95} < 1000ms` for `update-file` (small).
- `http_req_duration{p95} < 3000ms` for `update-file` (large).
- `http_req_duration{p95} < 5000ms` for `upload-file-media-object` (1 MB).
- `http_req_duration{p95} < 10000ms` for `assemble-file-media-object` (5 MB chunked).
- `http_req_failed < 1%` globally.
- `http_req_failed{code:revn-conflict} < 5%` for `workspace_edit_concurrent`.
**Correlation with backend metrics:**
- Scrape `/metrics` before, during, and after the test.
- Key Prometheus metrics to watch:
- `rpc_main_timing_seconds` (histogram/summary, labeled by command name)
- `rpc_management_timing_seconds`
- `http_server_dispatch_timing_seconds`
- `websocket_active_connections` (if any WS is active)
- `websocket_messages_total`
- JVM hotspot metrics (`process_cpu_seconds_total`, `jvm_memory_bytes_used`, `jvm_threads_current`)
- HikariCP metrics (if exposed; check `com.zaxxer.hikari:type=Pool` via JMX or custom Prometheus exporter)
- PostgreSQL: `pg_stat_activity` count by state.
- Redis: `INFO` `connected_clients`, `used_memory`.
---
### Phase 4 Advanced `update-file` Testing (Days 78)
Because `update-file` is the core of the product and the user explicitly noted that **file size matters** and **concurrent editing is difficult**, we need a dedicated deep-dive.
#### 4.1. File Size Tiers
Create a `file-size-matrix.js` script that parameterizes the file size:
- `SMALL_FILE_ID`: 1 page, 10 shapes.
- `MEDIUM_FILE_ID`: 1 page, 100 shapes.
- `LARGE_FILE_ID`: 1 page, 500 shapes.
- `XLARGE_FILE_ID`: 1 page, 1000+ shapes, or a multi-page file.
Run `workspace-edit.js` against each tier separately and plot:
- `update-file` latency vs file size.
- `get-file` latency vs file size.
- Backend CPU and DB time vs file size.
#### 4.2. Concurrent Editing Strategy
**Problem:** `update-file` uses optimistic concurrency control (`revn`). If two users submit the same `revn`, the second gets a conflict.
**Backend behavior:**
- `update-file` acquires an advisory lock (`db/xact-lock! conn id`) on the file ID.
- It checks `revn` and `vern` conflicts.
- It processes changes, validates, updates the file, and sends notifications via `msgbus`.
- The transaction duration is what we want to measure.
**Test design for concurrent editing:**
- **Shared file pool:** Pre-create 50 files of `MEDIUM` size.
- **VU grouping:** Use k6 `scenarios` with `per-vu-iterations` or `shared-iterations`. Assign groups of 3 VUs to the same file ID.
- **Conflict measurement:** Do *not* synchronize `revn` between VUs. Let them race.
- Track `http_req_failed{code:revn-conflict}`.
- Track retry latency (if a VU retries after fetching the latest `revn`).
- This gives us the **natural conflict rate** under load, which is a realistic product metric.
- **If the natural conflict rate is too high (>20%):**
- Add a small `sleep()` jitter (0500 ms) between `get-file` and `update-file` to spread out the requests.
- Or, use a tiny shared counter (e.g., a small HTTP endpoint or Redis) that VUs read to get the "next" `revn`. This is less realistic but gives a cleaner latency measurement.
**Action:** Create `workspace-edit-concurrent.js` with the grouping logic and conflict-rate thresholds.
---
### Phase 5 CI Integration & Reporting (Days 910)
1. **Runner script (`run.sh`):**
- `./run.sh smoke` for a 1-VU, 1-iteration smoke test. ✅ Done
- `./run.sh lifecycle -v 100 -n 10` for the standard run.
- Add `--scenario` flag to run individual flows or the full mix.
2. **Output:**
- k6 JSON/CSV output to `performance/results/<timestamp>/`.
- Prometheus snapshot diff (before vs after).
- Grafana screenshot or dashboard export.
3. **Grafana Dashboard:**
- Panel: `p95 latency by RPC command` (from `rpc_main_timing_seconds`).
- Panel: `HTTP requests/sec` (from k6).
- Panel: `Error rate by command` (from k6).
- Panel: `DB connection pool` (if available).
- Panel: `JVM heap used`.
- Panel: `update-file conflict rate` (custom metric from k6).
- Panel: `File size vs latency` (from the matrix test).
4. **Regression guard:**
- Store baseline results in `performance/baselines/`.
- After any backend change, run the baseline scenario. If p95 increases by >20% for any critical command, fail the CI step.
---
## Risks & Considerations
| Risk | Mitigation |
|------|------------|
| **Scale: 1000 VUs creating data simultaneously will exhaust DB connection pool or storage quota.** | Pre-seed the data pool. Use a dedicated perf DB. Monitor `pg_stat_activity` and HikariCP metrics. |
| **Media upload (images/fonts) will saturate network I/O before the backend is stressed.** | Run the load test from the same datacenter/VPC as the backend. Use small dummy files for most tests; reserve large files for a dedicated storage-stress scenario. |
| **`update-file` conflicts under 1000 VUs may be so high that the test becomes a conflict test, not a latency test.** | Measure both. The conflict rate is itself a critical metric. If it is too high, we can add jitter or use independent files. |
| **Exporter service is a separate bottleneck.** | `export.js` should target the backend queue endpoint, not the full export pipeline, unless we want to test the exporter too. If exporter is in scope, run it as a separate scenario. |
| **Chunked upload creates many temporary DB rows (`upload_chunk` table).** | The backend has a `upload-session-gc` cron job. Ensure it runs after the test, or clean up manually. |
| **Font upload shells out to FontForge and WOFF tools.** | This is CPU-intensive and may be a bottleneck. Run font upload as a separate, low-VU scenario to measure the processing time without blocking other tests. |
| **Prometheus metrics may not expose DB pool wait time.** | Add a custom JMX exporter for HikariCP if needed, or query `pg_stat_activity` directly. |
| **Cleanup:** 1000 VUs creating teams/files will leave logical deletions or orphaned storage objects.** | Use a dedicated perf environment. Run a cleanup script after the test that deletes all seeded data via the RPC API. |
---
## Testing Strategy
### How to verify the test harness itself works
1. **Smoke test:** Run each k6 script with `1 VU, 1 iteration` against a local devenv. Verify all requests return `200` and the response body is valid JSON.
2. **Baseline run:** Run `workspace-open.js` with `10 VUs, 60 s` against a clean devenv. Record baseline p95 and p99 latencies.
3. **Regression guard:** After any backend change, re-run the baseline. If p95 increases by >20%, flag it.
4. **Saturation test:** Ramp `workspace-edit.js` to 100 VUs editing independent files. Monitor backend CPU and DB connection pool. The test should reveal the breaking point where `update-file` latency spikes.
5. **Media upload stress test:** Run `media-upload.js` with 50 VUs uploading 1 MB files. Verify storage throughput and no `413` errors.
6. **Font upload stress test:** Run `font-upload.js` with 10 VUs. Verify FontForge CPU usage and no timeouts.
### Manual validation checklist
- [x] `POST /api/main/methods/login-with-password` with JSON body returns a session cookie. (Validated via k6 lifecycle)
- [x] `GET /api/main/methods/get-profile` with `Accept: application/json` returns JSON. (Validated via k6 lifecycle)
- [ ] `curl -H "Accept: application/json" http://localhost:6060/metrics` returns Prometheus text.
- [ ] Backend fixtures create at least 100 test users and 100 test files.
- [x] A `update-file` request with a minimal `changes` payload succeeds and returns `{"revn": N}`. (Validated — needs full shape with selrect, points, transform, frame-id)
- [x] A `upload-file-media-object` multipart request succeeds and returns a media object ID. (Validated via k6 lifecycle)
- [ ] A chunked upload (`create-upload-session``upload-chunk``assemble-file-media-object`) succeeds.
- [ ] A `create-font-variant` request with chunked uploads succeeds.
---
## Immediate Next Steps (if approved)
1. ~~Create `performance/` directory and `README.md`.~~ ✅ Done
2. ~~Write `penpot-client.js` (k6 shared module) with `login()`, `rpc()`, `uploadMultipart()`, and `uploadChunked()` helpers.~~ ✅ Done (388 lines, JSON transport, cookie auth, session headers, tagged metrics)
3. ~~Write a manual `curl` validation script~~ — Skipped; JSON compatibility confirmed via k6 smoke test.
4. Write a data seeding script (`performance/scripts/seed-data.js`) that creates 100+ users, teams, projects, and files of varying sizes.
5. ~~Write the first k6 script: `lifecycle.js`~~ ✅ Done (398 lines, 12-step CRUD flow, all checks passing)
6. ~~Run a 1-VU smoke test against local devenv and commit the baseline results.~~ ✅ Done (10/10 checks, 0% failure, ~10s iteration)
7. Write `workspace-open.js` and `workspace-edit.js`.
8. Write `media-upload.js` and `font-upload.js`.
9. Define the `1000-VU` scenario mix in `options.js` (shared scenario config).
10. Run the first 100-VU ramp test and capture Prometheus metrics.
---
**Plan Author:** Senior Software Architect
**Status:** Phase 1 complete, Phase 2 partially complete (lifecycle.js done). See "Current Progress" section above.