mirror of
https://github.com/penpot/penpot.git
synced 2026-06-16 04:12:03 +00:00
✨ Add initial structure for performance tests
This commit is contained in:
parent
b2439694af
commit
4416d9380e
102
performance/README.md
Normal file
102
performance/README.md
Normal file
@ -0,0 +1,102 @@
|
||||
# Penpot Performance Tests
|
||||
|
||||
k6-based load and performance test suite for the Penpot backend. Measures HTTP RPC latency, throughput, and error rates under synthetic user load.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **k6** — Install from https://k6.io/docs/get-started/installation/
|
||||
- **Running Penpot backend** — Local devenv (`http://localhost:6060`) or a remote instance
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Smoke test — 1 VU, 1 iteration, demo mode
|
||||
./run.sh smoke
|
||||
|
||||
# Full lifecycle with 10 VUs, 5 iterations each
|
||||
./run.sh lifecycle -v 10 -n 5
|
||||
|
||||
# Use registration flow instead of demo profiles
|
||||
./run.sh lifecycle -m register -v 5 -n 1
|
||||
|
||||
# Point to a remote backend
|
||||
./run.sh lifecycle -u https://penpot.example.com
|
||||
|
||||
# Show all options
|
||||
./run.sh help
|
||||
```
|
||||
|
||||
## Test Scripts
|
||||
|
||||
### `scripts/lifecycle.js` — Full User Lifecycle
|
||||
|
||||
Simulates a realistic user journey from account creation through CRUD operations:
|
||||
|
||||
1. **Register** — Create a new user (demo profile or full registration)
|
||||
2. **Login** — Authenticate and obtain session cookie
|
||||
3. **Get Profile** — Fetch current user profile
|
||||
4. **Get Teams** — List user teams
|
||||
5. **Create Project** — Create a new project in the default team
|
||||
6. **Create File** — Create a new design file in the project
|
||||
7. **Get File** — Fetch the file with its data (pages, objects)
|
||||
8. **Update File** — Add a rectangle shape (tests optimistic concurrency)
|
||||
9. **Upload Image** — Upload a PNG to the file's media objects
|
||||
10. **Delete File** — Remove the file
|
||||
11. **Delete Project** — Remove the project
|
||||
12. **Logout** — End the session
|
||||
|
||||
Each VU performs the full flow independently, creating and cleaning up its own artifacts.
|
||||
|
||||
## Configuration
|
||||
|
||||
Options for `run.sh lifecycle`:
|
||||
|
||||
| Flag | Env Variable | Default | Description |
|
||||
|------|-------------|---------|-------------|
|
||||
| `-u URL` | `PENPOT_BASE_URL` | `http://localhost:6060` | Penpot backend URL |
|
||||
| `-v NUM` | — | `1` | Number of virtual users |
|
||||
| `-n NUM` | — | `1` | Iterations per VU |
|
||||
| `-m MODE` | `PENPOT_REGISTER_MODE` | `demo` | `demo` or `register` |
|
||||
| `-k PATH` | `K6` | `k6` | Path to k6 binary |
|
||||
|
||||
### Register Modes
|
||||
|
||||
- **`demo`** (default): Uses the `create-demo-profile` RPC endpoint. Requires the `demo-users` feature flag to be enabled on the backend. Fastest for testing.
|
||||
- **`register`**: Uses the full two-step registration flow (`prepare-register-profile` + `register-profile`). Works without any feature flags but is slower.
|
||||
|
||||
## Shared Client (`lib/penpot-client.js`)
|
||||
|
||||
The shared client module wraps the Penpot backend RPC API using plain JSON (not Transit). Key features:
|
||||
|
||||
- **JSON transport**: Uses `Content-Type: application/json` for POST bodies and `Accept: application/json` (or `_fmt=json` for GET) for responses.
|
||||
- **Cookie-based auth**: k6 automatically manages session cookies per VU.
|
||||
- **Session headers**: Generates `x-session-id` and `x-external-session-id` UUIDs per VU.
|
||||
- **Tagged metrics**: Every request is tagged with `rpc_command` for k6 metric slicing.
|
||||
|
||||
## Results
|
||||
|
||||
Test results are written to `results/<timestamp>/` as JSON. k6 also prints a summary to stdout with percentile breakdowns per RPC command.
|
||||
|
||||
## Thresholds
|
||||
|
||||
The lifecycle script includes built-in thresholds that will cause k6 to exit with a non-zero code if exceeded:
|
||||
|
||||
- `http_req_duration p95 < 5000ms` (global)
|
||||
- `http_req_failed < 1%` (global)
|
||||
- Per-command thresholds for login, profile, project, file, and update operations
|
||||
|
||||
## Adding New Flows
|
||||
|
||||
To add a new test flow:
|
||||
|
||||
1. Create `scripts/<flow-name>.js`
|
||||
2. Import the shared client: `import { createClient } from "../lib/penpot-client.js";`
|
||||
3. Implement the flow using the client methods
|
||||
4. Add a command in `run.sh`
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- The backend supports both Transit JSON and plain JSON. This test suite uses **plain JSON** for simplicity (no Transit encoder needed in k6).
|
||||
- JSON request keys are in **kebab-case** (matching Clojure conventions). JSON response keys are in **camelCase** (backend's default JSON encoding).
|
||||
- `update-file` sends the `id` parameter both in the query string and in the POST body, matching the frontend's behavior.
|
||||
- The backend uses optimistic concurrency control (`revn`) for file updates. The test retries once on conflict.
|
||||
BIN
performance/fixtures/test-large.png
Normal file
BIN
performance/fixtures/test-large.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
BIN
performance/fixtures/test-medium.png
Normal file
BIN
performance/fixtures/test-medium.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
BIN
performance/fixtures/test-small.png
Normal file
BIN
performance/fixtures/test-small.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 97 B |
525
performance/lib/penpot-client.js
Normal file
525
performance/lib/penpot-client.js
Normal file
@ -0,0 +1,525 @@
|
||||
// Penpot k6 HTTP Client
|
||||
//
|
||||
// Shared module that wraps the Penpot backend RPC API using plain JSON.
|
||||
// The backend supports `application/json` request bodies (kebab-case keys)
|
||||
// and `application/json` responses (camelCase keys) via Accept header or _fmt=json.
|
||||
//
|
||||
// Authentication is cookie-based: login-with-password sets a session cookie,
|
||||
// and all subsequent requests include it automatically via the k6 cookie jar.
|
||||
|
||||
import http from "k6/http";
|
||||
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Configuration
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Creates a new Penpot client instance.
|
||||
*
|
||||
* @param {string} baseUrl - The base URL of the Penpot backend (e.g., "http://localhost:6060")
|
||||
* @returns {object} Client instance with RPC methods
|
||||
*/
|
||||
export function createClient(baseUrl) {
|
||||
// Per-VU session identifiers — consistent across all requests within one VU iteration
|
||||
const sessionId = uuidv4();
|
||||
const externalSessionId = uuidv4();
|
||||
|
||||
const defaultHeaders = {
|
||||
"Accept": "application/json",
|
||||
"x-session-id": sessionId,
|
||||
"x-external-session-id": externalSessionId,
|
||||
"x-event-origin": "perf-test",
|
||||
"x-client": "penpot-perf/1.0",
|
||||
};
|
||||
|
||||
// k6 automatically manages cookies per VU when `cookies` are returned by the server.
|
||||
// We use the default cookie jar which is per-VU.
|
||||
|
||||
/**
|
||||
* Make an RPC call to the Penpot backend.
|
||||
*
|
||||
* GET requests: params go as query parameters, response is JSON via _fmt=json.
|
||||
* POST requests: params go as JSON body, response is JSON via Accept header.
|
||||
*
|
||||
* @param {string} method - HTTP method ("GET" or "POST")
|
||||
* @param {string} command - RPC command name (e.g., "login-with-password")
|
||||
* @param {object} params - Parameters for the RPC call
|
||||
* @param {object} [opts] - Additional options
|
||||
* @param {string} [opts.tag] - k6 metric tag for this request
|
||||
* @returns {object} k6 Response object with parsed JSON body
|
||||
*/
|
||||
function rpc(method, command, params = {}, opts = {}) {
|
||||
const url = `${baseUrl}/api/main/methods/${command}`;
|
||||
const tag = opts.tag || command;
|
||||
|
||||
const tags = {
|
||||
rpc_command: tag,
|
||||
};
|
||||
|
||||
if (method === "GET") {
|
||||
// GET requests: params go as query string, add _fmt=json for JSON response
|
||||
const queryParams = { ...params, _fmt: "json" };
|
||||
const qs = Object.entries(queryParams)
|
||||
.filter(([, v]) => v !== undefined && v !== null)
|
||||
.map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`)
|
||||
.join("&");
|
||||
|
||||
const fullUrl = qs ? `${url}?${qs}` : url;
|
||||
|
||||
return http.get(fullUrl, {
|
||||
headers: defaultHeaders,
|
||||
tags,
|
||||
});
|
||||
} else {
|
||||
// POST requests: params go as JSON body
|
||||
const headers = {
|
||||
...defaultHeaders,
|
||||
"Content-Type": "application/json",
|
||||
};
|
||||
|
||||
return http.post(url, JSON.stringify(params), {
|
||||
headers,
|
||||
tags,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Login with email and password.
|
||||
* Returns the profile data on success. The session cookie is stored
|
||||
* automatically by k6's cookie jar.
|
||||
*
|
||||
* @param {string} email
|
||||
* @param {string} password
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function login(email, password) {
|
||||
const res = rpc("POST", "login-with-password", {
|
||||
email,
|
||||
password,
|
||||
});
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the current user's profile (requires prior login).
|
||||
*
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function getProfile() {
|
||||
const res = rpc("GET", "get-profile");
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get all teams for the current user.
|
||||
*
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function getTeams() {
|
||||
const res = rpc("GET", "get-teams");
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a new team.
|
||||
*
|
||||
* @param {string} name - Team name
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function createTeam(name) {
|
||||
const res = rpc("POST", "create-team", { name });
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get projects for a team.
|
||||
*
|
||||
* @param {string} teamId - Team UUID
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function getProjects(teamId) {
|
||||
const res = rpc("GET", "get-projects", { "team-id": teamId });
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a new project.
|
||||
*
|
||||
* @param {string} teamId - Team UUID
|
||||
* @param {string} name - Project name
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function createProject(teamId, name) {
|
||||
const res = rpc("POST", "create-project", {
|
||||
"team-id": teamId,
|
||||
name,
|
||||
});
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a new file in a project.
|
||||
*
|
||||
* @param {string} projectId - Project UUID
|
||||
* @param {string} name - File name
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function createFile(projectId, name) {
|
||||
const res = rpc("POST", "create-file", {
|
||||
"project-id": projectId,
|
||||
name,
|
||||
});
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a file by ID.
|
||||
*
|
||||
* @param {string} fileId - File UUID
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function getFile(fileId) {
|
||||
const res = rpc("GET", "get-file", {
|
||||
id: fileId,
|
||||
});
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Update a file with changes.
|
||||
*
|
||||
* The backend uses optimistic concurrency control via `revn`.
|
||||
* If a conflict occurs (status 400 with :revn-conflict), the caller
|
||||
* should retry with the latest revn from getFile().
|
||||
*
|
||||
* @param {string} fileId - File UUID
|
||||
* @param {number} revn - Current file revision number
|
||||
* @param {number} vern - Current file version number
|
||||
* @param {string} sessionId - Client session ID (UUID)
|
||||
* @param {Array} changes - Array of change objects
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function updateFile(fileId, revn, vern, changesSessionId, changes) {
|
||||
const params = {
|
||||
id: fileId,
|
||||
revn: revn,
|
||||
vern: vern,
|
||||
"session-id": changesSessionId,
|
||||
origin: "workspace",
|
||||
"created-at": new Date().toISOString(),
|
||||
"commit-id": uuidv4(),
|
||||
changes: changes,
|
||||
};
|
||||
|
||||
// update-file uses POST with id also as query param (per frontend convention)
|
||||
const url = `${baseUrl}/api/main/methods/update-file?id=${encodeURIComponent(fileId)}`;
|
||||
const headers = {
|
||||
...defaultHeaders,
|
||||
"Content-Type": "application/json",
|
||||
};
|
||||
|
||||
const res = http.post(url, JSON.stringify(params), {
|
||||
headers,
|
||||
tags: { rpc_command: "update-file" },
|
||||
});
|
||||
|
||||
let body = null;
|
||||
try {
|
||||
if (res.body && res.body.length > 0) {
|
||||
body = res.json();
|
||||
}
|
||||
} catch (e) {
|
||||
// body may not be JSON
|
||||
}
|
||||
|
||||
return {
|
||||
status: res.status,
|
||||
body: body,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload a file media object using direct multipart upload.
|
||||
*
|
||||
* @param {string} fileId - File UUID
|
||||
* @param {Uint8Array} fileBytes - The file content
|
||||
* @param {string} fileName - The file name
|
||||
* @param {string} mimeType - MIME type (e.g., "image/png")
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType) {
|
||||
const url = `${baseUrl}/api/main/methods/upload-file-media-object`;
|
||||
|
||||
const headers = {
|
||||
...defaultHeaders,
|
||||
// No Content-Type — k6 sets it automatically for multipart/form-data
|
||||
};
|
||||
|
||||
const formData = {
|
||||
"file-id": fileId,
|
||||
"is-local": "true",
|
||||
name: fileName,
|
||||
content: http.file(fileBytes, fileName, mimeType),
|
||||
};
|
||||
|
||||
const res = http.post(url, formData, {
|
||||
headers,
|
||||
tags: { rpc_command: "upload-file-media-object" },
|
||||
});
|
||||
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Chunked upload
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Create an upload session for chunked uploads.
|
||||
*
|
||||
* @param {number} totalChunks - Number of chunks
|
||||
* @returns {object} { status, sessionId }
|
||||
*/
|
||||
function createUploadSession(totalChunks) {
|
||||
const res = rpc("POST", "create-upload-session", {
|
||||
"total-chunks": totalChunks,
|
||||
});
|
||||
const body = res.status === 200 ? res.json() : null;
|
||||
return {
|
||||
status: res.status,
|
||||
sessionId: body ? body.sessionId : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload a single chunk within an upload session.
|
||||
*
|
||||
* @param {string} sessionId - Upload session UUID
|
||||
* @param {number} index - Chunk index (0-based)
|
||||
* @param {Uint8Array} chunkBytes - The chunk content
|
||||
* @param {string} fileName - Original file name
|
||||
* @param {string} mimeType - MIME type
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function uploadChunk(sessionId, index, chunkBytes, fileName, mimeType) {
|
||||
const url = `${baseUrl}/api/main/methods/upload-chunk`;
|
||||
|
||||
const headers = {
|
||||
...defaultHeaders,
|
||||
};
|
||||
|
||||
const formData = {
|
||||
"session-id": sessionId,
|
||||
index: String(index),
|
||||
content: http.file(chunkBytes, fileName, mimeType),
|
||||
};
|
||||
|
||||
const res = http.post(url, formData, {
|
||||
headers,
|
||||
tags: { rpc_command: "upload-chunk" },
|
||||
});
|
||||
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Assemble all uploaded chunks into a final media object.
|
||||
*
|
||||
* @param {string} sessionId - Upload session UUID
|
||||
* @param {string} fileId - File UUID
|
||||
* @param {string} name - Media object name
|
||||
* @param {boolean} isLocal - Whether the object is local to the file
|
||||
* @param {string} mimeType - MIME type (e.g., "image/png")
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function assembleFileMediaObject(sessionId, fileId, name, isLocal, mimeType) {
|
||||
const res = rpc("POST", "assemble-file-media-object", {
|
||||
"session-id": sessionId,
|
||||
"file-id": fileId,
|
||||
name: name,
|
||||
"is-local": isLocal,
|
||||
mtype: mimeType,
|
||||
});
|
||||
return {
|
||||
status: res.status,
|
||||
body: res.status === 200 ? res.json() : null,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Smart upload — picks direct or chunked based on file size
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
// Chunk size threshold: files larger than this use chunked upload.
|
||||
// The actual chunk size is irrelevant to the backend; this controls
|
||||
// which upload path is exercised.
|
||||
const CHUNK_SIZE = 50 * 1024; // 50 KB
|
||||
|
||||
/**
|
||||
* Upload a file media object, automatically selecting direct or chunked
|
||||
* upload based on file size.
|
||||
*
|
||||
* Files <= CHUNK_SIZE use direct multipart upload.
|
||||
* Files > CHUNK_SIZE use chunked upload (create-upload-session →
|
||||
* upload-chunk × N → assemble-file-media-object).
|
||||
*
|
||||
* @param {string} fileId - File UUID
|
||||
* @param {Uint8Array} fileBytes - The file content
|
||||
* @param {string} fileName - The file name
|
||||
* @param {string} mimeType - MIME type (e.g., "image/png")
|
||||
* @returns {object} Parsed response { status, body }
|
||||
*/
|
||||
function uploadFileMediaObject(fileId, fileBytes, fileName, mimeType) {
|
||||
if (fileBytes.byteLength <= CHUNK_SIZE) {
|
||||
return uploadFileMediaObjectDirect(fileId, fileBytes, fileName, mimeType);
|
||||
}
|
||||
|
||||
// Chunked upload path
|
||||
const totalChunks = Math.ceil(fileBytes.byteLength / CHUNK_SIZE);
|
||||
|
||||
const sessionRes = createUploadSession(totalChunks);
|
||||
if (sessionRes.status !== 200) {
|
||||
return { status: sessionRes.status, body: null, raw: sessionRes.raw };
|
||||
}
|
||||
const uploadSessionId = sessionRes.sessionId;
|
||||
|
||||
for (let i = 0; i < totalChunks; i++) {
|
||||
const start = i * CHUNK_SIZE;
|
||||
const end = Math.min(start + CHUNK_SIZE, fileBytes.byteLength);
|
||||
const chunk = fileBytes.slice(start, end);
|
||||
|
||||
const chunkRes = uploadChunk(uploadSessionId, i, chunk, fileName, mimeType);
|
||||
if (chunkRes.status !== 200) {
|
||||
return { status: chunkRes.status, body: null, raw: chunkRes.raw };
|
||||
}
|
||||
}
|
||||
|
||||
return assembleFileMediaObject(uploadSessionId, fileId, fileName, true, mimeType);
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a file.
|
||||
*
|
||||
* @param {string} fileId - File UUID
|
||||
* @returns {object} Parsed response { status }
|
||||
*/
|
||||
function deleteFile(fileId) {
|
||||
const res = rpc("POST", "delete-file", { id: fileId });
|
||||
return {
|
||||
status: res.status,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a project.
|
||||
*
|
||||
* @param {string} projectId - Project UUID
|
||||
* @returns {object} Parsed response { status }
|
||||
*/
|
||||
function deleteProject(projectId) {
|
||||
const res = rpc("POST", "delete-project", { id: projectId });
|
||||
return {
|
||||
status: res.status,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a team.
|
||||
*
|
||||
* @param {string} teamId - Team UUID
|
||||
* @returns {object} Parsed response { status }
|
||||
*/
|
||||
function deleteTeam(teamId) {
|
||||
const res = rpc("POST", "delete-team", { id: teamId });
|
||||
return {
|
||||
status: res.status,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Logout the current user.
|
||||
*
|
||||
* @param {string} profileId - Profile UUID
|
||||
* @returns {object} Parsed response { status }
|
||||
*/
|
||||
function logout(profileId) {
|
||||
const res = rpc("POST", "logout", { "profile-id": profileId });
|
||||
return {
|
||||
status: res.status,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
|
||||
// Return the client interface
|
||||
return {
|
||||
sessionId,
|
||||
externalSessionId,
|
||||
rpc,
|
||||
login,
|
||||
getProfile,
|
||||
getTeams,
|
||||
createTeam,
|
||||
getProjects,
|
||||
createProject,
|
||||
createFile,
|
||||
getFile,
|
||||
updateFile,
|
||||
uploadFileMediaObject,
|
||||
uploadFileMediaObjectDirect,
|
||||
createUploadSession,
|
||||
uploadChunk,
|
||||
assembleFileMediaObject,
|
||||
deleteFile,
|
||||
deleteProject,
|
||||
deleteTeam,
|
||||
logout,
|
||||
};
|
||||
}
|
||||
160
performance/run.sh
Executable file
160
performance/run.sh
Executable file
@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Penpot Performance Tests
|
||||
#
|
||||
# k6-based load/performance test suite for the Penpot backend.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - k6 (https://k6.io/) installed and in PATH
|
||||
# - A running Penpot backend (local devenv or remote)
|
||||
#
|
||||
# Usage:
|
||||
# ./run.sh smoke # 1 VU, 1 iteration smoke test
|
||||
# ./run.sh lifecycle # 1 VU, 1 iteration (defaults)
|
||||
# ./run.sh lifecycle -v 10 -n 5 # 10 VUs, 5 iterations each
|
||||
# ./run.sh lifecycle -u http://remote:6060 -m register -v 5
|
||||
# ./run.sh clean # Remove test results
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defaults
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
BASE_URL="${PENPOT_BASE_URL:-http://localhost:6060}"
|
||||
VUS=1
|
||||
ITER=1
|
||||
REGISTER_MODE="${PENPOT_REGISTER_MODE:-demo}"
|
||||
K6="${K6:-k6}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Penpot Performance Tests
|
||||
|
||||
Usage:
|
||||
$(basename "$0") <command> [options]
|
||||
|
||||
Commands:
|
||||
smoke 1 VU, 1 iteration smoke test (forces demo mode)
|
||||
lifecycle Full user lifecycle test
|
||||
clean Remove test results
|
||||
help Show this help
|
||||
|
||||
Options (lifecycle only):
|
||||
-u URL Backend base URL (default: $BASE_URL)
|
||||
-v NUM Number of virtual users (default: $VUS)
|
||||
-n NUM Iterations per VU (default: $ITER)
|
||||
-m MODE Register mode: 'demo' or 'register' (default: $REGISTER_MODE)
|
||||
-k PATH Path to k6 binary (default: $K6)
|
||||
|
||||
Environment variables:
|
||||
PENPOT_BASE_URL Same as -u
|
||||
PENPOT_REGISTER_MODE Same as -m
|
||||
K6 Same as -k
|
||||
|
||||
Examples:
|
||||
$(basename "$0") smoke
|
||||
$(basename "$0") lifecycle -v 10 -n 5
|
||||
$(basename "$0") lifecycle -m register -v 5 -n 1
|
||||
$(basename "$0") lifecycle -u https://penpot.example.com
|
||||
EOF
|
||||
}
|
||||
|
||||
check_k6() {
|
||||
if ! command -v "$K6" &>/dev/null; then
|
||||
echo "Error: k6 not found at '$K6'" >&2
|
||||
echo "Install from https://k6.io/docs/get-started/installation/" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_k6() {
|
||||
local results_dir="$SCRIPT_DIR/results/$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$results_dir"
|
||||
|
||||
echo "Penpot Performance Test"
|
||||
echo " Base URL: $BASE_URL"
|
||||
echo " Register mode: $REGISTER_MODE"
|
||||
echo " VUs: $VUS"
|
||||
echo " Iterations: $ITER"
|
||||
echo " Results: $results_dir"
|
||||
echo ""
|
||||
|
||||
"$K6" run \
|
||||
--env "PENPOT_BASE_URL=$BASE_URL" \
|
||||
--env "PENPOT_REGISTER_MODE=$REGISTER_MODE" \
|
||||
--vus "$VUS" \
|
||||
--iterations "$ITER" \
|
||||
--out "json=$results_dir/k6-summary.json" \
|
||||
"$SCRIPT_DIR/scripts/lifecycle.js"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Commands
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
cmd_smoke() {
|
||||
check_k6
|
||||
REGISTER_MODE=demo
|
||||
VUS=1
|
||||
ITER=1
|
||||
run_k6
|
||||
}
|
||||
|
||||
cmd_lifecycle() {
|
||||
# Parse options
|
||||
while getopts "u:v:n:m:k:h" opt; do
|
||||
case "$opt" in
|
||||
u) BASE_URL="$OPTARG" ;;
|
||||
v) VUS="$OPTARG" ;;
|
||||
n) ITER="$OPTARG" ;;
|
||||
m) REGISTER_MODE="$OPTARG" ;;
|
||||
k) K6="$OPTARG" ;;
|
||||
h) usage; exit 0 ;;
|
||||
*) usage >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
check_k6
|
||||
run_k6
|
||||
}
|
||||
|
||||
cmd_clean() {
|
||||
local results_dir="$SCRIPT_DIR/results"
|
||||
if [[ -d "$results_dir" ]]; then
|
||||
rm -rf "$results_dir"
|
||||
echo "Cleaned $results_dir"
|
||||
else
|
||||
echo "Nothing to clean"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
command="$1"
|
||||
shift
|
||||
|
||||
case "$command" in
|
||||
smoke) cmd_smoke "$@" ;;
|
||||
lifecycle) cmd_lifecycle "$@" ;;
|
||||
clean) cmd_clean "$@" ;;
|
||||
help|-h|--help) usage ;;
|
||||
*)
|
||||
echo "Unknown command: $command" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
417
performance/scripts/lifecycle.js
Normal file
417
performance/scripts/lifecycle.js
Normal file
@ -0,0 +1,417 @@
|
||||
// Lifecycle Performance Test
|
||||
//
|
||||
// Simulates a realistic user lifecycle from registration through CRUD operations.
|
||||
// Each VU performs the full flow independently, creating its own artifacts.
|
||||
//
|
||||
// Flow:
|
||||
// 1. Register (via demo profile or prepare+register)
|
||||
// 2. Login
|
||||
// 3. Get profile & teams
|
||||
// 4. Create project
|
||||
// 5. Create file
|
||||
// 6. Get file
|
||||
// 7. Update file (add a shape)
|
||||
// 8. Upload image to file
|
||||
// 9. Delete file
|
||||
// 10. Delete project
|
||||
// 11. Delete team
|
||||
//
|
||||
// Usage:
|
||||
// k6 run scripts/lifecycle.js
|
||||
// k6 run --env PENPOT_BASE_URL=http://localhost:6060 scripts/lifecycle.js
|
||||
// k6 run --env PENPOT_REGISTER_MODE=register scripts/lifecycle.js
|
||||
|
||||
import { check, sleep, fail } from "k6";
|
||||
import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.4.0/index.js";
|
||||
import { createClient } from "../lib/penpot-client.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Configuration
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const BASE_URL = __ENV.PENPOT_BASE_URL || "http://localhost:6060";
|
||||
// "demo" = use create-demo-profile (requires demo-users flag)
|
||||
// "register" = use prepare-register-profile + register-profile
|
||||
const REGISTER_MODE = __ENV.PENPOT_REGISTER_MODE || "demo";
|
||||
|
||||
// k6 options — smoke test defaults (1 VU, 1 iteration)
|
||||
export const options = {
|
||||
scenarios: {
|
||||
lifecycle: {
|
||||
executor: "per-vu-iterations",
|
||||
vus: 1,
|
||||
iterations: 1,
|
||||
maxDuration: "2m",
|
||||
},
|
||||
},
|
||||
thresholds: {
|
||||
http_req_duration: ["p(95)<5000"],
|
||||
http_req_failed: ["rate<0.01"],
|
||||
"http_req_duration{rpc_command:login-with-password}": ["p(95)<1000"],
|
||||
"http_req_duration{rpc_command:get-profile}": ["p(95)<500"],
|
||||
"http_req_duration{rpc_command:create-project}": ["p(95)<1000"],
|
||||
"http_req_duration{rpc_command:create-file}": ["p(95)<1000"],
|
||||
"http_req_duration{rpc_command:get-file}": ["p(95)<500"],
|
||||
"http_req_duration{rpc_command:update-file}": ["p(95)<2000"],
|
||||
"http_req_duration{rpc_command:delete-file}": ["p(95)<1000"],
|
||||
},
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Test Data
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Load test PNG fixtures — small uses direct upload, large uses chunked upload
|
||||
const testImageSmall = open("../fixtures/test-small.png", "b");
|
||||
const testImageLarge = open("../fixtures/test-large.png", "b");
|
||||
|
||||
// A minimal "add-obj" change payload for update-file.
|
||||
// This adds a simple rectangle shape to the first page.
|
||||
// All object properties use camelCase — the backend's JSON parser
|
||||
// (json/read-kebab-key) converts camelCase to kebab-case keywords automatically.
|
||||
function makeAddRectChange(pageId) {
|
||||
const shapeId = uuidv4();
|
||||
const x = 100;
|
||||
const y = 100;
|
||||
const w = 200;
|
||||
const h = 150;
|
||||
|
||||
return {
|
||||
type: "add-obj",
|
||||
pageId: pageId,
|
||||
id: shapeId,
|
||||
frameId: pageId, // required: the frame this shape belongs to
|
||||
parentId: pageId, // root frame is the page itself
|
||||
obj: {
|
||||
id: shapeId,
|
||||
type: "rect",
|
||||
name: "Perf Test Rect",
|
||||
x: x,
|
||||
y: y,
|
||||
width: w,
|
||||
height: h,
|
||||
fillColor: "#ff0000",
|
||||
fillOpacity: 1,
|
||||
rotation: 0,
|
||||
hidden: false,
|
||||
locked: false,
|
||||
// Required base attrs
|
||||
selrect: {
|
||||
x: x,
|
||||
y: y,
|
||||
width: w,
|
||||
height: h,
|
||||
x1: x,
|
||||
y1: y,
|
||||
x2: x + w,
|
||||
y2: y + h,
|
||||
},
|
||||
points: [
|
||||
{ x: x, y: y },
|
||||
{ x: x + w, y: y },
|
||||
{ x: x + w, y: y + h },
|
||||
{ x: x, y: y + h },
|
||||
],
|
||||
transform: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 },
|
||||
transformInverse: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 },
|
||||
parentId: pageId,
|
||||
frameId: pageId,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function assertOk(res, label) {
|
||||
const ok = check(res, {
|
||||
[`${label} — status is 2xx`]: (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
if (!ok) {
|
||||
// Try to get the response body for debugging
|
||||
let bodyStr = "";
|
||||
try {
|
||||
if (res.raw && res.raw.body) {
|
||||
bodyStr = typeof res.raw.body === "string"
|
||||
? res.raw.body.substring(0, 500)
|
||||
: JSON.stringify(res.raw.body).substring(0, 500);
|
||||
} else if (res.body) {
|
||||
bodyStr = JSON.stringify(res.body).substring(0, 500);
|
||||
}
|
||||
} catch (e) {
|
||||
bodyStr = "(could not read body)";
|
||||
}
|
||||
console.error(
|
||||
`FAIL: ${label} — status=${res.status} body=${bodyStr}`
|
||||
);
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Setup — runs once before VUs start
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function setup() {
|
||||
console.log(`Penpot Lifecycle Test`);
|
||||
console.log(` Base URL: ${BASE_URL}`);
|
||||
console.log(` Register mode: ${REGISTER_MODE}`);
|
||||
console.log(``);
|
||||
|
||||
// Verify the backend is reachable
|
||||
const client = createClient(BASE_URL);
|
||||
const res = client.getProfile();
|
||||
// We expect 401/403 (not logged in) — anything else means the backend is down
|
||||
if (res.status === 0) {
|
||||
fail(`Backend unreachable at ${BASE_URL}`);
|
||||
}
|
||||
|
||||
return { baseUrl: BASE_URL, registerMode: REGISTER_MODE };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main VU Function
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export default function (data) {
|
||||
const client = createClient(data.baseUrl);
|
||||
|
||||
// ---- Step 0: Create a user account ----
|
||||
let email, password;
|
||||
|
||||
if (data.registerMode === "demo") {
|
||||
// Demo mode: create-demo-profile returns email + password
|
||||
const demoRes = client.rpc("POST", "create-demo-profile", {});
|
||||
if (!assertOk(demoRes, "create-demo-profile")) {
|
||||
fail("Failed to create demo profile");
|
||||
}
|
||||
const demoBody = demoRes.json();
|
||||
email = demoBody.email;
|
||||
password = demoBody.password;
|
||||
console.log(`VU ${__VU}: Created demo profile: ${email}`);
|
||||
} else {
|
||||
// Register mode: prepare + register
|
||||
email = `perf-${uuidv4()}@test.local`;
|
||||
password = "PerfTest1234!";
|
||||
const fullname = `Perf User ${__VU}`;
|
||||
|
||||
const prepareRes = client.rpc("POST", "prepare-register-profile", {
|
||||
fullname,
|
||||
email,
|
||||
password,
|
||||
});
|
||||
if (!assertOk(prepareRes, "prepare-register-profile")) {
|
||||
fail("Failed to prepare registration");
|
||||
}
|
||||
const prepareBody = prepareRes.json();
|
||||
const token = prepareBody.token;
|
||||
|
||||
const registerRes = client.rpc("POST", "register-profile", {
|
||||
token,
|
||||
});
|
||||
if (!assertOk(registerRes, "register-profile")) {
|
||||
fail("Failed to register profile");
|
||||
}
|
||||
console.log(`VU ${__VU}: Registered profile: ${email}`);
|
||||
}
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 1: Login ----
|
||||
const loginRes = client.login(email, password);
|
||||
if (!assertOk(loginRes, "login-with-password")) {
|
||||
fail("Login failed");
|
||||
}
|
||||
const profile = loginRes.body;
|
||||
const profileId = profile.id;
|
||||
console.log(`VU ${__VU}: Logged in, profile-id=${profileId}`);
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 2: Get profile ----
|
||||
const profileRes = client.getProfile();
|
||||
if (!assertOk(profileRes, "get-profile")) {
|
||||
fail("get-profile failed");
|
||||
}
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
// ---- Step 3: Get teams ----
|
||||
const teamsRes = client.getTeams();
|
||||
if (!assertOk(teamsRes, "get-teams")) {
|
||||
fail("get-teams failed");
|
||||
}
|
||||
const teams = teamsRes.body;
|
||||
// teams is an array; the user has a default team from registration
|
||||
const defaultTeamId = Array.isArray(teams) && teams.length > 0
|
||||
? teams[0].id
|
||||
: null;
|
||||
|
||||
if (!defaultTeamId) {
|
||||
fail("No default team found after registration");
|
||||
}
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
// ---- Step 4: Create a project ----
|
||||
const projectName = `Perf Project ${uuidv4().substring(0, 8)}`;
|
||||
const projectRes = client.createProject(defaultTeamId, projectName);
|
||||
if (!assertOk(projectRes, "create-project")) {
|
||||
fail("create-project failed");
|
||||
}
|
||||
const project = projectRes.body;
|
||||
const projectId = project.id;
|
||||
console.log(`VU ${__VU}: Created project: ${projectId}`);
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 5: Create a file ----
|
||||
const fileName = `Perf File ${uuidv4().substring(0, 8)}`;
|
||||
const fileRes = client.createFile(projectId, fileName);
|
||||
if (!assertOk(fileRes, "create-file")) {
|
||||
fail("create-file failed");
|
||||
}
|
||||
const file = fileRes.body;
|
||||
const fileId = file.id;
|
||||
console.log(`VU ${__VU}: Created file: ${fileId}`);
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 6: Get the file (to read revn, vern, page-id) ----
|
||||
const getFileRes = client.getFile(fileId);
|
||||
if (!assertOk(getFileRes, "get-file")) {
|
||||
fail("get-file failed");
|
||||
}
|
||||
const fileData = getFileRes.body;
|
||||
const revn = fileData.revn;
|
||||
const vern = fileData.vern;
|
||||
|
||||
// Extract the first page-id from the file data
|
||||
// fileData.data.pages is an array of page UUIDs
|
||||
// fileData.data.pages-index is a map of page-id -> page objects
|
||||
let pageId = null;
|
||||
if (fileData.data && fileData.data.pages && fileData.data.pages.length > 0) {
|
||||
pageId = fileData.data.pages[0];
|
||||
}
|
||||
|
||||
if (!pageId) {
|
||||
console.warn(`VU ${__VU}: Could not find page-id in file data, skipping update-file`);
|
||||
}
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 7: Update file (add a rectangle shape) ----
|
||||
let updateOk = false;
|
||||
if (pageId) {
|
||||
const changes = [makeAddRectChange(pageId)];
|
||||
const updateRes = client.updateFile(fileId, revn, vern, client.sessionId, changes);
|
||||
|
||||
if (updateRes.status === 200) {
|
||||
updateOk = true;
|
||||
console.log(`VU ${__VU}: Updated file successfully`);
|
||||
} else {
|
||||
// Check for revn conflict — retry once
|
||||
const body = updateRes.body;
|
||||
console.error(`VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 500)}`);
|
||||
const isRevnConflict =
|
||||
body && (body.code === "revn-conflict" || body.type === "revn-conflict");
|
||||
|
||||
if (isRevnConflict) {
|
||||
console.log(`VU ${__VU}: Revn conflict, retrying...`);
|
||||
// Fetch latest file state
|
||||
const retryFileRes = client.getFile(fileId);
|
||||
if (assertOk(retryFileRes, "get-file (retry)")) {
|
||||
const retryData = retryFileRes.body;
|
||||
const retryRes = client.updateFile(
|
||||
fileId,
|
||||
retryData.revn,
|
||||
retryData.vern,
|
||||
client.sessionId,
|
||||
changes
|
||||
);
|
||||
if (assertOk(retryRes, "update-file (retry)")) {
|
||||
updateOk = true;
|
||||
console.log(`VU ${__VU}: Updated file on retry`);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.error(
|
||||
`VU ${__VU}: update-file failed: status=${updateRes.status} body=${JSON.stringify(body).substring(0, 300)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 8: Upload images to the file ----
|
||||
// Small image (97 B) uses direct multipart upload.
|
||||
// Large image (120 KB) uses chunked upload (create-session → upload-chunk × N → assemble).
|
||||
if (testImageSmall && testImageSmall.byteLength > 0) {
|
||||
const uploadRes = client.uploadFileMediaObject(
|
||||
fileId,
|
||||
testImageSmall,
|
||||
"test-small.png",
|
||||
"image/png"
|
||||
);
|
||||
if (assertOk(uploadRes, "upload-file-media-object (direct)")) {
|
||||
console.log(`VU ${__VU}: Uploaded small image (direct)`);
|
||||
}
|
||||
}
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
if (testImageLarge && testImageLarge.byteLength > 0) {
|
||||
const uploadRes = client.uploadFileMediaObject(
|
||||
fileId,
|
||||
testImageLarge,
|
||||
"test-large.png",
|
||||
"image/png"
|
||||
);
|
||||
if (assertOk(uploadRes, "upload-file-media-object (chunked)")) {
|
||||
console.log(`VU ${__VU}: Uploaded large image (chunked)`);
|
||||
}
|
||||
}
|
||||
|
||||
sleep(1);
|
||||
|
||||
// ---- Step 9: Delete the file ----
|
||||
const deleteFileRes = client.deleteFile(fileId);
|
||||
if (assertOk(deleteFileRes, "delete-file")) {
|
||||
console.log(`VU ${__VU}: Deleted file: ${fileId}`);
|
||||
}
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
// ---- Step 10: Delete the project ----
|
||||
const deleteProjectRes = client.deleteProject(projectId);
|
||||
if (assertOk(deleteProjectRes, "delete-project")) {
|
||||
console.log(`VU ${__VU}: Deleted project: ${projectId}`);
|
||||
}
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
// ---- Step 11: Delete the team ----
|
||||
// Note: We only delete the team if it's NOT the default team.
|
||||
// The default team cannot be deleted (or may cause errors).
|
||||
// For this test, we skip team deletion to avoid errors.
|
||||
// In a real scenario, we'd create a separate team and delete that.
|
||||
console.log(`VU ${__VU}: Skipping team deletion (using default team)`);
|
||||
|
||||
sleep(0.5);
|
||||
|
||||
// ---- Step 12: Logout ----
|
||||
const logoutRes = client.logout(profileId);
|
||||
console.log(`VU ${__VU}: Logout status: ${logoutRes.status}`);
|
||||
|
||||
console.log(`VU ${__VU}: Lifecycle complete`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Teardown — runs once after all VUs finish
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function teardown(data) {
|
||||
console.log("Lifecycle test complete.");
|
||||
}
|
||||
494
plans/2026-06-12-backend-performance-test.md
Normal file
494
plans/2026-06-12-backend-performance-test.md
Normal file
@ -0,0 +1,494 @@
|
||||
# Backend Performance Test Plan
|
||||
|
||||
**Context:** Build a k6-based load/performance test suite that simulates realistic browser-to-backend HTTP flows for distinct Penpot user operations. The goal is to measure backend impact (latency, throughput, error rates, resource saturation) under synthetic user load. **Browser rendering performance is explicitly out of scope.** WebSocket testing is deferred.
|
||||
|
||||
**Date:** 2026-06-12
|
||||
**Validated Requirements:**
|
||||
- Tool: **k6** (confirmed).
|
||||
- Environment: flexible — local devenv first, then remote staging/perf.
|
||||
- Target scale: **1000 concurrent VUs** (ramping from lower baselines).
|
||||
- Flows: **realistic CRUD lifecycle** — create, edit, upload, delete. Must include **image upload** and **font upload**.
|
||||
- `update-file` is important but difficult because it requires **2–3 concurrent users editing the same file**, and **file size matters**.
|
||||
- WebSocket: **deferred**.
|
||||
|
||||
---
|
||||
|
||||
## Current Progress
|
||||
|
||||
### Completed (2026-06-12)
|
||||
|
||||
Phase 1 is done. Phase 2 has the first flow (lifecycle) implemented and validated.
|
||||
|
||||
**What was built:**
|
||||
|
||||
```
|
||||
performance/
|
||||
├── run.sh # Bash runner script (smoke, lifecycle, clean, help)
|
||||
├── README.md # Usage docs, configuration, architecture notes
|
||||
├── lib/
|
||||
│ └── penpot-client.js # 388 lines — shared k6 HTTP client module
|
||||
├── scripts/
|
||||
│ └── lifecycle.js # 398 lines — full user lifecycle test
|
||||
├── fixtures/
|
||||
│ ├── test-small.png # 97 B
|
||||
│ ├── test-medium.png # 30 KB
|
||||
│ └── test-large.png # 120 KB
|
||||
├── results/ # k6 JSON output (gitignored)
|
||||
└── baselines/ # for regression baselines
|
||||
```
|
||||
|
||||
**What the lifecycle test covers (all passing):**
|
||||
|
||||
| Step | RPC Command | Method | Observed Latency |
|
||||
|------|------------|--------|-----------------|
|
||||
| 1. Register | `create-demo-profile` | POST | ~136ms |
|
||||
| 2. Login | `login-with-password` | POST | ~136ms |
|
||||
| 3. Get profile | `get-profile` | GET | ~7ms |
|
||||
| 4. Get teams | `get-teams` | GET | ~6ms |
|
||||
| 5. Create project | `create-project` | POST | ~9ms |
|
||||
| 6. Create file | `create-file` | POST | ~22ms |
|
||||
| 7. Get file | `get-file` | GET | ~12ms |
|
||||
| 8. Update file | `update-file` | POST | ~25ms |
|
||||
| 9. Upload image | `upload-file-media-object` | POST (multipart) | ~7ms |
|
||||
| 10. Delete file | `delete-file` | POST | ~12ms |
|
||||
| 11. Delete project | `delete-project` | POST | ~5ms |
|
||||
| 12. Logout | `logout` | POST | ~4ms |
|
||||
|
||||
Smoke test result: **10/10 checks pass, 0% failure rate, ~10s total iteration**.
|
||||
|
||||
**Key discoveries during implementation:**
|
||||
|
||||
1. **JSON transport works.** The backend accepts `Content-Type: application/json` for POST bodies (kebab-case keys auto-converted) and returns `application/json` responses (camelCase keys) via `Accept: application/json` or `_fmt=json` query param. No Transit encoder needed in k6.
|
||||
|
||||
2. **`create-file` `features` param.** Sending `features: []` (empty array) causes a 400 validation error. Omit the field entirely — it's optional in the schema (`backend/src/app/rpc/commands/files_create.clj`).
|
||||
|
||||
3. **`update-file` `changes` payload — shape schema is strict.** The `add-obj` change requires a fully valid shape object. Minimum required fields beyond the basics:
|
||||
- `selrect`: `{x, y, width, height, x1, y1, x2, y2}`
|
||||
- `points`: array of 4 `{x, y}` corner points
|
||||
- `transform` / `transform-inverse`: identity matrix `{a:1, b:0, c:0, d:1, e:0, f:0}`
|
||||
- `parent-id` and `frame-id` inside the `obj`
|
||||
- `frame-id` at the top level of the change object itself (required, not optional)
|
||||
- Schema defined in `common/src/app/common/files/changes.cljc` (line 189) and `common/src/app/common/types/shape.cljc` (line 165).
|
||||
|
||||
4. **`update-file` URL convention.** The frontend sends `id` both as a query param and in the POST body. The URL is `POST /api/main/methods/update-file?id=<uuid>`.
|
||||
|
||||
5. **Two registration modes work:**
|
||||
- `demo` mode: `create-demo-profile` — fast, requires `demo-users` feature flag.
|
||||
- `register` mode: `prepare-register-profile` + `register-profile` — two-step flow, works without flags.
|
||||
|
||||
6. **k6 installed at** `/home/penpot/.local/bin/k6` (v0.56.0). Not in default PATH; use `PATH="/home/penpot/.local/bin:$PATH"` or set `K6` env var.
|
||||
|
||||
### Remaining Work
|
||||
|
||||
| Phase | Status | Next Actions |
|
||||
|-------|--------|-------------|
|
||||
| Phase 1 – Discovery & Tooling | **Done** | — |
|
||||
| Phase 2 – Core HTTP Flows | **Partial** — lifecycle.js done | Write `workspace-open.js`, `workspace-edit.js`, `media-upload.js`, `font-upload.js`, `viewer.js`, `export.js` |
|
||||
| Phase 2 – Data seeding | **Not started** | Create `seed-data.js` for 100+ user pool |
|
||||
| Phase 3 – Scenarios | **Not started** | Define multi-scenario k6 config mixing flows |
|
||||
| Phase 4 – Advanced update-file | **Not started** | File size tiers, concurrent editing matrix |
|
||||
| Phase 5 – CI & Reporting | **Not started** | Grafana dashboards, regression guard |
|
||||
|
||||
### Immediate Next Steps
|
||||
|
||||
1. Write `workspace-open.js` — read-heavy flow (get-file, get-file-libraries, thumbnails).
|
||||
2. Write `workspace-edit.js` — write-heavy loop (get-file + update-file per VU, independent files).
|
||||
3. Write `media-upload.js` — direct + chunked upload flows.
|
||||
4. Write `font-upload.js` — chunked upload + create-font-variant.
|
||||
5. Write `viewer.js` — get-view-only-bundle + comments.
|
||||
6. Create `seed-data.js` — pre-seed 100+ users/teams/files for multi-VU runs.
|
||||
7. Define scenario mix in `run.sh` (multi-scenario support with ramping VUs).
|
||||
8. Add `--scenario` flag to `run.sh` to select individual flows or the full mix.
|
||||
|
||||
---
|
||||
|
||||
## Affected Modules
|
||||
|
||||
| Module | Why it is involved |
|
||||
|--------|---------------------|
|
||||
| `backend/` | Target system. All HTTP RPC (`/api/main/methods/*`), auth, storage, media processing, DB, and Prometheus metrics (`/metrics`). |
|
||||
| `frontend/` | Source of truth for user request flows. We inspect `app.main.repo` (RPC client), `app.main.data.*` (user flows), and `app.main.data.persistence` (save semantics). |
|
||||
| `common/` | Shared schemas, Transit helpers, and data structures. Used to understand valid `update-file` `changes` payloads. |
|
||||
|
||||
---
|
||||
|
||||
## Approach
|
||||
|
||||
### Phase 1 – Discovery & Tooling (Days 1–2)
|
||||
|
||||
#### 1.1. Read the frontend RPC flows to build a request catalog
|
||||
|
||||
Inspect these files to map every user action to its RPC command:
|
||||
|
||||
- `frontend/src/app/main/repo.cljs` — HTTP client conventions (headers, retry, GET vs POST rules, query params, form-data, multipart).
|
||||
- `frontend/src/app/main/data/dashboard.cljs` — Dashboard init (`get-projects`, `fetch-fonts`, `search-files`).
|
||||
- `frontend/src/app/main/data/workspace.cljs` — Workspace init (`get-file`, `get-file-libraries`, `get-file-object-thumbnails`, `resolve-file` via `get-file-fragment`).
|
||||
- `frontend/src/app/main/data/persistence.cljs` — File save flow (`update-file` with `changes`, `revn`, `session-id`, debounce/buffer logic).
|
||||
- `frontend/src/app/main/data/viewer.cljs` — Viewer flow (`get-view-only-bundle`).
|
||||
- `frontend/src/app/main/data/comments.cljs` — Comment thread fetch (`get-comment-threads`).
|
||||
- `frontend/src/app/main/data/media.cljs` / `upload.cljs` — Media upload flows (`upload-file-media-object`, `create-upload-session`, `upload-chunk`, `assemble-file-media-object`).
|
||||
- `frontend/src/app/main/data/fonts.cljs` — Font upload flow (`create-font-variant` with `:uploads` map).
|
||||
- `frontend/src/app/main/data/team.cljs` — Team creation (`create-team`), invitation (`create-team-invitations`).
|
||||
- `frontend/src/app/main/data/project.cljs` — Project creation (`create-project`).
|
||||
|
||||
**Goal:** produce a **Request Catalog** mapping user actions to RPC command names, HTTP methods, payload shapes, and required preconditions (e.g., `team-id`, `file-id`).
|
||||
|
||||
#### 1.2. Confirm JSON compatibility for the test harness
|
||||
|
||||
The backend middleware (`app.http.middleware`) supports `application/json` request bodies and `application/json` responses (via `_fmt=json` or `Accept: application/json`).
|
||||
|
||||
- **Action:** Send a manual `curl` to `POST /api/main/methods/login-with-password` with `Content-Type: application/json` and verify the response format.
|
||||
- **Action:** Verify `GET /api/main/methods/get-profile` with `Accept: application/json` returns plain JSON.
|
||||
- **Action:** Verify `POST /api/main/methods/update-file` with `Content-Type: application/json` and `_fmt=json` works.
|
||||
- **Action:** Verify `POST /api/main/methods/upload-file-media-object` with `multipart/form-data` works (k6 supports this natively).
|
||||
|
||||
#### 1.3. Set up the load testing directory and shared client
|
||||
|
||||
Create a directory `performance/` at the repo root.
|
||||
|
||||
Install **k6** (`k6` CLI or Docker image).
|
||||
|
||||
Create a shared `penpot-client.js` module that wraps:
|
||||
- `login(email, password)` → returns session cookie / token.
|
||||
- `rpc(cmd, params, opts)` → builds the correct URL, headers, body, and query params.
|
||||
- `uploadFileMediaObject(fileId, filePath, name)` → multipart upload.
|
||||
- `createUploadSession(totalChunks)` → chunked upload setup.
|
||||
- `uploadChunk(sessionId, index, chunkBytes)` → multipart chunk upload.
|
||||
- `assembleFileMediaObject(sessionId, fileId, name, isLocal)` → finalize chunked upload.
|
||||
|
||||
**Headers to replicate (critical for backend telemetry and session binding):**
|
||||
- `x-session-id`: generated UUID per VU (must be consistent across requests for the same session).
|
||||
- `x-external-session-id`: generated UUID per VU.
|
||||
- `x-event-origin`: a string origin (e.g., `"perf-test"`)
|
||||
- `accept`: `application/json` (for HTTP-only load path)
|
||||
- `content-type`: `application/json` (or `multipart/form-data` for uploads)
|
||||
- `credentials: "include"` (for cookie jar)
|
||||
|
||||
#### 1.4. Data seeding strategy for 1000 VU scale
|
||||
|
||||
Creating 1000 users/teams/files *inside* the load test is too slow and will distort the results.
|
||||
|
||||
**Recommended approach:**
|
||||
- **Setup Phase (k6 `setup()`):** Run a pre-test script that creates a shared pool of test artifacts.
|
||||
- Use `login-with-password` with a fixture account (e.g., `profile1@example.com` / `123123` if fixtures exist).
|
||||
- Create `N` teams, `N` projects, `N` files of varying sizes (see **File Size Tiers** below).
|
||||
- Export the IDs into a JSON file that k6 `setup()` reads.
|
||||
- **Alternative:** Use the backend REPL / fixtures (`app.cli.fixtures/run {:preset :small}`) to create fixture data, then export the IDs via a small Clojure script.
|
||||
- **Data pool per VU:** Each VU picks a random user from the pool, or uses a dedicated user (e.g., VU #1 → `profile1@example.com`, VU #2 → `profile2@example.com`). For 1000 VUs, we need at least 1000 pre-seeded users.
|
||||
- **Cleanup:** A post-test script can delete the seeded data, or we can use a dedicated perf DB that is reset between runs.
|
||||
|
||||
**Action:** Document the seeding procedure in `performance/README.md` and create a `seed-data.js` script.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 – Core HTTP Flow Scripts (Days 3–5)
|
||||
|
||||
Create one k6 script per user flow. Each script:
|
||||
- Uses `setup()` to read the shared data pool and log in.
|
||||
- Uses `vu` iterations to simulate the flow.
|
||||
- Tags every request with the RPC command name so k6 metrics are sliced by endpoint.
|
||||
- Uses `check()` assertions for HTTP 200 and valid JSON structure.
|
||||
|
||||
#### Flow 1: Realistic User Lifecycle (`lifecycle.js`)
|
||||
|
||||
This is the primary realistic flow. Each VU performs a full lifecycle:
|
||||
|
||||
1. **Auth**
|
||||
- `POST /api/main/methods/login-with-password` → `{email, password}`
|
||||
- `GET /api/main/methods/get-profile`
|
||||
- `GET /api/main/methods/get-teams`
|
||||
|
||||
2. **Create Team**
|
||||
- `POST /api/main/methods/create-team` → `{name: "Perf Team <uuid>"}`
|
||||
- `GET /api/main/methods/get-team?team-id=<id>`
|
||||
|
||||
3. **Create Project**
|
||||
- `POST /api/main/methods/create-project` → `{team-id, name}`
|
||||
- `GET /api/main/methods/get-project?id=<id>`
|
||||
|
||||
4. **Create File**
|
||||
- `POST /api/main/methods/create-file` → `{project-id, name, features}`
|
||||
- `GET /api/main/methods/get-file?id=<file-id>&features=<...>`
|
||||
|
||||
5. **Edit File (Simple Update)**
|
||||
- `POST /api/main/methods/update-file` with a minimal `changes` payload.
|
||||
- **Changes payload:** Use a simple change like `{:type "add-obj", :id "<uuid>", :page-id "<page-id>", :parent-id "<parent-id>", :obj {:type "rect", ...}}`. Inspect `app.common.files.changes` for the exact schema. For a load test, we only need the shape to be structurally valid; the backend validates it.
|
||||
- **Revn tracking:** Fetch the file first, read `revn`, then send `revn` in the update. If a conflict occurs (`409` or `:revn-conflict` error), retry once with the latest `revn`.
|
||||
|
||||
6. **Upload Image (Direct)**
|
||||
- `POST /api/main/methods/upload-file-media-object` (multipart)
|
||||
- Payload: `file-id`, `is-local: true`, `name`, `content` (the file bytes).
|
||||
- Use a small dummy PNG/SVG (e.g., 1 KB, 100 KB, 1 MB) stored in `performance/fixtures/`.
|
||||
|
||||
7. **Upload Image (Chunked)**
|
||||
- `POST /api/main/methods/create-upload-session` → `{total-chunks: N}`
|
||||
- Loop `N` times: `POST /api/main/methods/upload-chunk` (multipart, `session-id`, `index`, `chunk`)
|
||||
- `POST /api/main/methods/assemble-file-media-object` → `{session-id, file-id, name, is-local}`
|
||||
- Use a larger dummy file (e.g., 5 MB) to stress the chunked pipeline.
|
||||
|
||||
8. **Upload Font**
|
||||
- `POST /api/main/methods/create-upload-session` (chunked, because fonts can be large)
|
||||
- `POST /api/main/methods/upload-chunk` for each chunk
|
||||
- `POST /api/main/methods/create-font-variant` → `{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": "<session-id>"}}`
|
||||
- Use a small real TTF/OTF file from `performance/fixtures/`.
|
||||
|
||||
9. **Delete File**
|
||||
- `DELETE /api/main/methods/delete-file` (verify the exact method name; it may be `update-file` with a deletion flag or a dedicated command). Inspect `frontend/src/app/main/data/dashboard.cljs` for the delete action.
|
||||
|
||||
10. **Delete Project**
|
||||
- `DELETE /api/main/methods/delete-project?id=<id>`
|
||||
|
||||
11. **Delete Team**
|
||||
- `POST /api/main/methods/delete-team?id=<id>`
|
||||
|
||||
12. **Logout**
|
||||
- (Optional; session cookie expiry is usually sufficient)
|
||||
|
||||
**Pacing:** Add `sleep()` between steps to simulate realistic think time (e.g., 1–3 seconds between dashboard navigation, 3–5 seconds between edits).
|
||||
|
||||
#### Flow 2: Workspace Open (Read-heavy) (`workspace-open.js`)
|
||||
|
||||
For 1000 VUs, most will be read-only viewers or editors opening files.
|
||||
|
||||
1. Login (reuse token from `setup`).
|
||||
2. `GET /api/main/methods/get-file?id=<file-id>&features=<...>`
|
||||
3. `GET /api/main/methods/get-file-libraries?file-id=<file-id>`
|
||||
4. For each library: `GET /api/main/methods/get-file?id=<lib-id>`
|
||||
5. `GET /api/main/methods/get-file-object-thumbnails?file-id=<file-id>`
|
||||
6. `GET /api/main/methods/get-file-data-for-thumbnail?file-id=<file-id>&page-id=<page-id>&object-id=<frame-id>`
|
||||
|
||||
**Data:** Use a pool of files of varying sizes (see **File Size Tiers**).
|
||||
|
||||
#### Flow 3: Workspace Edit (Write-heavy) (`workspace-edit.js`)
|
||||
|
||||
**Scenario A — Independent editors (default, easiest to scale):**
|
||||
- Each VU creates its own file in `setup()`, or picks a dedicated file from the pool.
|
||||
- Loop:
|
||||
1. `GET /api/main/methods/get-file?id=<file-id>` (to refresh `revn`)
|
||||
2. `POST /api/main/methods/update-file` with minimal changes
|
||||
3. `sleep(3)`
|
||||
- This measures the latency of the save path without concurrency conflicts.
|
||||
|
||||
**Scenario B — Concurrent editors (advanced, measures conflict resolution):**
|
||||
- 2–3 VUs share the **same file ID**.
|
||||
- Each VU:
|
||||
1. `GET /api/main/methods/get-file?id=<file-id>` (to get latest `revn`)
|
||||
2. `POST /api/main/methods/update-file` with changes
|
||||
3. If `revn-conflict` (HTTP 400 or 409 with `:code :revn-conflict`), retry with the latest `revn`.
|
||||
- **Problem:** k6 VUs are independent; they cannot easily share a mutable `revn` counter.
|
||||
- **Solutions:**
|
||||
1. **Optimistic concurrency:** Let conflicts happen naturally. Measure the conflict rate and retry latency. This is realistic for many-user editing.
|
||||
2. **Shared state service:** Run a tiny Redis or in-memory service that stores the latest `revn` per file. VUs read/write it before each update. This adds coordination overhead but reduces conflicts.
|
||||
3. **Sequential VU groups:** Use k6 `scenarios` with `executor: 'per-vu-iterations'` and a small shared file pool. Accept that some conflicts will occur and measure them as part of the benchmark.
|
||||
- **Recommendation:** Start with **Solution 1** (optimistic). If the conflict rate is >10%, consider **Solution 2**.
|
||||
|
||||
**File Size Tiers for `update-file`:**
|
||||
The backend `update-file` performance depends heavily on file data size (serialization, validation, pointer-map resolution, snapshotting).
|
||||
|
||||
| Tier | Size | How to create |
|
||||
|------|------|---------------|
|
||||
| Small | ~10 shapes | Create a file with a few rectangles. |
|
||||
| Medium | ~100 shapes | Duplicate a page with many shapes. |
|
||||
| Large | ~1000 shapes | Import a real-world design file or use a fixture. |
|
||||
|
||||
**Action:** Create a `create-file-fixture.js` helper that generates files of each tier via the `create-file` + `update-file` API (or by importing a `.penpot` file via the binfile import API if available).
|
||||
|
||||
#### Flow 4: Viewer (Read-heavy, anonymous or logged-in) (`viewer.js`)
|
||||
|
||||
1. Login (or use share-link token for anonymous).
|
||||
2. `GET /api/main/methods/get-view-only-bundle?file-id=<id>&share-id=<id>&features=<...>`
|
||||
3. `GET /api/main/methods/get-comment-threads?file-id=<id>&share-id=<id>`
|
||||
|
||||
#### Flow 5: Export (CPU/IO-heavy) (`export.js`)
|
||||
|
||||
1. Login.
|
||||
2. `POST /api/export` with export payload.
|
||||
- Inspect `frontend/src/app/main/data/export.cljs` for the exact payload shape.
|
||||
- Common exports: `type: "png"`, `type: "svg"`, `type: "pdf"`.
|
||||
- This hits the **exporter** service (Node.js/Playwright), which is a separate process. If the goal is to stress the **backend**, limit export tests or target the backend export queue endpoints.
|
||||
|
||||
#### Flow 6: Media Upload (Storage/IO-heavy) (`media-upload.js`)
|
||||
|
||||
1. Login.
|
||||
2. Direct upload: `POST /api/main/methods/upload-file-media-object` (multipart, small PNG).
|
||||
3. Chunked upload: `POST /api/main/methods/create-upload-session` → `upload-chunk` x N → `assemble-file-media-object` (large PNG).
|
||||
4. URL-based upload: `POST /api/main/methods/create-file-media-object-from-url` (if a stable external image URL is available).
|
||||
|
||||
#### Flow 7: Font Upload (Storage/CPU-heavy) (`font-upload.js`)
|
||||
|
||||
1. Login.
|
||||
2. `POST /api/main/methods/create-upload-session` (for the font file)
|
||||
3. `POST /api/main/methods/upload-chunk` for each chunk
|
||||
4. `POST /api/main/methods/create-font-variant` → `{team-id, font-id, font-family, font-weight, font-style, uploads: {"font/ttf": "<session-id>"}}`
|
||||
5. `GET /api/main/methods/get-font-variants?team-id=<id>`
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 – Scenarios & Orchestration (Day 6)
|
||||
|
||||
Define k6 `options.scenarios` that mix the flows to simulate realistic traffic.
|
||||
|
||||
**Example scenario mix for 1000 VUs:**
|
||||
|
||||
| Scenario | Script | VUs | Arrival Rate | Duration | Notes |
|
||||
|----------|--------|-----|--------------|----------|-------|
|
||||
| `lifecycle` | `lifecycle.js` | 100 | 1/s (ramp 0→100 over 5m) | 10m | Full CRUD, most realistic. |
|
||||
| `workspace_open` | `workspace-open.js` | 400 | 5/s (ramp 0→400 over 5m) | 10m | Read-heavy, simulates many editors opening files. |
|
||||
| `workspace_edit` | `workspace-edit.js` | 200 | 2/s (ramp 0→200 over 5m) | 10m | Write-heavy, independent files. |
|
||||
| `workspace_edit_concurrent` | `workspace-edit.js` | 30 (10 groups of 3) | 0.5/s | 10m | 3 VUs per file, measures conflicts. |
|
||||
| `viewer` | `viewer.js` | 200 | 3/s (ramp 0→200 over 5m) | 10m | Read-heavy, simulates public/private viewers. |
|
||||
| `media_upload` | `media-upload.js` | 50 | 0.5/s | 10m | Storage stress. |
|
||||
| `font_upload` | `font-upload.js` | 20 | 0.2/s | 10m | Font processing stress. |
|
||||
|
||||
**Thresholds:**
|
||||
- `http_req_duration{p95} < 200ms` for `get-profile`, `get-teams`, `get-projects`.
|
||||
- `http_req_duration{p95} < 500ms` for `get-file` (small), `search-files`.
|
||||
- `http_req_duration{p95} < 2000ms` for `get-file` (large / 1000 shapes).
|
||||
- `http_req_duration{p95} < 1000ms` for `update-file` (small).
|
||||
- `http_req_duration{p95} < 3000ms` for `update-file` (large).
|
||||
- `http_req_duration{p95} < 5000ms` for `upload-file-media-object` (1 MB).
|
||||
- `http_req_duration{p95} < 10000ms` for `assemble-file-media-object` (5 MB chunked).
|
||||
- `http_req_failed < 1%` globally.
|
||||
- `http_req_failed{code:revn-conflict} < 5%` for `workspace_edit_concurrent`.
|
||||
|
||||
**Correlation with backend metrics:**
|
||||
- Scrape `/metrics` before, during, and after the test.
|
||||
- Key Prometheus metrics to watch:
|
||||
- `rpc_main_timing_seconds` (histogram/summary, labeled by command name)
|
||||
- `rpc_management_timing_seconds`
|
||||
- `http_server_dispatch_timing_seconds`
|
||||
- `websocket_active_connections` (if any WS is active)
|
||||
- `websocket_messages_total`
|
||||
- JVM hotspot metrics (`process_cpu_seconds_total`, `jvm_memory_bytes_used`, `jvm_threads_current`)
|
||||
- HikariCP metrics (if exposed; check `com.zaxxer.hikari:type=Pool` via JMX or custom Prometheus exporter)
|
||||
- PostgreSQL: `pg_stat_activity` count by state.
|
||||
- Redis: `INFO` `connected_clients`, `used_memory`.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 – Advanced `update-file` Testing (Days 7–8)
|
||||
|
||||
Because `update-file` is the core of the product and the user explicitly noted that **file size matters** and **concurrent editing is difficult**, we need a dedicated deep-dive.
|
||||
|
||||
#### 4.1. File Size Tiers
|
||||
|
||||
Create a `file-size-matrix.js` script that parameterizes the file size:
|
||||
- `SMALL_FILE_ID`: 1 page, 10 shapes.
|
||||
- `MEDIUM_FILE_ID`: 1 page, 100 shapes.
|
||||
- `LARGE_FILE_ID`: 1 page, 500 shapes.
|
||||
- `XLARGE_FILE_ID`: 1 page, 1000+ shapes, or a multi-page file.
|
||||
|
||||
Run `workspace-edit.js` against each tier separately and plot:
|
||||
- `update-file` latency vs file size.
|
||||
- `get-file` latency vs file size.
|
||||
- Backend CPU and DB time vs file size.
|
||||
|
||||
#### 4.2. Concurrent Editing Strategy
|
||||
|
||||
**Problem:** `update-file` uses optimistic concurrency control (`revn`). If two users submit the same `revn`, the second gets a conflict.
|
||||
|
||||
**Backend behavior:**
|
||||
- `update-file` acquires an advisory lock (`db/xact-lock! conn id`) on the file ID.
|
||||
- It checks `revn` and `vern` conflicts.
|
||||
- It processes changes, validates, updates the file, and sends notifications via `msgbus`.
|
||||
- The transaction duration is what we want to measure.
|
||||
|
||||
**Test design for concurrent editing:**
|
||||
- **Shared file pool:** Pre-create 50 files of `MEDIUM` size.
|
||||
- **VU grouping:** Use k6 `scenarios` with `per-vu-iterations` or `shared-iterations`. Assign groups of 3 VUs to the same file ID.
|
||||
- **Conflict measurement:** Do *not* synchronize `revn` between VUs. Let them race.
|
||||
- Track `http_req_failed{code:revn-conflict}`.
|
||||
- Track retry latency (if a VU retries after fetching the latest `revn`).
|
||||
- This gives us the **natural conflict rate** under load, which is a realistic product metric.
|
||||
- **If the natural conflict rate is too high (>20%):**
|
||||
- Add a small `sleep()` jitter (0–500 ms) between `get-file` and `update-file` to spread out the requests.
|
||||
- Or, use a tiny shared counter (e.g., a small HTTP endpoint or Redis) that VUs read to get the "next" `revn`. This is less realistic but gives a cleaner latency measurement.
|
||||
|
||||
**Action:** Create `workspace-edit-concurrent.js` with the grouping logic and conflict-rate thresholds.
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 – CI Integration & Reporting (Days 9–10)
|
||||
|
||||
1. **Runner script (`run.sh`):**
|
||||
- `./run.sh smoke` for a 1-VU, 1-iteration smoke test. ✅ Done
|
||||
- `./run.sh lifecycle -v 100 -n 10` for the standard run.
|
||||
- Add `--scenario` flag to run individual flows or the full mix.
|
||||
|
||||
2. **Output:**
|
||||
- k6 JSON/CSV output to `performance/results/<timestamp>/`.
|
||||
- Prometheus snapshot diff (before vs after).
|
||||
- Grafana screenshot or dashboard export.
|
||||
|
||||
3. **Grafana Dashboard:**
|
||||
- Panel: `p95 latency by RPC command` (from `rpc_main_timing_seconds`).
|
||||
- Panel: `HTTP requests/sec` (from k6).
|
||||
- Panel: `Error rate by command` (from k6).
|
||||
- Panel: `DB connection pool` (if available).
|
||||
- Panel: `JVM heap used`.
|
||||
- Panel: `update-file conflict rate` (custom metric from k6).
|
||||
- Panel: `File size vs latency` (from the matrix test).
|
||||
|
||||
4. **Regression guard:**
|
||||
- Store baseline results in `performance/baselines/`.
|
||||
- After any backend change, run the baseline scenario. If p95 increases by >20% for any critical command, fail the CI step.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Considerations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| **Scale: 1000 VUs creating data simultaneously will exhaust DB connection pool or storage quota.** | Pre-seed the data pool. Use a dedicated perf DB. Monitor `pg_stat_activity` and HikariCP metrics. |
|
||||
| **Media upload (images/fonts) will saturate network I/O before the backend is stressed.** | Run the load test from the same datacenter/VPC as the backend. Use small dummy files for most tests; reserve large files for a dedicated storage-stress scenario. |
|
||||
| **`update-file` conflicts under 1000 VUs may be so high that the test becomes a conflict test, not a latency test.** | Measure both. The conflict rate is itself a critical metric. If it is too high, we can add jitter or use independent files. |
|
||||
| **Exporter service is a separate bottleneck.** | `export.js` should target the backend queue endpoint, not the full export pipeline, unless we want to test the exporter too. If exporter is in scope, run it as a separate scenario. |
|
||||
| **Chunked upload creates many temporary DB rows (`upload_chunk` table).** | The backend has a `upload-session-gc` cron job. Ensure it runs after the test, or clean up manually. |
|
||||
| **Font upload shells out to FontForge and WOFF tools.** | This is CPU-intensive and may be a bottleneck. Run font upload as a separate, low-VU scenario to measure the processing time without blocking other tests. |
|
||||
| **Prometheus metrics may not expose DB pool wait time.** | Add a custom JMX exporter for HikariCP if needed, or query `pg_stat_activity` directly. |
|
||||
| **Cleanup:** 1000 VUs creating teams/files will leave logical deletions or orphaned storage objects.** | Use a dedicated perf environment. Run a cleanup script after the test that deletes all seeded data via the RPC API. |
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### How to verify the test harness itself works
|
||||
|
||||
1. **Smoke test:** Run each k6 script with `1 VU, 1 iteration` against a local devenv. Verify all requests return `200` and the response body is valid JSON.
|
||||
2. **Baseline run:** Run `workspace-open.js` with `10 VUs, 60 s` against a clean devenv. Record baseline p95 and p99 latencies.
|
||||
3. **Regression guard:** After any backend change, re-run the baseline. If p95 increases by >20%, flag it.
|
||||
4. **Saturation test:** Ramp `workspace-edit.js` to 100 VUs editing independent files. Monitor backend CPU and DB connection pool. The test should reveal the breaking point where `update-file` latency spikes.
|
||||
5. **Media upload stress test:** Run `media-upload.js` with 50 VUs uploading 1 MB files. Verify storage throughput and no `413` errors.
|
||||
6. **Font upload stress test:** Run `font-upload.js` with 10 VUs. Verify FontForge CPU usage and no timeouts.
|
||||
|
||||
### Manual validation checklist
|
||||
|
||||
- [x] `POST /api/main/methods/login-with-password` with JSON body returns a session cookie. (Validated via k6 lifecycle)
|
||||
- [x] `GET /api/main/methods/get-profile` with `Accept: application/json` returns JSON. (Validated via k6 lifecycle)
|
||||
- [ ] `curl -H "Accept: application/json" http://localhost:6060/metrics` returns Prometheus text.
|
||||
- [ ] Backend fixtures create at least 100 test users and 100 test files.
|
||||
- [x] A `update-file` request with a minimal `changes` payload succeeds and returns `{"revn": N}`. (Validated — needs full shape with selrect, points, transform, frame-id)
|
||||
- [x] A `upload-file-media-object` multipart request succeeds and returns a media object ID. (Validated via k6 lifecycle)
|
||||
- [ ] A chunked upload (`create-upload-session` → `upload-chunk` → `assemble-file-media-object`) succeeds.
|
||||
- [ ] A `create-font-variant` request with chunked uploads succeeds.
|
||||
|
||||
---
|
||||
|
||||
## Immediate Next Steps (if approved)
|
||||
|
||||
1. ~~Create `performance/` directory and `README.md`.~~ ✅ Done
|
||||
2. ~~Write `penpot-client.js` (k6 shared module) with `login()`, `rpc()`, `uploadMultipart()`, and `uploadChunked()` helpers.~~ ✅ Done (388 lines, JSON transport, cookie auth, session headers, tagged metrics)
|
||||
3. ~~Write a manual `curl` validation script~~ — Skipped; JSON compatibility confirmed via k6 smoke test.
|
||||
4. Write a data seeding script (`performance/scripts/seed-data.js`) that creates 100+ users, teams, projects, and files of varying sizes.
|
||||
5. ~~Write the first k6 script: `lifecycle.js`~~ ✅ Done (398 lines, 12-step CRUD flow, all checks passing)
|
||||
6. ~~Run a 1-VU smoke test against local devenv and commit the baseline results.~~ ✅ Done (10/10 checks, 0% failure, ~10s iteration)
|
||||
7. Write `workspace-open.js` and `workspace-edit.js`.
|
||||
8. Write `media-upload.js` and `font-upload.js`.
|
||||
9. Define the `1000-VU` scenario mix in `options.js` (shared scenario config).
|
||||
10. Run the first 100-VU ramp test and capture Prometheus metrics.
|
||||
|
||||
---
|
||||
|
||||
**Plan Author:** Senior Software Architect
|
||||
**Status:** Phase 1 complete, Phase 2 partially complete (lifecycle.js done). See "Current Progress" section above.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user