197 Commits

Author SHA1 Message Date
Juan de la Cruz
ea61730f61
✨ Add review mode, review comments and tasks to design reviews (#12939)
* ✨ Add review mode, review comments and tasks to design reviews

A design review now opens a read-only review mode. The left panel shows
the pages and layers with the View only badge, the toolbar offers the
review tools (move, comment, comment an area, create a task, show the
changes) and the right panel holds the inspect of the selection, the
comments of the review and its tasks. A bar at the bottom names the
review, shows the progress of its tasks and opens its details.

Comments can point at a spot, a layer or an area, and a comment written
in a review says so with its own mark, wherever the file is open. The
layers panel counts the open comments of each layer.

Tasks are new: work on a file, assigned to someone and tied to a layer
or to a spot of the canvas. A comment can become a task and stays
attached to it. A task moves through open, done, verified, reopened and
won't do; anybody may move it, and each move is an event that records
who made it. Deleting a layer with open tasks moves them to its board
and offers to undo the deletion or to mark them done; a task can then
be linked to another layer or get its layer back. Assignees are told by
email. Merging a branch moves its tasks and their threads into main:
the open ones keep going, the closed ones stay as its history, and the
merge dialog lists them and can tell their assignees.

The details of a review show its overview, its tasks and its activity:
what the saves of the branch did to each layer since the review opened,
by person and day, with the values before and after. The changes can be
shown on the canvas one after another.

The backend adds the review fields of a comment thread, the task and
task event tables, the activity of a review (recorded by each save of a
branch under review) and the commands of the tasks.

* ♻️ Bring the design review styles and components up to the DS

Move what the review UI repeated into the design system: layer-chip*,
progress-bar*, comment-card*, link-button* and textarea*, each with its
story and docs. The screens keep the look they had: the same review states
compare equal, pixel by pixel, before and after.

The stylesheets follow the rules of the UI guide: local variables for the
colors that change by state, the right tokens, no unused imports and fewer
element and nested selectors. What has no DS token or typography says so in
a TODO. The aliases left in comments.scss resolve to their tokens.

The components get complete schemas and a single owner for the copyright.
Translations take literal keys and templates instead of dynamic keys and
glued strings, and comments.cljs no longer depends on the review tasks: the
workspace passes it what a thread shows about its task.

Fix three bugs of the review:

- The inspect took the width of a narrower panel and its height from the
  window, so it looked squeezed and cut its last rows. It now fills the
  panel that holds it.
- Enter on the menu of a task opened the task. The row now reacts only to
  keys on itself, and the menu keeps Enter from the workspace shortcuts.
- The resolved and lost marks ignored the pointer: hover now wins over the
  state of a mark.

* ✨ Turn each review message into a task and restore the filter dropdown

Each message of a thread can now become a task, so one thread can hold
several. The new file_task.comment_id column records the message, the
migration fills it with the first message of the thread for existing tasks,
and the backend rejects a message from another thread or one that already
has a task.

Comments and tasks of a review behave like normal ones. The canvas marks
show a short preview on hover and there is one mark per task. A blue label
marks what belongs to a review, in the thread and in the sidebar, and a
selected card takes a blue border there and the primary color elsewhere.
The new review-label* component holds the label.

The sidebar and the review panel go back to a filter dropdown in place of
the chips. Comments offer the views all, yours and mentions, with an option
to hide the resolved ones. Tasks offer all, yours, open and to verify, with
an option to hide the closed ones. Resolved comments now show by default.

"New task" outside a review now arms the task tool: it shows the task cursor
and the next click opens the task form, not a comment. Then the comment tool
returns, and Esc leaves the tool.

The details modal shows changed properties in the format of the compare
dialogs, and showing a task on the canvas syncs the WASM viewport. The sidebar
cards put the layer chip on the line of the replies, and task states lose the
dot and take a color each.

Fix these bugs:

- The hover card of the activity tab crashed on a change with no token.
- The assignee and reviewer dropdowns came out cut. The select now portals
  into the closest dialog, because the modal makes the rest of the page inert.
- The review label cut the header of the thread. It now sits below it.

* ✨ Group overlapping review marks and let tasks change assignee

Comments and tasks of a review that fall on the same spot now show as one
mark with a counter in a white bubble. A click fans them out, as the comment
bubbles do, and the user picks one. They group on the same layer and, by
proximity, at a low zoom. The card of the mark lists up to five items.

Hovering a mark outlines its layer, or its board when the layer is gone. The
WASM viewport draws no outlines under the comments tool, so it now draws the
highlighted ones there. Opening a comment from a mark closes the open task.

A task can change assignee from the selector in its state and from "Assign
to" in its menu, which holds the team in the same popover. The change shows
right away and goes to the history of the task. A closed task keeps its
assignee.

Deleting a task or a comment now asks first, with the usual confirmation. A
task deleted from the details of a review brings the dialog back on its tab.

The state of a task takes two rows, in its panel and in a thread: state and
layer, then assignee and the review it comes from, which a review omits. The
text of tasks and comments sits in a box. The history no longer says that the
task was created.

* ✨ Keep the branch bar and the palettes at one level on a branch

The open branch bar covered the horizontal scroll of the canvas. It now sits
as far from the bottom as the review bar. Folded, it stays flush.

On a branch the palettes no longer lift over the branch bar: both sit at the
same level and work as a toggle. Opening one folds the other, also through
the palette shortcuts, and both can stay folded. The new `:bottom-bar` state
says which one is open. It exists only while the branch bar does, so a file
that is not a branch keeps the palettes on their layout flags and at their
usual place, and what happens on a branch does not reach them.

The branch bar comes open on a branch and centers on the canvas, not on the
window, so that it clears the folded palettes at usual widths. The room it
published for the palettes goes away with the lifting.

* ✨ Give every task a thread and tidy the design review screens

Every task now lives in a message of a comment thread, so anyone can go on
commenting on it. A task made from scratch gets a thread whose first message
is its title, on its layer and in its review, created in the same
transaction. A comment turned into a task keeps its thread, as before. The
panel of a task with no thread is gone: a task always opens as its thread.
Migration 0814 gives a thread to the tasks that had none. Deleting a thread
deletes its tasks, after a confirmation that says so.

The anchor of a task was a plain map, not a point, so a thread on a layer
refused replies. It is now a point.

The details of a review filter the tasks by state with the chips of the
activity, in the colour of each state, and show how many are closed on the
right. The "New task" button leaves the dialog.

Descriptions look the same everywhere: the box of the branch info dialog,
an "Edit" button and a dashed box when empty. Read, they grow up to 400px
and scroll inside; edited, the textarea starts at 400px. The text of a
review in the sidebar does the same once unfolded.

In the review bar, the author gets a "Publish latest changes" button when
the branch moved past the review, with a hint that approvals must be
renewed. The others keep a badge, now "Unpublished changes", with a hint.
The bar refreshes once the publish lands.

Smaller changes: the resolve checkbox lines up with the task button and has
a tooltip, messages sit in a filled box, task events take an outline, and
the change navigator sits right above the review bar.

* 💄 Sort the test runner requires that cljfmt rejects

The Frontend Tests lint step fails on runner.cljs: the requires for
workspace-tasks-test, review-details-test and review-markers-test sat
out of alphabetical order. This is the output of cljfmt fix.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 🐛 Test the design review mode in the pull request spec

The review bar replaced the branch banner, so four tests still drove
the branch bar: a visible Compare button, the rectangle tool and the
Review button. They now drive the design review instead.

* Opening a file with a pr-id enters the review: the bar names the
  review, the header trades the file name for the review title, and
  comparing, merging and cancelling sit in the bar's options menu.
* The compare dialog opens from that menu.
* The review is read-only rather than an editable file: the review
  toolbar offers its five tools, the drawing tools are gone, the
  sidebar says View only, and the right panel is the review's own
  Inspect, Comments and Tasks tabs.
* The verdict comes from the bar's Submit review button and the
  dialog it opens, scoped by role so the bar action and the dialog
  action do not collide.

The tests also reach the review bar by its region label instead of
walking up three parents from the Reviewing the design review text,
which broke as soon as the bar's markup changed.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 🐛 Count only the test thread's acquisitions in the climit test

acquires-the-command-own-limits-before-the-root-limits failed on CI
with an extra "root/global" ahead of the expected order. The test
records metrics through with-redefs, which is global, so a call of
another thread of the test JVM, such as an RPC an earlier test left
running, can take a root limiter inside the window and land in the
same atom.

Record the thread with each metrics call and keep, in
acquired-in-order, only the acquisitions of the calling thread. Both
tests that read it acquire their limiters on the test thread. The
same class of flake was fixed on develop for the waiting-count test
(12d73d0f63); this one is still open there.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 📚 Describe the task moves the server allows in the table comment

The comment on file_task said a task is reopened only from done and
that editors move tasks. files_task.clj::transitions also reopens a
verified or won't-do task, and set-task-status lets anybody who can
read the file move one. Say the same in the schema.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

---------

Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-11-17 22:30:22 +01:00
Álvaro Tejero Cantero
4cb0e452ff
Merge branch 'develop' into iop-vc-preview
Bring iop-vc-preview up to develop b438676d6c.

Conflicts:
- migrations.clj: develop's 0157-migrate-legacy-file-data and
  0158-del-forbidden-file-library-rels keep develop's order, and the
  branch's 0800-0808 follow them. No file names collide, so nothing
  is renumbered.
- files_update.clj: develop's save-path rewrite wins (commit-by-commit
  rejection, page-scoped validation, the file carrying ::changes and
  ::rejected). The branch's op-log save is ported onto it instead of
  the earlier [file op-changes media-index] triple:
  process-changes-and-validate attaches the media fix-up remap as
  ::media-index, and update-file* builds the op-log vector from the
  accepted changes (remapped) behind the repair changes. A branch
  save persists only when develop's saved? holds, and its audit and
  log outcome is "rejected" when the first commit is refused.
  The tpoint threading, persist-branch-file!, the public
  invalidate-caches!, and the branch audit props are kept.
- Repair and validation timing tests: take develop's calls, since
  process-changes-and-validate returns a file again.
- binfile_test.clj: keep the branch import-overwrite test and
  develop's library relation tests.
- dashboard.cljs, dashboard_test.cljs: union of requires and tests.
- viewer.cljs: develop's optional page-id plus the branch's pr-id.
- left_header.cljs: develop's raw-svg logo, recent-files href and
  project link, inside the branch's wrapper with the review badge,
  branch title and branch bar. The unused deprecated-icon require
  and nav-to-project go.
- en.po, es.po: three-way merge by msgid. Every entry of both sides
  is kept, develop's removals of a duplicate errors.save-retrying and
  of integrations.mcp-server.title.beta stand, and labels.retry
  carries develop's references plus the branch's.

Adaptations to develop:
- email.clj, files_pull_request.clj: the review request email joins
  develop's template registry and goes through send-template, as the
  comment emails it mirrors now do. The registry coverage test lists
  it, and the rollback test of the pull request commands injects its
  failure into send-template, the function the create path now calls.
- rpc_file_pull_request_test.clj: pr-flags is read at test time
  instead of at namespace load. The loaded value held the production
  defaults, which now enable the admin console (#12475), and it
  replaced th/state-init's test flags, so creating a profile called
  nitrate and failed with :nitrate-not-configured.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-11-11 17:51:20 +01:00
Andrey Antukh
41654aa1b7
✨ Restrict management RPC methods to their declared shared keys (#12736)
* ✨ Restrict management RPC methods to their shared keys

Management methods now declare the shared-key ids allowed to call them
with ::rpc/shared-keys. A wrapper in wrap-management checks the key id
from the request before opening a transaction, so a valid key outside
the declared set answers 403 shared-key-not-allowed. Methods without
the metadata keep accepting any configured key.

Scope upload-tempfile to exporter and media-processor, file analysis to
nexus, durable jobs to exporter, and the Nitrate methods to
admin-console. The test helper injects a declared key for the allowed
path, and a coverage test fails when a management method is missing or
declares the wrong set.

Closes #12730

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Harden shared-key authorization scope and tests

Rename the per-method metadata to ::rpc/allowed-shared-keys so it reads
as the allowed key ids, not the secrets. Fix the management test helper
to resolve the key from the supplied request and to keep an explicit nil,
which prevents a rejection test from silently using an allowed key.

Strengthen coverage: assert every management namespace is scanned, check
all four job methods reject a key they do not declare, verify a rejected
complete-job leaves the row and its events untouched, and exercise both
keys allowed on upload-tempfile. Log denied keys with the method name and
key id, without the secret, and update the audit-log memory for
push-audit-events.

Closes #12730

AI-assisted-by: deepseek-v4.1-flash
2026-11-03 13:49:36 +01:00
Álvaro Tejero Cantero
0cc2e97338
Merge develop into iop-vc-preview
Bring iop-vc-preview up to develop 43f3bda8b5, which carries the
final squash of penpot#12420 (93d36d04d8).

iop-vc-preview carried an earlier copy of penpot#12420 (42a2e33ffe
and a6c20b3e95). Every file the early or the final copy touches was
merged three-way with a6c20b3e95 as the base, so develop's side wins
for every penpot#12420 hunk and iop-vc-preview keeps only its own
deltas. Changes from other develop PRs (#12524, #12192, #12731,
#12686, #12442) are kept.

Adaptations to develop:
- files_update.clj: apply-changes returns the triple
  [file op-changes media-index] on the #12524 auto-repair path too,
  and a repaired branch save logs the repair changes ahead of the
  client's ones.
- files.clj: absorb-library-by-file takes develop's update-file! map.
- migrations.cljc: migrate keeps its target arity and develop's
  ordered :migrations set.
- climit.edn: the branch global limits declare a queue, as #12686's
  example-config-bounds-every-wait requires.
- Tests: the repair tests destructure the triple, the branch import
  test uses the #12192 export cfg keys, and the ref-paths test
  classifies :set-own-tokens-status.
- Dashboard comments and placeholder take develop's design-system
  icons around iop-vc-preview's review list; en.po and es.po keep the
  union of both sides.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-11-03 12:47:08 +01:00
Dr. Dominik Jain
c0f0b14540
🎉 Add direct MCP feedback channel via dedicated tool (#12551)
* ♻️ Move devenv-only tools to a separate folder

* 🎉 Add an MCP feedback channel

Add a user-approved feedback tool and an MCP-token-authenticated backend
endpoint that queues plain-text email. Enable the tool through the
configured backend URI in both local and multi-user mode.

Validate subject and feedback limits, enforce per-profile rate limits,
and document configuration. Cover authentication, queueing, tool
availability, and failure handling with backend and MCP tests.

Closes #12490

AI-assisted-by: gpt-6

* ♻️ Move MCP feedback submission into RPC

Use the standard RPC wrappers for feedback and remove the dedicated
HTTP endpoint, route wiring, and custom rate limiter.

Restrict MCP tokens to opted-in routes and commands. Add a shared RPC
call helper to the MCP backend client and adapt the existing tests.

AI-assisted-by: gpt-6

* 🐛 Harden MCP feedback auth, validation and messages

Accept MCP tokens when either the mcp or the access-tokens flag is on, so
instances that enable MCP without personal access tokens can send
feedback. Personal tokens still need access-tokens. Keep the issuer in one
constant, decode tokens without raising to reject unknown issuers, and
name the issuer and type checks.

Reject control characters in the subject, and in the feedback all but
tabs and line breaks, in both the command and the tool, so a NUL answers
400 instead of failing when the mail job is stored. Answer 403 with
profile-blocked or profile-not-active for unusable profiles, record only
who sent feedback in the audit event, and add doc metadata.

Replace on OAuth approval an MCP token issued before MCP tokens had their
own issuer, or one this instance cannot decrypt, since the backend rejects
both.

In the MCP server, mention disabled access tokens on 401, drop the unused
503 case, and say that delivery happens in the background. Document the
flags in the README.

AI-assisted-by: claude-opus-5-5

* 📚 Document the MCP feedback tool and MCP token rules

List submit_feedback among the MCP tools and add an MCP feedback section to
the configuration guide covering PENPOT_MCP_BACKEND_URI,
PENPOT_MCP_FEEDBACK_EMAIL, the flags and the email settings. Add commented
examples of both variables to the compose file.

In the agent memories, record when OAuth approval replaces an MCP token,
how the MCP server reaches the backend, and which flags, routes and
commands accept MCP tokens.

AI-assisted-by: claude-opus-5-5
2026-10-29 10:35:46 +01:00
Álvaro Tejero Cantero
1342b27f6f
Merge remote-tracking branch 'origin/develop' into iop-vc-preview
Merges develop at f6f9df9a1a. Twelve files conflicted; each keeps
both intents.

files_update.clj: develop's per-stage validation timings wrap the
branch save path. `update-file` keeps the branch's early timer and
passes it to `update-file*`, inside develop's try/catch timing log.
`process-changes-and-validate` times the changes application and
still returns the file, the fixed-up change vector and the media
index. The audit props carry develop's revn and modified-at plus the
branch props. The timing test destructures the new return shape.

v3.clj keeps the branch-file overwrite guard and takes develop's
`configure-import-transaction`. migrations.clj puts develop's 0155
to 0157 before the branch's 0800 to 0805, so the branch's stay the
tail of the vector.

errors.cljs checks the merged and archived branch refusals before
develop's retrying case. left_header.cljs keeps the branch wrapper,
the review badge and the branch bar on develop's markup: the status
dot without icons and the DS library icon on the shared badge.
palette.scss and assets.scss keep the branch clearance terms on
develop's logical properties. The test runners, requires and
deftests are unions.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-10-28 06:28:08 +01:00
Andrey Antukh
b633093336 Merge remote-tracking branch 'origin/staging' into develop 2026-10-17 16:28:59 +02:00
Andrey Antukh
70b317a976
🐛 Harden upload path handling and validation error bodies (#12067)
* 🚑 Reject forged upload paths outside multipart

Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.

app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Sanitize raw explain in generic validation error bodies

The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Add e2e path traversal tests for binfile and media upload

Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in binfile SVG test setup

The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in remaining upload tests

The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-17 16:01:04 +02:00
Andrey Antukh
dfc55a7192 Merge remote-tracking branch 'origin/staging' into develop 2026-10-16 12:48:49 +02:00
Andrey Antukh
a53481c3b7
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids

Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:

- prepare-rpc-event adds :team-id when the event carries a project or
  file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
  per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.

Closes #12219

AI-assisted-by: deepseek-v4.1-flash

* ✨ Emit delete and hard-delete audit events with entity data

The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:

- delete-team keeps returning nil but now carries the deleted row
  (id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
  props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
  with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
  params already carry team-id and member-id.

The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Announce cascade deletes and missing ids in audit events

The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.

Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile delete in audit events

The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile restore in audit events

The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Stop attributing SREPL events to the subject profile

profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Expose file analysis data for the nexus pipeline

The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.

Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename deletion audit events to action names

Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Read worker parallelism knobs with plain config keys

The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix binfile audit test with a real file row

`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix audit event fidelity from review findings

The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.

Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.

Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Nest cascade ids under :cascade in delete event

The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.

AI-assisted-by: muse-spark-1.3-contributor

* 📎 Update copyright name on file header

The company name changed; a single header in this branch still
carried the old one. Align it with the current form.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix nil event-id on webhook trigger audit rows

process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-15 13:01:18 +02:00
Álvaro Tejero Cantero
6ab7b3ebd6
Merge branch 'develop' into iop-vc-preview
Both conflicts are tail appends onto regions develop rewrote. In
migrations.clj keep both appended blocks, develop's two entries
first so the branch's four stay the tail of the vector. In
climit.edn take develop's rewritten entries and re-append the
branch's provisional block, moving the map's closing brace to the
last line.

AI-assisted-by: glm-5.3-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-10-14 16:34:48 +02:00
Andrey Antukh
9abcfebf60
✨ Run binfile import and export as durable jobs (#12056)
* ✨ Run binfile import and export as durable jobs

Import and export of `.penpot` files become durable jobs: two commands
create them and answer at once with a job id the caller follows, while
a worker of the `binfile` queue does the work. Each job keeps its own
ledger (owner, params frozen at creation, result, progress history and
an expiry), and the legacy commands and their SSE keep working.

Why: both operations ran inside the request. A large import held it
open for minutes, progress only existed while the caller kept the
stream open, and a client that went away left work nobody recorded,
retried or could cancel. The unified `job` substrate already dispatched,
leased, retried and logged events, but it had no user-facing ledger.

How:
- The substrate gains the pieces a user-facing job needs: optional
  job-def metadata (family, resource-role), an expiry on submit, the
  owner in the runner context, and job resources in their own bucket,
  written with the profile of the job and never deduplicated.
- One generic command per family: the caller names the job and sends its
  business params, the registry checks the family, the job-def decodes
  and validates, and permission and quota are checked first.
- Progress taps of the core become milestones of the job: a shared stage
  vocabulary with self-contained counters, throttled heartbeats, and an
  interrupt that stops the run inside its own transaction on cancel.
- Cancel reaches pending and running jobs from both modals, and closing
  a modal mid-run cancels the job it was following.
- The export re-checks the read permission at run time and answers with
  the package descriptor; the import re-checks the edition permission,
  runs the core in a transaction so a cancellation rolls it back, and
  releases the consumed package and its temporary copy.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Handle externally-cancelled binfile jobs end to end

The `cancelled` outcome of a job nobody followed no longer hangs its
screens: the data layers emit an explicit terminal message for it.

- The export emits `{:cancelled true}` per file and the modal paints a
  neutral cancelled row instead of success; a message without an
  artifact never reaches the download branch.
- The import closes every entry with a dedicated cancelled message
  through the existing error rendering, so the wizard completes.
- Both flows are pinned by data-layer tests, including the red state
  they replace.

Also in this change: the progress-contract comment of the management
API, early business-params validation in create-import-job (before
assembling the upload), a `release-resource` helper without fabricated
job ids, a note on the heartbeat write cost, and a memory cleanup.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Pre-translate the cancelled import message

The cancelled entry of the import wizard carried its message as a key
for the UI to translate, which needed a rehash marker comment. Like
its neighboring errors, it now translates eagerly in the data layer:
the literal call keeps rehash clean with no marker, and the tests
assert through the same call instead of the raw key.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Move cancelled strings under generic labels keys

The dashboard-specific cancelled key never shipped: before use it moves
under `labels.*` next to the generic strings both flows already use,
with Spanish translations following the established wording.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add minor improvements

* ♻️ Rename export job command to create-export-binfile-job

Export now has one command per job type, the shape the :export-assets
command needs: create-export-binfile-job freezes exactly the
:export-binfile job, so :name no longer travels in the body and only
:params does. Import keeps the generic envelope.

submit-job reads the queue from the job-def metadata ::jobs/queue-name
(falling back to :binfile), read by the command and never by the
substrate, so a job-def can route its work to another queue without a
change in the command.

The rename reaches the frontend and the tests: the binfile export is
asked for under its own name, and the exporter service keeps
create-export-job.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Polish the import and export status rows

The status row of a file already carries what the file is doing, so the
generic "Uploading file…"/"Downloading file…" lines are gone: the
import wizard drops its own (the export one went with the jobs
refactor), along with the now-unused style.

In the export, every file is queued up front, even the ones whose job
is created only after the previous one is over, and a file stays queued
until its first milestone: a worker picking the job up no longer ends
the queue. The queued text is a step smaller than the milestone.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Show the cancelled export and fix the import error

The cancelled export was unreachable: mark-file-cancelled sets :loading
false and the label lived inside the loading branch, so a cancelled
file only painted a neutral notification. The terminal state now
carries the label, and the export flags lose the `?` suffix
(predicates keep it; a data prop is not a question).

The import wizard rendered (:error entry) through tr, but every
producer writes user-facing text (a hint or a translated label), never
a key: it renders the text as-is now, and the dynamic-key ignores and
the stale comment are gone.

The total-text-limit comment no longer references the 200MiB it
replaced; the value is 500MiB in develop already.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Reflect the real job params in the creation commands

The creation commands declared their params as an open empty map, so the
RPC contract and the generated docs said nothing about them. Each command
now imports the schema from the job namespace
(`export-binfile/schema:params`, `import-binfile/schema:create-params`),
and the RPC validates the business params before the command runs. The
job-def still owns the params and re-validates the assembled ones at run
time. The commands no longer decode the params again: the RPC did it.

Import joins export as one command per job type: `create-import-job`
becomes `create-import-binfile-job`, its body no longer carries `:name`
and the job-def is resolved by name like export's. Its creation schema
keeps the version optional, because the command reads it from the package
header when the caller does not say it, and fills the manifest metadata
kept for audit.

A version out of range is now refused by the schema (`:params-validation`)
instead of the command's own `:unsupported-version` check, which is gone.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Stop the test system from running job runners

The test system is main/system-config plus main/worker-config with a
few components dissoc'ed. The runners of default, webhook and cron were
listed by hand; the binfile runner added with the durable binfile jobs
was not, so a real penpot/job-runner/binfile/0 thread ran during the
whole suite.

That runner races the tests that drive the loop by hand (run-batch plus
run-worker-loop): it takes the job the test just dispatched and
completes it, and the test then reads a state it did not produce. That
is the reported flake of binfile-jobs-integration-test, whose middle
assertion waits for a runner of another queue to not take the job. It
only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant
baked into main/worker-config differ from the one the test builds, so
the two used different queue keys locally.

Drop every [<profile> :app.worker/runner] by key shape instead of
listing the known queues, so a new queue cannot bring the race back.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Upload the package one chunk at a time and as a percentage

The upload milestone counts chunks, not units of work, so the row wrote
"Uploading the package 3/9" over a file the user had uploaded once: it
read as if nine packages were on their way. Write the stage as a
percentage of the file instead ("Uploading file… (33%)"), and make the
English text of the stage say file, not package.

The chunks also go one at a time now (concat-all instead of merge-all
with two in flight), so the percentage moves in a straight line instead
of jumping. upload-blob-chunked is the generic uploader, so media and
font uploads become sequential too.

AI-assisted-by: deepseek-v4.1-flash

* 🌐 Complete the Spanish catalogue

es.po was 25 keys short of en.po: the 15 that the durable binfile jobs
introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps
that predate them (errors.connection-error, errors.save-retrying, the
find shortcuts, workspace.header.retrying,
workspace.tokens.stroke-width and the mcp session labels).

sync -l es then sorts the entries, syncs the #: comments from en and
drops the 7 dashboard.import.progress.* keys en no longer has. The
msgid sets of en and es are now identical (0 missing, 0 extra).

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Move the job status keys into the jobs namespace

The three status words the durable import and export rows render were
filed under labels.*, next to the generic labels bucket, while their
progress stages live under jobs.progress.stage.*: the strings of one
feature sat in two places.

Rename them so the whole job vocabulary is one namespace:

  labels.queued           -> jobs.queued
  labels.export-cancelled -> jobs.export-cancelled
  labels.import-cancelled -> jobs.import-cancelled

No other locale has these keys yet, so the rename costs no translation.
The convention (a key names the feature that renders it; labels.* holds
the generic reusable words) is written down in the frontend/translations
memory.

AI-assisted-by: deepseek-v4.1-flash

* 🔥 Remove the translation keys nothing references

Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...)
call site left in frontend/src or common/src, and no literal reference
either:

  labels.uploading-file    a duplicate of jobs.progress.stage.upload
                           since the upload stage says "Uploading file…"
  labels.downloading-file  dead since the durable jobs replaced the old
                           import and export progress rows
  inspect.attributes.stroke.alignment.center
  inspect.attributes.stroke.alignment.inner
  inspect.attributes.stroke.alignment.outer
  workspace.toolbar.mcp-connect-here

The first two are leftovers of the durable binfile jobs, the other four
predate them. Both catalogues keep an identical msgid set afterwards.

AI-assisted-by: deepseek-v4.1-flash
2026-10-07 11:52:48 +02:00
Álvaro Tejero Cantero
2a5a235063
✨ Add the design preview of file branching and pull requests
This commit carries the design-branching preview by Juan de la Cruz:
elhombretecla/penpot:branching at 0ac265a3f5, which holds 45 commits
and 3 develop merges and was last synced with develop on 2026-09-16.
The branch is squashed together with a merge of origin/develop at
7c039231f7, so the tree is the merged result.

The merge resolved conflicts in six files:

- backend/src/app/migrations.clj: develop's 0155 and 0156 steps run
  first, then the branching steps; the names differ.
- backend/src/app/rpc/commands/viewer.clj: both requires stay.
- common/src/app/common/flags.cljc: both flags stay.
- frontend/src/app/main/ui/dashboard/file_menu.cljs: "Create branch"
  becomes a menu-item* with the same id, guard and handler.
- frontend/src/app/main/ui/dashboard/grid.cljs: the portal goes with
  develop's menu rewrite; the branches popover stays in the trigger.
- frontend/translations/en.po: both sides' entries stay, with
  develop's source references.

Co-authored-by: Juan de la Cruz <delacruzgarciajuan@gmail.com>
AI-assisted-by: mixed models
2026-10-06 10:36:54 +02:00
Andrey Antukh
d3ca9a0b83
✨ Add unified jobs substrate replacing legacy one-shot task system (#11542)
* 🎉 Move backend jobs to the unified job table substrate

Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.

New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.

New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.

New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.

All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.

Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.

AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5

* ✨ Address review comments on jobs substrate

Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.

The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-06 07:33:12 +02:00
Andrey Antukh
2255266d45
✨ Enable closed schemas for RPC methods (#11136)
* ✨ Enable closed schemas for RPC methods

* 🐛 Fix duplicate make-dummy-request test helper definition

The branch added a variadic DummyRequest/make-dummy-request pair but
left the pre-existing single-arg definition in place. Because it was
loaded last, zero-arg (make-dummy-request) calls added by
prepare-rpc-params and rpc-nitrate-test threw ArityException, which
broke 384 tests and caused 14 downstream assertion failures.

Remove the stale duplicate so the variadic definition is the only
one, and drop the now-unused yrq alias and duplicate yres alias.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Add focused tests for make-dummy-request helper

Pin the call contract of make-dummy-request, which the suite uses
in three styles: no arguments, a single options map, and keyword
arguments. The helper's redefinition shadowing in 8ca95adb98 was
only caught by a full-suite run with hundreds of unrelated errors;
these tests fail locally in a focused --focus run.

Cover the zero-arg defaults, map and keyword overrides, the
:body-bytes -> ByteArrayInputStream wrapping, :body-stream
precedence, and cookie readback. Also clarify the docstring to
list all supported call styles.

AI-assisted-by: deepseek-v4.1-flash

* 🚑 Prevent RPC client params from overriding auth context

Strip qualified keys from decoded request params before merging
them with the server-built auth context, so transit bodies can
no longer override ::profile-id, ::auth-type or ::token-perms.
Adds a regression test proving the override and the fix.

AI-assisted-by: muse-spark-1.3-contributor

* 📚 Merge backend subtleties memories under generic name

Rename rpc-db-worker-subtleties to subtleties and fold in
http-storage-filedata-subtleties, so the name no longer
enumerates topics. Update all mem: references accordingly.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add realistic tests for RPC auth override

Cover the transit wire vector and the real wrapped :get-profile
method with two database profiles, proving a session cannot read
another profile by smuggling :app.rpc/profile-id in the body.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add e2e test for RPC auth context override

Parametrize rpcPost with contentType, accept and query so e2e
can send hand-written transit bodies without new dependencies.
The new test proves a transit-smuggled :app.rpc/profile-id no
longer overrides the session in get-profile. Also fix the demo
email assertion in auth-flow to the current uuid format.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 14:03:02 +02:00
Andrey Antukh
a8e0b3c1f8 🐛 Add dedicated RPC methods for plugin registry operations
Add `add-profile-plugin` and `remove-profile-plugin` RPC methods for
atomic plugin registry operations, preventing manipulation via the
broader `update-profile-props` endpoint.

- Close the `:plugins` field in `update-profile-props` schema to
  eliminate the mass assignment attack vector for plugin data.
- Define `valid-permissions` and a closed `schema:permissions` enum to
  restrict plugin permissions to known values.
- Migrate the frontend to use the new granular RPC methods with
  optimistic updates and rollback on failure.
- Add comprehensive backend tests covering valid/invalid permissions,
  updates, removal, and rejection via old endpoint.

AI-assisted-by: qwen3.7-plus
2026-09-07 09:30:01 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Pablo Alba
ed04d509ed
🐛 Fix bad managed error on backend sso failure (#11247) 2026-08-17 12:40:18 +02:00
Andrey Antukh
49119e0339
♻️ Rename nitrate config to admin-console (#10929)
* ♻️ Rename nitrate config to admin-console

Rename user-facing configuration from 'nitrate' to 'admin-console':
- Feature flags: :nitrate -> :admin-console, :nitrate-bulk-create-profiles -> :admin-console-bulk-create-profiles
- Config keys: :nitrate-shared-key -> :admin-console-shared-key, :nitrate-backend-uri -> :admin-console-uri
- Shared-keys map entry: :nitrate -> :admin-console (setup.clj + main.clj)
- Env vars: PENPOT_NITRATE_SHARED_KEY -> PENPOT_ADMIN_CONSOLE_SHARED_KEY, PENPOT_NITRATE_BACKEND_URI removed (consolidated into PENPOT_ADMIN_CONSOLE_URI)
- Docker/nginx: PENPOT_NITRATE_URI -> PENPOT_ADMIN_CONSOLE_URI

Code namespaces, file paths, CSS classes, and i18n keys stay as-is.

AI-assisted-by: mimo-v2.5-pro

* ♻️ Rename initialize-user-in-nitrate-organization to initialize-user-in-organization

Part of the nitrate -> admin-console rename series. The function and all 9 references across 6 files have been renamed.

* ♻️ Rename :nitrate-bulk-create-profiles-not-allowed to :bulk-create-profiles-not-allowed

* ♻️ Inline nitrate-permissions into app.common.types.organization

- Delete app.common.types.nitrate-permissions and its test
- Move permission rules (allowed?, can-send-invitations?, etc.) into organization.cljc
- Harmonize all consumers to use alias cto for app.common.types.organization
- Update test runner and create organization_test.cljc
2026-08-03 17:23:55 +02:00
Marina López
94f51afb20
♻️ Normalize organization naming across Penpot (#10977) 2026-07-31 12:04:10 +02:00
Andrey Antukh
c467d98c9e Merge remote-tracking branch 'origin/staging' into develop 2026-07-23 17:06:56 +02:00
Andrey Antukh
2344ba22a6 🎉 Add error reports API and CLI tool
Implement RPC methods for querying server error reports with pagination
and filtering. Add CLI tool (tools/error-reports.mjs) for convenient
access with table and JSON output formats. Extract profile-id from audit
events and logging context for better error categorization. Build
improved HREF using request path when available.

AI-assisted-by: qwen3.7-plus
2026-07-22 14:18:51 +02:00
Pablo Alba
cc454de9ce
🐛 Fix wrap nitrate sso when there is no profile (#10592) 2026-07-08 14:21:34 +02:00
Pablo Alba
8fa15f240f
✨ Check for nitrate sso on move team to another organization (#10379) 2026-06-24 10:02:13 +02:00
Pablo Alba
b984e7bbe8 ✨ Add nitrate sso wards to organization navigation 2026-06-19 12:03:21 +02:00
Andrey Antukh
c5de4c27b0 Merge remote-tracking branch 'origin/main' into staging 2026-06-01 12:57:39 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
7d4be33d4f 🎉 Add telemetry anonymous event collection (#9483)
* 🎉 Add telemetry anonymous event collection

Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:

Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
  telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
  telemetry flag active, audit-log flag inactive

Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
  prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
  webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
  Keeps UUID/boolean/number values; for login/identify events preserves
  lang, auth-backend, email-domain; for navigate events preserves route,
  file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
  Backend: version, initiator, client-version, client-user-agent.
  Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
  storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
  access-token fields removed from context

Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option

Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
  extraction from email addresses

Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)

RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation

RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API

SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys

Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
  payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
  per iteration, encodes and sends each page, deletes on success,
  stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
  blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
  oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id

Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
  safe base64 for JSON-safe string transport

Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
  efficient telemetry batch collection queries

Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support

Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
  happy-path batch send and delete, failure retention, payload anonymity,
  context stripping, timestamp day precision, batch encoding round-trip,
  multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📎 Add pr feedback changes

---------

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-05-11 12:42:01 +02:00
Andrey Antukh
cd4a4da0f2
🎉 Add telemetry anonymous event collection (#9483)
* 🎉 Add telemetry anonymous event collection

Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:

Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
  telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
  telemetry flag active, audit-log flag inactive

Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
  prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
  webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
  Keeps UUID/boolean/number values; for login/identify events preserves
  lang, auth-backend, email-domain; for navigate events preserves route,
  file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
  Backend: version, initiator, client-version, client-user-agent.
  Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
  storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
  access-token fields removed from context

Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option

Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
  extraction from email addresses

Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)

RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation

RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API

SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys

Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
  payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
  per iteration, encodes and sends each page, deletes on success,
  stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
  blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
  oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id

Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
  safe base64 for JSON-safe string transport

Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
  efficient telemetry batch collection queries

Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support

Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
  happy-path batch send and delete, failure retention, payload anonymity,
  context stripping, timestamp day precision, batch encoding round-trip,
  multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📎 Add pr feedback changes

---------

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-05-11 12:19:59 +02:00
Andrey Antukh
a395768987 🐛 Fix incorrect handlig of version restore operation (#9041)
- Add session ID tracking to RPC layer (backend and frontend)
- Send session ID header with RPC requests for request correlation
- Rename file-restore to file-restored for consistency
- Extract initialize-file function from initialize-workspace flow
- Improve file restoration initialization with wait-for-persistence
- Extract initialize-version event handler for version restoration
- Fix viewport key generation with file version numbers for proper re-renders
- Update layout item schema and constraints to use internal sizing state
- Add v-sizing state retrieval in layout-size-constraints component
- Refactor file-change notifications stream handling with rx/map
- Fix team-id lookup in restore-version-from-plugins

Improves request traceability across frontend/backend sessions and streamlines
the workspace initialization flow for file restoration scenarios.

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-04-21 20:50:41 +02:00
Andrey Antukh
6eccffb8bb
🐛 Fix incorrect handlig of version restore operation (#9041)
- Add session ID tracking to RPC layer (backend and frontend)
- Send session ID header with RPC requests for request correlation
- Rename file-restore to file-restored for consistency
- Extract initialize-file function from initialize-workspace flow
- Improve file restoration initialization with wait-for-persistence
- Extract initialize-version event handler for version restoration
- Fix viewport key generation with file version numbers for proper re-renders
- Update layout item schema and constraints to use internal sizing state
- Add v-sizing state retrieval in layout-size-constraints component
- Refactor file-change notifications stream handling with rx/map
- Fix team-id lookup in restore-version-from-plugins

Improves request traceability across frontend/backend sessions and streamlines
the workspace initialization flow for file restoration scenarios.

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-04-21 19:19:51 +02:00
Pablo Alba
90288e32d5 ✨ Show different info on nitrate dialog by connectivity 2026-02-20 10:19:25 +01:00
Andrey Antukh
fbf1240998 ⚡ Add several optimizations for fonts zip download
Mainly prevent hold the whole zip in memory and uses an
unified response type, leavin frontend fetching the blob
data from the assets/storage subsystem.
2026-02-16 10:14:50 +01:00
Andrey Antukh
4a7b89a1da
Merge pull request #8327 from penpot/niwinz-develop-rlimit-notifications
✨ Add proper mattermost notifications for rlimit rejects
2026-02-13 17:11:54 +01:00
Andrey Antukh
89935e2174 ✨ Make nitrate module loading conditional to flag
This removes the flag checking on each rpc method
2026-01-27 15:16:36 +01:00
Andrey Antukh
9c539dfb2f 🔥 Remove subscriptions related management module 2026-01-27 15:14:32 +01:00
Andrey Antukh
f5996a7235 ♻️ Make several improvements to management API authentication 2026-01-27 15:14:32 +01:00
Pablo Alba
d5abc52dac
🎉 Add first integration with nitrate (#7803)
* 🐛 Display missing selected tokens set info (#8098)

* 🐛 Display missing selected tokens set info

* ✨ Add integration tests to verify current active set

* 🎉 Integration with nitrate platform

* 🐛 Fix nitrate get-teams returns deleted teams

* ✨ Add nitrate to tmux devenv

* ✨ Add retry and validation to nitrate module

* ✨ Add photoUrl to profile on nitrate authenticate

* ✨ Move nitrate url to an env variable

* ♻️ Change Nitrate organization-id schema to text

* ♻️ Cleanup unused imports

* 🔧 Add control-center to nginx

* ✨ Add create org link

* 🔧 Fix nginx entrypoint

* 🐛 Fix control-center proxy pass

* 🎉 Add nitrate licence check

* Revert "✨ Add nitrate to tmux devenv"

This reverts commit dc6f6c458995dac55cab7be365ced0972760a058.

* ✨ Add feature flag check

* 🐛 Rename licences for licenses

* ✨ MR changes

* ✨ MR changes 2

* 📎 Add the ability to have local config on start backend

* 📎 Add FIXME comment

---------

Co-authored-by: Xaviju <xavier.julian@kaleidos.net>
Co-authored-by: Juanfran <juanfran.ag@gmail.com>
Co-authored-by: Yamila Moreno <yamila.moreno@kaleidos.net>
Co-authored-by: Marina López <marina.lopez.yap@gmail.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-01-27 10:04:53 +01:00
Andrey Antukh
88dcf9d1fe
🐛 Mark rpc calls as authenticated when shared key is used (#7901) 2025-12-22 12:18:36 +01:00
Andrey Antukh
81632a03dd ♻️ Make exporter upload resources using backend management api
Instead of custon shared fs approach. This commit fixes the main
scalability issue of exporter removing the need of shared-fs
for make it work with multiple instances.
2025-11-20 12:20:13 +01:00
Andrey Antukh
4fddf3d986 ♻️ Make management key derivable from secret key
Still preserves the ability to set management
2025-11-20 12:20:13 +01:00
Andrey Antukh
363b4e3778
♻️ Make the SSO code more modular (#7575)
* 📎 Disable by default social auth on devenv

* 🎉 Add the ability to import profile picture from SSO provider

* 📎 Add srepl helper for insert custom sso config

* 🎉 Add custom SSO auth flow
2025-11-12 12:49:10 +01:00
Andrey Antukh
28cf67e7ff
🎉 Add management RPC API (#7700)
* 🎉 Add management RPC API

And refactor internal http auth flow

* 📎 Adjust final url namings

* 📚 Update changelog
2025-11-10 17:10:59 +01:00
Andrey Antukh
cbae3dca34 ✨ Simplify the approach for return streamable body
Removing unnecesary syntax overhead with simplier abstraction
2025-11-04 10:56:05 +01:00
Andrey Antukh
62dffd30a4 ♻️ Refactor redis internal API
The main idea behind this refactor is make the
API less especialized for specific use of out internal
submidules and make it more general and usable
for more general purposes (per example cache)
2025-10-13 12:32:29 +02:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
Andrey Antukh
36914d1dc4 🐛 Relax ldap provider requirements 2024-11-14 09:40:38 +01:00
Andrey Antukh
88fb5e7ab5 ♻️ Update integrant to latest version
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
607e0c5c1d ✨ Move team invitations and access requests to a separate namespace
This commit also comes with:

- a fix for incorrect conflict handling on team access request creation
- a fix for incorrect handling of file-data when it is offloaded
- replace some inneficient queries with effcient ones
- remove redundant validation on creation of request-access
2024-11-13 14:16:18 +01:00