penpot/backend/test/backend_tests/climit_test.clj
Juan de la Cruz ea61730f61
✨ Add review mode, review comments and tasks to design reviews (#12939)
* ✨ Add review mode, review comments and tasks to design reviews

A design review now opens a read-only review mode. The left panel shows
the pages and layers with the View only badge, the toolbar offers the
review tools (move, comment, comment an area, create a task, show the
changes) and the right panel holds the inspect of the selection, the
comments of the review and its tasks. A bar at the bottom names the
review, shows the progress of its tasks and opens its details.

Comments can point at a spot, a layer or an area, and a comment written
in a review says so with its own mark, wherever the file is open. The
layers panel counts the open comments of each layer.

Tasks are new: work on a file, assigned to someone and tied to a layer
or to a spot of the canvas. A comment can become a task and stays
attached to it. A task moves through open, done, verified, reopened and
won't do; anybody may move it, and each move is an event that records
who made it. Deleting a layer with open tasks moves them to its board
and offers to undo the deletion or to mark them done; a task can then
be linked to another layer or get its layer back. Assignees are told by
email. Merging a branch moves its tasks and their threads into main:
the open ones keep going, the closed ones stay as its history, and the
merge dialog lists them and can tell their assignees.

The details of a review show its overview, its tasks and its activity:
what the saves of the branch did to each layer since the review opened,
by person and day, with the values before and after. The changes can be
shown on the canvas one after another.

The backend adds the review fields of a comment thread, the task and
task event tables, the activity of a review (recorded by each save of a
branch under review) and the commands of the tasks.

* ♻️ Bring the design review styles and components up to the DS

Move what the review UI repeated into the design system: layer-chip*,
progress-bar*, comment-card*, link-button* and textarea*, each with its
story and docs. The screens keep the look they had: the same review states
compare equal, pixel by pixel, before and after.

The stylesheets follow the rules of the UI guide: local variables for the
colors that change by state, the right tokens, no unused imports and fewer
element and nested selectors. What has no DS token or typography says so in
a TODO. The aliases left in comments.scss resolve to their tokens.

The components get complete schemas and a single owner for the copyright.
Translations take literal keys and templates instead of dynamic keys and
glued strings, and comments.cljs no longer depends on the review tasks: the
workspace passes it what a thread shows about its task.

Fix three bugs of the review:

- The inspect took the width of a narrower panel and its height from the
  window, so it looked squeezed and cut its last rows. It now fills the
  panel that holds it.
- Enter on the menu of a task opened the task. The row now reacts only to
  keys on itself, and the menu keeps Enter from the workspace shortcuts.
- The resolved and lost marks ignored the pointer: hover now wins over the
  state of a mark.

* ✨ Turn each review message into a task and restore the filter dropdown

Each message of a thread can now become a task, so one thread can hold
several. The new file_task.comment_id column records the message, the
migration fills it with the first message of the thread for existing tasks,
and the backend rejects a message from another thread or one that already
has a task.

Comments and tasks of a review behave like normal ones. The canvas marks
show a short preview on hover and there is one mark per task. A blue label
marks what belongs to a review, in the thread and in the sidebar, and a
selected card takes a blue border there and the primary color elsewhere.
The new review-label* component holds the label.

The sidebar and the review panel go back to a filter dropdown in place of
the chips. Comments offer the views all, yours and mentions, with an option
to hide the resolved ones. Tasks offer all, yours, open and to verify, with
an option to hide the closed ones. Resolved comments now show by default.

"New task" outside a review now arms the task tool: it shows the task cursor
and the next click opens the task form, not a comment. Then the comment tool
returns, and Esc leaves the tool.

The details modal shows changed properties in the format of the compare
dialogs, and showing a task on the canvas syncs the WASM viewport. The sidebar
cards put the layer chip on the line of the replies, and task states lose the
dot and take a color each.

Fix these bugs:

- The hover card of the activity tab crashed on a change with no token.
- The assignee and reviewer dropdowns came out cut. The select now portals
  into the closest dialog, because the modal makes the rest of the page inert.
- The review label cut the header of the thread. It now sits below it.

* ✨ Group overlapping review marks and let tasks change assignee

Comments and tasks of a review that fall on the same spot now show as one
mark with a counter in a white bubble. A click fans them out, as the comment
bubbles do, and the user picks one. They group on the same layer and, by
proximity, at a low zoom. The card of the mark lists up to five items.

Hovering a mark outlines its layer, or its board when the layer is gone. The
WASM viewport draws no outlines under the comments tool, so it now draws the
highlighted ones there. Opening a comment from a mark closes the open task.

A task can change assignee from the selector in its state and from "Assign
to" in its menu, which holds the team in the same popover. The change shows
right away and goes to the history of the task. A closed task keeps its
assignee.

Deleting a task or a comment now asks first, with the usual confirmation. A
task deleted from the details of a review brings the dialog back on its tab.

The state of a task takes two rows, in its panel and in a thread: state and
layer, then assignee and the review it comes from, which a review omits. The
text of tasks and comments sits in a box. The history no longer says that the
task was created.

* ✨ Keep the branch bar and the palettes at one level on a branch

The open branch bar covered the horizontal scroll of the canvas. It now sits
as far from the bottom as the review bar. Folded, it stays flush.

On a branch the palettes no longer lift over the branch bar: both sit at the
same level and work as a toggle. Opening one folds the other, also through
the palette shortcuts, and both can stay folded. The new `:bottom-bar` state
says which one is open. It exists only while the branch bar does, so a file
that is not a branch keeps the palettes on their layout flags and at their
usual place, and what happens on a branch does not reach them.

The branch bar comes open on a branch and centers on the canvas, not on the
window, so that it clears the folded palettes at usual widths. The room it
published for the palettes goes away with the lifting.

* ✨ Give every task a thread and tidy the design review screens

Every task now lives in a message of a comment thread, so anyone can go on
commenting on it. A task made from scratch gets a thread whose first message
is its title, on its layer and in its review, created in the same
transaction. A comment turned into a task keeps its thread, as before. The
panel of a task with no thread is gone: a task always opens as its thread.
Migration 0814 gives a thread to the tasks that had none. Deleting a thread
deletes its tasks, after a confirmation that says so.

The anchor of a task was a plain map, not a point, so a thread on a layer
refused replies. It is now a point.

The details of a review filter the tasks by state with the chips of the
activity, in the colour of each state, and show how many are closed on the
right. The "New task" button leaves the dialog.

Descriptions look the same everywhere: the box of the branch info dialog,
an "Edit" button and a dashed box when empty. Read, they grow up to 400px
and scroll inside; edited, the textarea starts at 400px. The text of a
review in the sidebar does the same once unfolded.

In the review bar, the author gets a "Publish latest changes" button when
the branch moved past the review, with a hint that approvals must be
renewed. The others keep a badge, now "Unpublished changes", with a hint.
The bar refreshes once the publish lands.

Smaller changes: the resolve checkbox lines up with the task button and has
a tooltip, messages sit in a filled box, task events take an outline, and
the change navigator sits right above the review bar.

* 💄 Sort the test runner requires that cljfmt rejects

The Frontend Tests lint step fails on runner.cljs: the requires for
workspace-tasks-test, review-details-test and review-markers-test sat
out of alphabetical order. This is the output of cljfmt fix.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 🐛 Test the design review mode in the pull request spec

The review bar replaced the branch banner, so four tests still drove
the branch bar: a visible Compare button, the rectangle tool and the
Review button. They now drive the design review instead.

* Opening a file with a pr-id enters the review: the bar names the
  review, the header trades the file name for the review title, and
  comparing, merging and cancelling sit in the bar's options menu.
* The compare dialog opens from that menu.
* The review is read-only rather than an editable file: the review
  toolbar offers its five tools, the drawing tools are gone, the
  sidebar says View only, and the right panel is the review's own
  Inspect, Comments and Tasks tabs.
* The verdict comes from the bar's Submit review button and the
  dialog it opens, scoped by role so the bar action and the dialog
  action do not collide.

The tests also reach the review bar by its region label instead of
walking up three parents from the Reviewing the design review text,
which broke as soon as the bar's markup changed.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 🐛 Count only the test thread's acquisitions in the climit test

acquires-the-command-own-limits-before-the-root-limits failed on CI
with an extra "root/global" ahead of the expected order. The test
records metrics through with-redefs, which is global, so a call of
another thread of the test JVM, such as an RPC an earlier test left
running, can take a root limiter inside the window and land in the
same atom.

Record the thread with each metrics call and keep, in
acquired-in-order, only the acquisitions of the calling thread. Both
tests that read it acquire their limiters on the test thread. The
same class of flake was fixed on develop for the waiting-count test
(12d73d0f63); this one is still open there.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 📚 Describe the task moves the server allows in the table comment

The comment on file_task said a task is reopened only from done and
that editors move tasks. files_task.clj::transitions also reopens a
verified or won't-do task, and set-task-status lets anybody who can
read the file move one. Say the same in the schema.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

---------

Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-11-17 22:30:22 +01:00

636 lines
27 KiB
Clojure

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.climit-test
(:require
[app.common.logging :as l]
[app.common.uuid :as uuid]
[app.main :as main]
[app.metrics :as mtx]
[app.rpc :as-alias rpc]
[app.rpc.climit :as climit]
[app.util.services :as-alias sv]
[backend-tests.helpers :as th]
[clojure.edn :as edn]
[clojure.java.io :as io]
[clojure.test :as t]
[integrant.core :as ig]))
(def ^:private calls (atom []))
(defn- record-metrics
"Run `f` recording every metrics call made while it runs, with the thread
that made it: `with-redefs` is global, so other threads of the test JVM
record into the same atom."
[f]
(reset! calls [])
(with-redefs [mtx/run!
(fn [_ & {:keys [id labels val]}]
(swap! calls conj {:id id :labels (vec labels) :val val
:thread (Thread/currentThread)}))]
(f)))
(defn- acquired-in-order
"Limiter names the calling thread acquired so far, in acquisition order. A
call of another thread, such as an RPC left running by an earlier test, can
take a root limiter inside the window and must not count."
[]
(let [thread (Thread/currentThread)]
(->> @calls
(filter #(= :rpc-climit-timing (:id %)))
(filter #(identical? thread (:thread %)))
(mapcat :labels)
(vec))))
(defn- invoke!
"Run `f` under the limits the cfg declares, the way a command does: the key
of each limit comes from the params, and the root limiters come after the
ones the cfg declares."
[cfg f params]
(let [wrapped (climit/wrap cfg
f
{::sv/name "test-command"
::climit/id (::climit/id cfg)})]
(wrapped cfg params)))
(defn- failed-type
"Run `f` and return its value, or the type of the error it raised."
[f]
(try
(f)
(catch Throwable cause
(:type (ex-data cause)))))
(def ^:private no-timeout-hint
"limiter without timeout, a saturated one parks its threads forever")
(defn- warned-ids
"Ids of the captured log records that warn about a missing timeout."
[records]
(->> records
(keep (fn [record]
(when (= no-timeout-hint (get (::l/props record) :hint))
(get (::l/props record) :id))))
(set)))
(defn- warned?
"Wait until a captured log record warns about a missing timeout."
[records timeout]
(let [deadline (+ (System/currentTimeMillis) timeout)]
(loop []
(cond
(seq (warned-ids @records)) true
(> (System/currentTimeMillis) deadline) false
:else (do (Thread/sleep 10) (recur))))))
;; One permit and a bounded wait per limiter: a call either gets its permit, or
;; fails with a concurrency limit error, never parks a thread forever.
(def ^:private test-config
{:root/global {:permits 1 :timeout 3000}
:root/by-profile {:permits 1 :timeout 3000}
:process-image/global {:permits 1 :timeout 3000}
:process-image/by-profile {:permits 1 :timeout 3000}})
;; The limits `upload-file-media-object` and `assemble-file-media-object`
;; declare, per profile.
(def ^:private image-limits
[[:process-image/by-profile ::rpc/profile-id]
[:process-image/global]])
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; ACQUISITION ORDER
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest acquires-the-command-own-limits-before-the-root-limits
;; A command declaring its own limits must meet them before the root
;; limiters. No command enters climit from inside its handler any more, so
;; every path takes the limiters in the same order.
(record-metrics
(fn []
(th/with-climit [climit test-config]
(let [mdata {::sv/name "test-command" ::climit/id image-limits}
wrapped (climit/wrap {::rpc/climit climit} (fn [_ _] :ok) mdata)
result (wrapped {::rpc/climit climit} {::rpc/profile-id (uuid/next)})]
(t/is (= :ok result))
(t/is (= ["process-image/by-profile"
"process-image/global"
"root/by-profile"
"root/global"]
(acquired-in-order))))))))
(t/deftest the-key-of-a-limit-comes-from-the-params
;; The key of a limit is a function of the params: a keyword is looked up in
;; them, and that is what tells one profile from another. Two calls with the
;; same key share the limiter, and two calls with a different key do not.
(th/with-climit [climit {:test-key/by-profile {:permits 1 :timeout 2000}}]
(let [pid (uuid/next)
limits [[:test-key/by-profile ::rpc/profile-id]]
cfg {::rpc/climit climit
::climit/id limits}
gate (promise)
entered (promise)
wrapped (climit/wrap cfg
(fn [_ params]
;; only the call of the first profile waits
(when (= pid (::rpc/profile-id params))
(deliver entered true)
@gate)
:ok)
{::sv/name "test-command" ::climit/id limits})
holder (future (wrapped cfg {::rpc/profile-id pid}))]
;; The call of the same profile waits for the permit this thread holds.
(t/is (true? (deref entered 5000 false)))
(t/is (= :timeout (deref (future (wrapped cfg {::rpc/profile-id pid})) 300 :timeout)))
;; The call of another profile takes the permit of its own key.
(t/is (= :ok (deref (future (wrapped cfg {::rpc/profile-id (uuid/next)})) 5000 :timeout)))
(deliver gate true)
(t/is (= :ok (deref holder 5000 :timeout))))))
(t/deftest a-command-without-limits-takes-the-root-limits
;; A command that declares no limits of its own still takes the root ones,
;; and in the same order as the ones of a command that declares them.
(record-metrics
(fn []
(th/with-climit [climit test-config]
(let [mdata {::sv/name "test-command"}
wrapped (climit/wrap {::rpc/climit climit} (fn [_ _] :ok) mdata)
result (wrapped {::rpc/climit climit} {::rpc/profile-id (uuid/next)})]
(t/is (= :ok result))
(t/is (= ["root/by-profile"
"root/global"]
(acquired-in-order))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SHIPPED CONFIG
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest example-config-bounds-every-wait
;; A limiter without `:timeout` parks the calling thread forever, which is
;; how a limit nobody releases turns into lost worker threads and days of
;; silent 429s. Every entry of the shipped config must carry one. A limiter
;; without `:queue` lets its waiters pile up without bound, and a command
;; takes its own limits before the root ones, so the queue of a limit is
;; where its own pile-up waits.
(let [path (io/resource "climit.edn")]
(t/is (some? path))
(let [data (edn/read-string (slurp path))]
(t/is (seq data))
(doseq [[id config] data]
(t/is (int? (:timeout config))
(str id " must declare a timeout"))
(t/is (int? (:queue config))
(str id " must declare a queue"))))))
(t/deftest warns-when-a-limiter-has-no-timeout
;; The config that production loads lives outside the repository and is
;; maintained by hand, so a limiter without a timeout has to say so at
;; startup instead of parking threads in silence months later.
(let [records (atom [])]
(add-watch l/log-record ::capture
(fn [_ _ _ record] (swap! records conj record)))
(try
(th/with-climit [_climit {:process-image/by-profile {:permits 1 :timeout 1000}
:process-image/global {:permits 8}}]
(t/is (true? (warned? records 2000)))
(t/is (= #{:process-image/global} (warned-ids @records))))
(finally
(remove-watch l/log-record ::capture)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; WAITING COUNTER
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- waiting-value-of
"Values published to the waiting gauge for one limiter name, in order."
[name]
(->> @calls
(filter #(= :rpc-climit-waiting (:id %)))
(filter #(= name (first (:labels %))))
(map :val)
(vec)))
(defn- waiting-last
"Last value published to the waiting gauge for one limiter name. How many
publications land inside a window is not part of the contract, the value is,
so this is what the assertions read."
[name]
(last (waiting-value-of name)))
(defn- waiting-count
"Current waiting count of a limiter name, read from its component."
[climit name]
(.get ^java.util.concurrent.atomic.AtomicLong
(#'climit/waiting-counter (:counters climit) name)))
(defn- wait-until
"Wait until `pred` holds, or give up."
[pred timeout]
(let [deadline (+ (System/currentTimeMillis) timeout)]
(loop []
(cond
(pred) true
(> (System/currentTimeMillis) deadline) false
:else (do (Thread/sleep 10) (recur))))))
(defn- limiter-name
"The name climit publishes for a limiter id: the keyword without the colon."
[id]
(subs (str id) 1))
(defn- limiter-cfg
[climit name key]
{::rpc/climit climit
::climit/id [[name key]]})
(t/deftest waiting-gauge-follows-the-queue
;; The value is the number of calls waiting for a permit: up when a call
;; starts waiting, down when the wait ends. The gauges that sampled the
;; limiter stats could not show this, because the value of whichever limiter
;; of the name ran last was the one that survived.
(record-metrics
(fn []
(th/with-climit [climit {:test-waiting/by-profile {:permits 1 :timeout 3000}}]
(let [name (limiter-name :test-waiting/by-profile)
entered (promise)
gate (promise)
holder (future (invoke! (limiter-cfg climit :test-waiting/by-profile "a")
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
;; the holder is running, not waiting
(reset! calls [])
(let [waiter (future (invoke! (limiter-cfg climit :test-waiting/by-profile "a")
(fn [_ _] :waited)
{}))]
(t/is (true? (wait-until #(= 1 (waiting-last name)) 5000)))
(t/is (= 1 (waiting-count climit name)))
(deliver gate true)
(t/is (= :waited (deref waiter 5000 :timeout)))
(t/is (= :held (deref holder 5000 :timeout)))
(t/is (true? (wait-until #(= 0 (waiting-last name)) 5000)))
(t/is (= 0 (waiting-count climit name)))))))))
(t/deftest waiting-gauge-sums-every-key-of-a-name
;; Two profiles queueing on the same limiter name add up. That is the shape
;; that stayed invisible in production: one series per name, counting the
;; entries and the exits, instead of one value per limiter overwritten by the
;; next one.
(record-metrics
(fn []
(th/with-climit [climit {:test-waiting-keys/by-profile {:permits 1 :timeout 3000}}]
(let [name (limiter-name :test-waiting-keys/by-profile)
gates {"a" (promise) "b" (promise)}
holders
(into {}
(map (fn [k]
(let [entered (promise)]
[k {:entered entered
:future (future (invoke! (limiter-cfg climit :test-waiting-keys/by-profile k)
(fn [_ _]
(deliver entered true)
@(gates k)
:held)
{}))}])))
["a" "b"])]
(doseq [k ["a" "b"]]
(t/is (true? (deref (:entered (holders k)) 5000 false))))
;; one call per key, waiting behind its own holder
(let [waiters (into {}
(map (fn [k]
[k (future (invoke! (limiter-cfg climit :test-waiting-keys/by-profile k)
(fn [_ _] :waited)
{}))]))
["a" "b"])]
(t/is (true? (wait-until #(= 2 (waiting-count climit name)) 5000)))
;; the increment that reaches 2 always publishes 2, whichever waiter
;; does it last: the order of two concurrent publications is not the
;; contract, the value is.
(t/is (some #(= 2 %) (waiting-value-of name)))
(doseq [k ["a" "b"]] (deliver (gates k) true))
(doseq [k ["a" "b"]]
(t/is (= :waited (deref (waiters k) 5000 :timeout)))
(t/is (= :held (deref (:future (holders k)) 5000 :timeout))))
(t/is (true? (wait-until #(= 0 (waiting-count climit name)) 5000)))
(t/is (some #(= 0 %) (waiting-value-of name)))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; REJECTIONS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- rejections
"Rejections published, as [name reason] pairs."
[]
(->> @calls
(filter #(= :rpc-climit-rejected (:id %)))
(map (fn [call] [(first (:labels call)) (second (:labels call))]))
(vec)))
(t/deftest rejection-counter-tells-why
;; Rejections had no metric: the panel counted them from the logs. Now a call
;; rejected because the queue is full and a call rejected because it waited
;; too long are counted apart.
(record-metrics
(fn []
(th/with-climit [climit {:test-rejected/by-profile {:permits 1 :queue 1 :timeout 60000}}]
(let [name (limiter-name :test-rejected/by-profile)
cfg (limiter-cfg climit :test-rejected/by-profile "a")
entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(reset! calls [])
;; the queue holds one waiter, so the next call is rejected right away
(let [queued (future (failed-type #(invoke! cfg (fn [_ _] :queued) {})))]
(t/is (true? (wait-until #(= 1 (waiting-last name)) 5000)))
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :extra) {}))))
(t/is (= [[name "queue-full"]] (rejections)))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout)))
(t/is (= :queued (deref queued 5000 :timeout))))))
;; a wait that times out is counted as such, not as a full queue
(th/with-climit [climit {:test-timeout/by-profile {:permits 1 :timeout 300}}]
(let [name (limiter-name :test-timeout/by-profile)
cfg (limiter-cfg climit :test-timeout/by-profile "a")
entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(reset! calls [])
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :late) {}))))
(t/is (= [[name "timeout"]] (rejections)))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout))))))))
(t/deftest the-waiting-count-comes-back-after-a-rejection
;; A rejected call increments and decrements exactly once: a limiter that
;; rejects under pressure must not drift the panel up.
(record-metrics
(fn []
(th/with-climit [climit {:test-rejected-baseline/by-profile {:permits 1 :queue 0 :timeout 3000}}]
(let [name (limiter-name :test-rejected-baseline/by-profile)
cfg (limiter-cfg climit :test-rejected-baseline/by-profile "a")
entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(reset! calls [])
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :extra) {}))))
(t/is (= 0 (waiting-count climit name)))
(t/is (true? (wait-until #(= 0 (waiting-last name)) 5000)))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout))))))))
(t/deftest the-waiting-count-comes-back-when-the-handler-throws
;; A handler that throws still leaves the count where it was.
(record-metrics
(fn []
(th/with-climit [climit {:test-handler-throws/by-profile {:permits 1 :timeout 3000}}]
(let [name (limiter-name :test-handler-throws/by-profile)
cfg (limiter-cfg climit :test-handler-throws/by-profile "a")]
(reset! calls [])
(t/is (thrown? clojure.lang.ExceptionInfo
(invoke! cfg (fn [_ _] (throw (ex-info "boom" {}))) {})))
(t/is (= 0 (waiting-count climit name)))
(t/is (true? (wait-until #(= 0 (waiting-last name)) 5000))))))))
(defn- failing-first-waiting-publish
"A `mtx/run!` replacement that fails the first waiting publication and records
the rest, so the decrement of the unwinding path still lands."
[calls]
(let [pending (atom true)]
(fn [_ & {:keys [id labels val]}]
(if (and @pending (= :rpc-climit-waiting id))
(do (reset! pending false)
(throw (ex-info "publishing is down" {})))
(swap! calls conj {:id id :labels (vec labels) :val val})))))
(t/deftest a-throwable-while-waiting-leaves-no-mark-and-no-count
;; `waiting-inc` sits inside the `try`: a throwable from the publishing path
;; unwinds through the `finally`, so the thread neither keeps the limiter
;; marked as held (it would skip that limiter for that key for ever) nor
;; leaves the count up.
(th/with-climit [climit {:test-throw-waiting/by-profile {:permits 1 :timeout 3000}}]
(let [name (limiter-name :test-throw-waiting/by-profile)
cfg (limiter-cfg climit :test-throw-waiting/by-profile "a")]
(reset! calls [])
(with-redefs [mtx/run! (failing-first-waiting-publish calls)]
(t/is (thrown? clojure.lang.ExceptionInfo
(invoke! cfg (fn [_ _] :never) {}))))
;; the count came back down instead of staying up for good. Only the
;; last publication is asserted, not the exact list: the sampler
;; republishes the same count when its first tick lands inside the
;; window.
(t/is (= 0 (waiting-count climit name)))
(t/is (= 0 (waiting-last name)))
;; and the thread dropped the mark: with it left behind, this call would
;; run inline instead of waiting for the permit another thread holds.
(let [entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :inline) {}))))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout)))))))
(defn- gauge-value
"Current value of the waiting gauge for one limiter name, read from a real
prometheus collector."
[metrics name]
(let [collector (:app.metrics.definition/instance
(mtx/get-collector metrics :rpc-climit-waiting))]
(.get ^io.prometheus.client.Gauge$Child
(.labels ^io.prometheus.client.Gauge collector (into-array String [name])))))
(t/deftest the-waiting-gauge-reaches-a-real-registry
;; The recorded calls prove the arguments, not that a collector with that name
;; and those labels exists: a typo in the metric id or a wrong label arity
;; would only show up in production. Here the value is read back from a real
;; prometheus registry.
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
name (limiter-name :test-registry/by-profile)]
(th/with-climit [climit {:test-registry/by-profile {:permits 1 :timeout 3000}} metrics]
(let [cfg (limiter-cfg climit :test-registry/by-profile "a")
entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(let [waiter (future (invoke! cfg (fn [_ _] :waited) {}))]
(t/is (true? (wait-until #(= 1.0 (gauge-value metrics name)) 5000)))
(deliver gate true)
(t/is (= :waited (deref waiter 5000 :timeout)))
(t/is (= :held (deref holder 5000 :timeout)))
(t/is (true? (wait-until #(= 0.0 (gauge-value metrics name)) 5000))))))))
(defn- rejected-record
"The first captured log record that reports a rejection with `reason`."
[records reason]
(let [deadline (+ (System/currentTimeMillis) 5000)]
(loop []
(let [record (first (filter (fn [record]
(let [props (::l/props record)]
(and (= "rejected" (get props :hint))
(= reason (get props :reason)))))
@records))]
(cond
record record
(> (System/currentTimeMillis) deadline) nil
:else (do (Thread/sleep 10) (recur)))))))
(t/deftest rejection-logs-say-why-and-how-big-the-limits-are
;; The line has to say whether the queue was full or the wait timed out, and
;; how many permits and how much queue the limiter has, or nobody can tell
;; whether the limiter did its job.
(let [records (atom [])]
(add-watch l/log-record ::capture
(fn [_ _ _ record] (swap! records conj record)))
(try
(record-metrics
(fn []
(th/with-climit [climit {:test-log/by-profile {:permits 1 :queue 2 :timeout 60000}}]
(let [name (limiter-name :test-log/by-profile)
cfg (limiter-cfg climit :test-log/by-profile "a")
entered (promise)
gate (promise)
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(reset! calls [])
(let [q1 (future (invoke! cfg (fn [_ _] :q1) {}))
q2 (future (invoke! cfg (fn [_ _] :q2) {}))]
;; the two of them fill the queue, so the next one is rejected
(t/is (true? (wait-until #(= 2 (waiting-count climit name)) 5000)))
(t/is (some #(= 2 %) (waiting-value-of name)))
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :x) {}))))
(let [props (::l/props (rejected-record records "queue-full"))]
(t/is (= 1 (get props :permits)))
(t/is (= 2 (get props :queue-max)))
(t/is (= :capacity-limit-reached (get props :code)))
(t/is (= "queue-full" (get props :reason))))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout)))
(deref q1 5000 nil)
(deref q2 5000 nil))))
;; A wait that times out carries its own reason in the line, next to the
;; code the limiter raised. The line is emitted at TRACE when nothing
;; else is left queued behind it (the caller was the only one waiting),
;; so the call to `log` is captured instead of the record, which the
;; logger would drop.
(th/with-climit [climit {:test-log-timeout/by-profile {:permits 1 :timeout 300}}]
(let [cfg (limiter-cfg climit :test-log-timeout/by-profile "a")
lines (atom [])
entered (promise)
gate (promise)
capture (fn [_ action _req-id _stats _elapsed code reason]
(swap! lines conj [action code reason]))
holder (future (invoke! cfg
(fn [_ _] (deliver entered true) @gate :held)
{}))]
(t/is (true? (deref entered 5000 false)))
(with-redefs [climit/log capture]
(t/is (= :concurrency-limit (failed-type #(invoke! cfg (fn [_ _] :late) {}))))
(t/is (= [["rejected" :capacity-limit-reached "timeout"]]
(filterv #(= "rejected" (first %)) @lines))))
(deliver gate true)
(t/is (= :held (deref holder 5000 :timeout)))))))
(finally
(remove-watch l/log-record ::capture)))))
(t/deftest republish-writes-the-current-waiting-count
;; A call that never comes back keeps the count up, but if it is also the
;; last thing that happened nothing rewrites that series and it goes stale in
;; the scrape. The republish is what keeps it visible. The counters are built
;; on their own: with a component, its sampler would also publish the same
;; count the moment its first tick lands inside the window.
(let [counters (#'climit/create-waiting-counters)
name (limiter-name :test-republish/by-profile)
counter ^java.util.concurrent.atomic.AtomicLong (#'climit/waiting-counter counters name)]
(.set counter 3)
(record-metrics
(fn []
(#'climit/republish-waiting (th/metrics-stub) counters)
(t/is (= [3] (waiting-value-of name)))))))
(t/deftest halting-the-component-stops-the-republisher
;; The republisher is a daemon thread started with the component: halting it
;; has to stop the thread, or every restart leaves another one running.
(let [climit (th/init-climit {:test-halt/by-profile {:permits 1 :timeout 3000}})
sampler ^java.util.concurrent.ScheduledThreadPoolExecutor (:sampler climit)]
(t/is (false? (.isShutdown sampler)))
(ig/halt-key! ::rpc/climit climit)
(t/is (true? (.isShutdown sampler)))))
(t/deftest the-climit-metrics-that-remain-are-defined
;; A typo in a metric id, or a definition that is not there, makes the
;; publishing silent, which is the failure mode this work removes. And the
;; gauges that could not tell the truth must be gone.
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})]
(t/is (= ["name"]
(:app.metrics.definition/labels (mtx/get-collector metrics :rpc-climit-waiting))))
(t/is (= ["name" "reason"]
(:app.metrics.definition/labels (mtx/get-collector metrics :rpc-climit-rejected))))
(t/is (some? (mtx/get-collector metrics :rpc-climit-timing)))
(t/is (nil? (mtx/get-collector metrics :rpc-climit-queue)))
(t/is (nil? (mtx/get-collector metrics :rpc-climit-permits)))
(t/is (nil? (mtx/get-collector metrics :http-worker-busy-threads)))))