2 Commits

Author SHA1 Message Date
Andrey Antukh
3cc91e8ba2
♻️ Take the command limits before the root ones in the RPC limiter (#12686)
* ♻️ Take the command limits before the root ones in the RPC limiter

The RPC concurrency limiter took the root limiters (the ones every request
passes through) before the ones a command declares, so a request waiting for
the limits of its own command held the root permits of its profile while it
waited: a profile with a pile-up of heavy requests saw its other requests
queue behind them and be rejected with 429.

That order was forced by two commands that entered the limiter from inside
their handler: a command that declares no limits of its own can only take
them deeper than the root ones, so two paths of one profile could take the
same limiters in opposite order and park for good.

The commands declare their limits in their metadata now and their handlers
call the work directly, so no command enters the limiter from inside its
handler and every path takes the limiters in the same order. The root
limiters are appended to the sequence first, so the limits a command
declares end up outermost and are taken first: a request waiting for them
does not hold the root permits of its profile. The photo of a team keeps its
limits, because it declares them now that the helper does not.

A call from inside a handler can only reenter the limits that thread
already holds, and it is refused when it asks for a new one, because that
would take it after the root ones and invert the order against the other
paths. The key of a limit is resolved the same way in every path, so both
paths point at the same limiter and the reentrancy check sees it.

Every entry of the shipped climit.edn declares a queue, so the number of
requests waiting for the same limit is bounded instead of piling up until
the timeout. The production file lives outside the repository and has to be
mirrored to pick them up.

The tests cover the acquisition order of a command with and without limits
of its own, the order of the limits as a whole (a ring of any size fails,
not only a pair in opposite order), the nested call that asks for a limit
the thread does not hold, and the queues and timeouts of the shipped
configuration. The import of a binfile declares its request as multipart in
its test, the way the commands that take an upload map do.

AI-assisted-by: minimax-m3.1-flash
AI-assisted-by: deepseek-v4.1-flash

* 🔧 Keep the commands' limits and the shipped config in sync

`profile-plugin-ops` guarded only an UPDATE on the caller's own profile row,
which the row lock already serializes, so its global cap could only reject a
burst of different profiles toggling plugins. Both commands declare the
by-profile limit alone now, and the example config drops the entry.

A declared id without an entry does not limit anything: `wrap` logs a warning
and runs the handler with no limiter at all, which is how a command ends up
unbounded in production. A test fails now when a command declares an id the
config does not list, and the header of the example says the production file
mirrors the shape and lists every declared id, not the values.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Record that background work is not waited for with a sleep

A long command moved to the background reports its own completion, so
polling it with a sleep inside a foreground command only blocks the session
and the notification arrives anyway. The rule says to keep working on
something else, or to end the response and wait for it.

AI-assisted-by: minimax-m3.1-flash
2026-10-28 18:44:36 +01:00
Andrey Antukh
f242ff9574 🐛 Keep concurrency limits from deadlocking on nested calls
An instance left the audit events pipeline dead: every push-audit-events
answered with 429 and 69 worker threads parked for good inside a limiter.

Two code paths of one profile acquired the same two limiters in opposite
order: a command declaring its own limits (upload-file-media-object) took
process-image/by-profile and then root/by-profile, while a command with no
limits of its own took the roots first and then re-entered climit from
inside its handler with the same ids (create-file-media-object-from-url).
Each path held what the other one waited for, and no limiter had a timeout
to break the wait.

Now wrap puts the root limiters after the command's own, so they end up
outermost and every path meets them first: the nested chains carry only the
command's own limits and run inside them, which puts both paths in the same
order. impl-invoke also keeps, per thread, the set of limiters that thread
is inside of and runs the handler inline when a call asks for one it
already holds, so a handler can no longer queue behind itself.

The shipped climit.edn sets a timeout on every limiter (60s for operations
that can be slow, 30s otherwise), so no wait can park a thread for ever.
The production file lives outside the repository and has to be mirrored.

Tests cover the acquisition order, the deadlock, the reentrancy and the
timeouts of the shipped config.

AI-assisted-by: deepseek-v4.1-flash
2026-10-13 14:33:32 +02:00