Daoyuan Li
44990ff194
fix(mcp): use threading.Lock for OAuth token refresh to avoid cross-thread deadlock ( #4240 ) ...
* fix(mcp): use threading.Lock for OAuth token refresh to avoid cross-thread deadlock
OAuthTokenManager created one asyncio.Lock per server for the process
lifetime. The embedded/TUI sync tool-call path (DeerFlowClient.stream()
-> LangGraph's ToolNode._func -> a ThreadPoolExecutor ->
make_sync_tool_wrapper's per-call asyncio.run()) invokes
get_authorization_header from a fresh event loop on a fresh OS thread
for every concurrent tool call. asyncio.Lock binds to whichever loop
first contends on it; when a caller on a different loop later releases
or wakes a waiter, it does so without call_soon_threadsafe, so the
waiting loop's selector is never woken and that caller hangs forever
with no exception. A third concurrent caller instead raises a
synchronous RuntimeError ("bound to a different event loop"). Either
way, two concurrent OAuth-protected tool calls (including the very
first cold-start token fetch) can freeze the entire agent turn.
Gateway's async path (ToolNode._afunc) is unaffected.
Replace the asyncio.Lock with a plain threading.Lock, acquired via
asyncio.to_thread so the blocking wait never blocks the event loop,
and released synchronously in a finally block. This keeps the
single-fetch de-duplication the lock provided while making it safe
across however many event loops/threads call into the same server's
lock.
Adds a regression test that runs three threads, each with its own
event loop, calling get_authorization_header concurrently for the same
server, and asserts (with a bounded join timeout so a regression fails
fast instead of hanging the suite) that none hang or raise, and that
only one real token fetch happens.
* fix(mcp): make OAuth lock acquisition cancellation-safe
get_authorization_header acquired the per-server threading.Lock via a
bare `await asyncio.to_thread(lock.acquire)`, with the try/finally that
guarantees release only starting after that await returned. Once the
executor thread had actually started running lock.acquire(), cancelling
the awaiting caller only stopped the caller -- Python cannot interrupt a
running OS thread. CancelledError was still delivered to the caller
immediately, but the thread kept blocking until the current holder
released, then silently acquired the lock with nobody left to call
release() for it. The lock stayed locked forever and every later OAuth
token refresh for that server blocked permanently at the same line --
the exact cross-thread deadlock this lock was introduced to prevent,
reintroduced via a different path under cancellation (e.g. a caller
wrapped in asyncio.wait_for/asyncio.timeout, or task-group cancellation).
Run the acquisition as an explicit asyncio.create_task, awaited via
asyncio.shield() so cancelling the caller no longer cancels the
underlying acquisition task. If the caller is cancelled, keep
(re-)waiting on the still-shielded acquisition task -- tolerating
further cancellation during this cleanup by simply retrying -- until it
actually finishes, release the lock immediately, and only then
re-raise. This guarantees the lock is released regardless of when or
how many times the caller is cancelled: before the acquisition is even
scheduled, while queued, or after it has already been silently granted.
Adds a regression test that holds the per-server lock, starts a second
caller that has to wait for it, cancels that caller while it is
genuinely blocked in its executor thread, releases the original holder,
and asserts a third caller completes within a bounded asyncio.wait_for
and still performs exactly one token fetch. Every potentially-hanging
await is bounded so a regression fails the test quickly instead of
hanging the suite.
2026-07-22 19:58:43 +08:00
2026-07-21 09:09:40 +08:00
2026-07-22 14:02:56 +08:00
2026-07-19 18:26:26 +08:00
2026-07-22 13:55:40 +08:00
2026-05-12 23:18:54 +08:00
2026-06-30 18:58:02 +08:00
2026-04-26 15:09:25 +08:00
2026-06-01 15:50:39 +08:00
2026-07-22 08:33:29 +08:00
2026-07-21 11:46:33 +08:00
2026-06-21 20:56:42 +08:00
2026-06-08 12:35:03 +08:00
2026-07-22 14:47:08 +08:00
2026-07-17 15:11:15 +08:00
2026-07-22 13:44:55 +08:00
2026-07-22 13:44:55 +08:00
2026-07-21 09:09:40 +08:00
2026-07-21 09:09:40 +08:00
2026-05-21 14:44:34 +08:00
2026-07-03 21:52:31 +08:00
2026-06-23 10:39:40 +08:00
2026-07-22 08:33:29 +08:00
2026-07-19 07:31:29 +08:00
2026-05-29 23:05:59 +08:00
2026-05-16 09:24:40 +08:00
2026-04-26 11:08:11 +08:00
2026-06-16 23:20:20 +08:00
2026-07-18 22:17:15 +08:00
2026-06-21 15:47:53 +08:00
2026-07-17 14:49:51 +08:00
2026-07-17 14:49:51 +08:00
2026-07-21 09:23:14 +08:00
2026-06-24 13:57:12 +08:00
2026-07-09 17:21:25 +08:00
2026-07-09 17:21:25 +08:00
2026-07-02 11:00:49 +08:00
2026-07-21 11:46:33 +08:00
2026-07-21 11:46:33 +08:00
2026-07-19 20:08:52 +08:00
2026-07-14 07:37:59 +08:00
2026-06-16 23:04:39 +08:00
2026-06-18 10:15:31 +08:00
2026-07-12 08:54:03 +08:00
2026-05-02 15:19:28 +08:00
2026-07-17 14:49:51 +08:00
2026-06-18 10:45:33 +08:00
2026-07-22 14:52:42 +08:00
2026-04-14 10:29:44 +08:00
2026-07-22 08:33:29 +08:00
2026-07-22 08:33:29 +08:00
2026-07-12 08:15:32 +08:00
2026-07-22 08:33:29 +08:00
2026-07-06 22:34:41 +08:00
2026-03-30 07:41:18 +08:00
2026-04-25 19:40:06 +08:00
2026-04-07 18:21:22 +08:00
2026-07-22 14:02:56 +08:00
2026-07-22 08:33:29 +08:00
2026-03-29 21:03:58 +08:00
2026-05-04 09:56:16 +08:00
2026-07-22 14:40:57 +08:00
2026-05-02 15:04:11 +08:00
2026-06-10 21:36:25 +08:00
2026-07-22 08:33:29 +08:00
2026-07-05 09:40:16 +08:00
2026-07-22 08:33:29 +08:00
2026-04-26 11:05:47 +08:00
2026-07-05 09:54:55 +08:00
2026-03-29 15:31:18 +08:00
2026-07-22 08:33:29 +08:00
2026-05-07 16:15:15 +08:00
2026-06-12 15:24:58 +08:00
2026-07-22 07:46:03 +08:00
2026-07-17 19:16:42 +08:00
2026-06-08 07:59:50 +08:00
2026-07-22 14:40:57 +08:00
2026-07-13 17:57:34 +08:00
2026-06-02 22:43:22 +08:00
2026-06-05 15:21:41 +08:00
2026-07-12 00:17:11 +08:00
2026-07-04 23:09:29 +08:00
2026-06-02 22:43:22 +08:00
2026-07-01 22:49:17 +08:00
2026-07-13 19:26:07 +08:00
2026-07-22 08:33:29 +08:00
2026-07-22 08:33:29 +08:00
2026-07-04 09:21:19 +08:00
2026-07-22 13:55:40 +08:00
2026-07-04 21:31:01 +08:00
2026-07-21 11:46:33 +08:00
2026-06-12 17:16:01 +08:00
2026-06-09 15:29:40 +08:00
2026-07-20 23:00:44 +08:00
2026-06-12 15:24:58 +08:00
2026-06-13 23:27:17 +08:00
2026-03-31 22:19:27 +08:00
2026-07-02 11:00:49 +08:00
2026-07-13 19:26:07 +08:00
2026-07-16 09:39:09 +08:00
2026-07-22 14:32:54 +08:00
2026-04-26 11:09:56 +08:00
2026-04-08 17:13:39 +08:00
2026-07-05 09:54:55 +08:00
2026-07-21 11:46:33 +08:00
2026-07-14 10:43:13 +08:00
2026-07-17 11:18:20 +08:00
2026-07-02 08:24:15 +08:00
2026-07-04 21:31:01 +08:00
2026-07-19 22:12:30 +08:00
2026-07-14 11:57:48 +08:00
2026-07-22 08:33:29 +08:00
2026-07-22 08:33:29 +08:00
2026-05-11 17:38:37 +08:00
2026-07-04 11:27:19 +08:00
2026-07-22 14:40:57 +08:00
2026-07-22 08:33:29 +08:00
2026-07-22 08:33:29 +08:00
2026-07-18 17:50:20 +08:00
2026-07-22 08:41:50 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-19 22:08:48 +08:00
2026-07-04 22:56:24 +08:00
2026-07-14 22:56:50 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-16 09:12:29 +08:00
2026-07-20 08:01:18 +08:00
2026-07-15 22:30:34 +08:00
2026-07-22 08:33:29 +08:00
2026-06-21 15:55:10 +08:00
2026-07-17 14:49:51 +08:00
2026-03-14 22:55:52 +08:00
2026-07-06 07:21:12 +08:00
2026-07-14 10:43:13 +08:00
2026-07-06 22:34:41 +08:00
2026-07-10 11:08:30 +08:00
2026-03-26 14:20:18 +08:00
2026-06-28 16:10:44 +08:00
2026-07-08 17:10:27 +08:00
2026-07-22 14:02:56 +08:00
2026-07-04 13:54:04 +08:00
2026-06-17 15:29:22 +08:00
2026-07-22 14:47:08 +08:00
2026-06-08 23:25:29 +08:00
2026-07-12 11:51:31 +08:00
2026-06-10 16:11:00 +08:00
2026-07-22 13:44:55 +08:00
2026-07-17 23:22:16 +08:00
2026-07-21 11:46:33 +08:00
2026-07-22 14:02:56 +08:00
2026-07-21 08:07:49 +08:00
2026-04-11 16:52:10 +08:00
2026-07-20 23:33:09 +08:00
2026-07-09 17:21:25 +08:00
2026-07-03 07:51:22 +08:00
2026-07-13 00:01:22 +08:00
2026-07-04 13:54:04 +08:00
2026-07-13 00:01:22 +08:00
2026-07-04 13:54:04 +08:00
2026-07-03 08:01:46 +08:00
2026-04-30 22:27:14 +08:00
2026-05-07 16:15:15 +08:00
2026-07-19 17:13:44 +08:00
2026-07-16 08:19:52 +08:00
2026-07-19 22:12:30 +08:00
2026-07-03 15:39:12 +08:00
2026-07-09 16:26:31 +08:00
2026-04-25 09:18:13 +08:00
2026-07-04 13:54:04 +08:00
2026-07-22 19:58:43 +08:00
2026-07-10 07:54:36 +08:00
2026-07-09 16:26:31 +08:00
2026-07-09 16:26:31 +08:00
2026-07-14 09:49:43 +08:00
2026-07-04 09:40:05 +08:00
2026-06-25 22:57:00 +08:00
2026-07-14 09:49:43 +08:00
2026-07-21 09:36:09 +08:00
2026-07-22 14:40:57 +08:00
2026-07-22 14:40:57 +08:00
2026-07-22 07:46:03 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 11:21:04 +08:00
2026-07-15 20:25:41 +08:00
2026-07-22 14:40:57 +08:00
2026-07-15 11:21:04 +08:00
2026-07-21 09:36:09 +08:00
2026-07-22 07:46:03 +08:00
2026-07-22 07:46:03 +08:00
2026-07-22 07:46:03 +08:00
2026-04-26 11:09:55 +08:00
2026-07-22 14:40:57 +08:00
2026-07-15 11:21:04 +08:00
2026-07-22 14:40:57 +08:00
2026-07-22 14:02:56 +08:00
2026-07-20 07:24:34 +08:00
2026-07-16 08:19:52 +08:00
2026-07-04 13:54:04 +08:00
2026-07-17 14:23:44 +08:00
2026-03-22 20:39:26 +08:00
2026-07-22 13:44:55 +08:00
2026-07-14 08:58:06 +08:00
2026-07-21 11:46:33 +08:00
2026-07-16 08:19:52 +08:00
2026-06-22 18:55:20 +08:00
2026-07-18 17:48:42 +08:00
2026-06-21 15:47:53 +08:00
2026-05-28 08:20:52 +08:00
2026-07-11 16:05:30 +08:00
2026-07-04 21:44:22 +08:00
2026-05-28 18:24:32 +08:00
2026-06-08 22:04:38 +08:00
2026-03-26 15:07:05 +08:00
2026-06-09 18:01:43 +08:00
2026-06-12 15:24:58 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-06-24 13:57:12 +08:00
2026-06-24 13:57:12 +08:00
2026-07-16 08:19:52 +08:00
2026-07-21 08:52:31 +08:00
2026-06-24 13:57:12 +08:00
2026-07-19 17:01:28 +08:00
2026-05-21 16:22:09 +08:00
2026-04-26 11:13:01 +08:00
2026-07-20 07:24:34 +08:00
2026-04-10 20:40:30 +08:00
2026-07-09 23:25:38 +08:00
2026-07-14 07:51:59 +08:00
2026-07-21 09:23:14 +08:00
2026-07-06 20:57:41 +08:00
2026-07-11 23:20:53 +08:00
2026-06-17 21:11:44 +08:00
2026-03-14 22:55:52 +08:00
2026-03-14 22:55:52 +08:00
2026-07-03 08:01:46 +08:00
2026-07-14 07:51:59 +08:00
2026-06-08 17:32:41 +08:00
2026-06-09 21:58:31 +08:00
2026-07-14 23:08:33 +08:00
2026-07-11 15:58:07 +08:00
2026-07-14 21:37:47 +08:00
2026-06-24 17:47:04 +08:00
2026-07-02 07:43:09 +08:00
2026-04-26 15:09:25 +08:00
2026-07-12 13:03:02 +08:00
2026-07-16 09:39:09 +08:00
2026-07-16 09:39:09 +08:00
2026-07-16 08:19:52 +08:00
2026-05-21 14:48:28 +08:00
2026-07-14 07:37:59 +08:00
2026-07-22 08:33:29 +08:00
2026-07-20 07:54:15 +08:00
2026-06-24 10:24:36 +08:00
2026-07-22 08:33:29 +08:00
2026-07-19 22:12:30 +08:00
2026-05-22 21:20:28 +08:00
2026-05-22 21:20:28 +08:00
2026-05-22 21:20:28 +08:00
2026-06-25 22:43:28 +08:00
2026-06-03 22:02:27 +08:00
2026-07-14 23:02:26 +08:00
2026-04-09 17:21:23 +08:00
2026-07-21 09:09:40 +08:00
2026-07-21 09:09:40 +08:00
2026-07-13 18:34:41 +08:00
2026-06-25 08:06:17 +08:00
2026-07-12 11:32:41 +08:00
2026-07-12 23:42:58 +08:00
2026-07-04 13:54:04 +08:00
2026-06-12 10:20:38 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-06-12 09:45:26 +08:00
2026-07-21 23:41:07 +08:00
2026-06-19 11:23:07 +08:00
2026-03-26 14:20:18 +08:00
2026-07-21 08:15:33 +08:00
2026-06-27 23:30:54 +08:00
2026-07-07 23:09:06 +08:00
2026-06-21 16:08:25 +08:00
2026-05-12 23:18:54 +08:00
2026-06-14 10:40:16 +08:00
2026-07-05 19:55:39 +08:00
2026-06-19 21:57:03 +08:00
2026-07-04 23:09:29 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 23:09:29 +08:00
2026-07-20 08:18:23 +08:00
2026-07-13 10:40:22 +08:00
2026-05-28 15:48:32 +08:00
2026-07-17 14:39:35 +08:00
2026-07-16 14:12:02 +08:00
2026-07-11 15:58:07 +08:00
2026-07-11 15:58:07 +08:00
2026-07-04 13:54:04 +08:00
2026-07-16 14:12:02 +08:00
2026-03-25 16:28:33 +08:00
2026-07-16 14:12:02 +08:00
2026-07-07 21:44:28 +08:00
2026-07-19 22:36:15 +08:00
2026-07-16 14:12:02 +08:00
2026-07-04 23:09:29 +08:00
2026-07-17 23:22:16 +08:00
2026-07-21 09:39:44 +08:00
2026-05-07 08:34:43 +08:00
2026-07-21 00:02:16 +08:00
2026-06-18 10:09:46 +08:00
2026-07-08 21:58:33 +08:00
2026-07-16 14:12:02 +08:00
2026-07-13 19:33:40 +08:00
2026-03-30 16:02:23 +08:00
2026-06-12 15:24:58 +08:00
2026-07-22 09:20:54 +08:00
2026-07-11 22:46:10 +08:00
2026-06-14 10:30:45 +08:00
2026-06-08 23:17:22 +08:00
2026-07-14 08:14:21 +08:00
2026-07-22 14:59:33 +08:00
2026-07-13 19:26:07 +08:00
2026-07-14 22:51:21 +08:00
2026-05-02 06:37:49 +08:00
2026-07-11 15:41:57 +08:00
2026-07-11 15:41:57 +08:00
2026-07-13 19:26:07 +08:00
2026-06-19 21:42:42 +08:00
2026-07-08 17:10:27 +08:00
2026-07-15 11:21:04 +08:00
2026-07-07 19:55:33 +08:00
2026-07-22 08:33:29 +08:00
2026-06-24 07:28:17 +08:00
2026-07-22 14:02:56 +08:00
2026-05-21 21:18:10 +08:00
2026-06-18 10:15:31 +08:00
2026-07-11 15:53:14 +08:00
2026-07-10 22:13:50 +08:00
2026-06-21 21:43:43 +08:00
2026-07-14 10:43:13 +08:00
2026-06-12 15:24:58 +08:00
2026-07-22 08:33:29 +08:00
2026-07-04 09:21:19 +08:00
2026-07-18 22:12:57 +08:00
2026-06-02 22:43:22 +08:00
2026-07-22 08:33:29 +08:00
2026-05-22 21:42:14 +08:00
2026-07-22 08:33:29 +08:00
2026-07-22 08:33:29 +08:00
2026-07-08 14:41:16 +08:00
2026-07-15 11:21:04 +08:00
2026-03-14 22:55:52 +08:00
2026-07-11 23:15:03 +08:00
2026-06-12 22:48:47 +08:00
2026-07-08 22:26:06 +08:00
2026-07-04 23:14:46 +08:00
2026-05-10 22:00:57 +08:00
2026-06-23 23:37:41 +08:00
2026-03-26 14:20:18 +08:00
2026-05-10 23:09:03 +08:00
2026-05-19 22:11:46 +08:00
2026-07-22 14:02:56 +08:00
2026-07-04 11:27:19 +08:00
2026-07-16 16:41:04 +08:00
2026-04-06 15:09:57 +08:00
2026-07-06 20:57:41 +08:00
2026-07-06 20:57:41 +08:00
2026-07-12 11:24:24 +08:00
2026-07-14 09:49:43 +08:00
2026-07-19 09:13:51 +08:00
2026-07-19 09:13:51 +08:00
2026-07-14 08:58:06 +08:00
2026-07-14 08:58:06 +08:00
2026-07-03 08:01:46 +08:00
2026-07-22 07:53:50 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-22 07:53:50 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-15 22:27:00 +08:00
2026-06-25 20:10:49 +08:00
2026-07-20 23:56:37 +08:00
2026-05-12 23:18:54 +08:00
2026-07-22 13:44:55 +08:00
2026-07-01 16:43:56 +08:00
2026-07-22 14:02:56 +08:00
2026-07-19 17:24:31 +08:00
2026-04-26 15:09:25 +08:00
2026-07-16 14:12:02 +08:00
2026-07-11 09:18:12 +08:00
2026-07-14 10:43:13 +08:00
2026-05-02 15:16:16 +08:00
2026-06-09 15:29:40 +08:00
2026-07-22 08:41:50 +08:00
2026-07-14 23:02:26 +08:00
2026-04-06 15:18:34 +08:00
2026-07-04 09:21:19 +08:00
2026-07-06 07:21:12 +08:00
2026-07-20 23:05:47 +08:00
2026-07-18 17:53:30 +08:00
2026-07-12 07:52:45 +08:00
2026-07-19 09:13:51 +08:00
2026-07-12 11:51:31 +08:00
2026-07-21 17:48:04 +08:00
2026-06-07 17:47:11 +08:00
44990ff194