* fix(channels): follow Buzz threads mapped in the connection repository
With channel_connections enabled, ChannelManager stores a bound author's
thread mapping only in the connection repository, but Buzz's thread-follow
gate read only the JSON ChannelStore. engaged_thread was therefore always
False for authors bound with /connect, and every unmentioned reply in a
thread the bot was already answering was silently dropped.
Buzz now resolves the connection identity before the mention gate and looks
the thread up through lookup_thread_id, a helper extracted from
ChannelManager._lookup_thread_id that both sides now share, so the gate and
the manager can no longer disagree about which store owns a mapping.
* fix(channels): drop the manager's own JSON-store read in the skill whitelist
_resolve_available_skill_names fell back to reading the JSON ChannelStore
when the caller's lookup_thread_id found nothing. For a bound message that
lookup is repo-only, so a legacy unbound mapping for the same chat decided
which custom agent's skill whitelist applied. The parameter is now required
and the caller passes "" when nothing is mapped.
Also pin the Buzz thread-follow fix against the real SQL connection
repository (bound through a real /connect) instead of a fake.
* docs(changelog): reference #6232 for the Buzz thread-follow fix
* fix(channels): log Buzz drops of unmentioned replies with no engaged thread
The thread-follow gate was the one drop in _handle_chat_event without a
log line. Log it at DEBUG with the thread root and whether the author was
bound, matching the allowlist drop, so a report that the bot stopped
following replies can be told apart from a lost bind or missing mapping.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers
* fix(gateway): name the worker spelling an operator can act on
The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.
Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.
* docs: state the single-worker requirement under both spellings
The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.
* test(features): isolate the browser capability surface from the shell's worker count
browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the multi-worker safety gates
The SQLite/scheduler gates in deps.py derived the worker count from
GATEWAY_WORKERS alone, but backend/Dockerfile and scripts/serve.sh launch
uvicorn without --workers, and uvicorn then takes the count from
WEB_CONCURRENCY. A Gateway scaled that way starts N processes with every
gate inert, which is exactly what the gates exist to prevent.
Read both variables, preferring GATEWAY_WORKERS, treat a blank value as
unset the way the compose command's ${GATEWAY_WORKERS:-1} does, and name
the variable that actually set the count in the refusal text. The channel
connection gate already reads both (channel_connections.py).
* fix(gateway): fall through to the fallback worker count
A non-numeric GATEWAY_WORKERS made the shared reader return one worker
immediately, so on the launches that pass no --workers (backend/Dockerfile,
scripts/serve.sh, backend/Makefile gateway) an operator's
GATEWAY_WORKERS=abc plus WEB_CONCURRENCY=4 started four processes while every
gate stayed inert. Keep looking at the remaining spellings and report a single
worker only after all of them have been tried.
The gate file also needs to be independent of the invoking shell: with
WEB_CONCURRENCY exported, the pre-existing
test_gate_treats_invalid_env_as_single_worker already failed on this branch,
because the reader now consults that variable too.
* fix(telegram): deny all users when allowed_users has no valid ID
TelegramChannel parsed channels.telegram.allowed_users with int() and
dropped every entry that failed without logging, and _check_user treated
an empty result as "no allowlist". A list of @usernames therefore opened
the bot to everyone. A single string ID was iterated character by
character ("123456" allowed users 1-6 and blocked 123456), which is what
an allowed_users: $ENV reference resolves to, and null or a bare integer
crashed the channel at startup.
The allowlist is now parsed into frozenset[int] | None: None (unset,
null, empty list, blank string) still allows everyone, a single ID is a
one-entry list, entries that are not positive numeric IDs (@usernames,
floats, booleans, comma-joined strings) are dropped with a warning, and
a configured list with no valid ID denies every user and logs an error.
* docs(changelog): reference #6230 for the telegram allowed_users fix
* fix(telegram): say "positive" in the allowed_users drop warning
0 and negative integers are numeric but still dropped, so "expected one
numeric Telegram user ID" misdescribed them. The warning now asks for a
positive numeric ID, matching _parse_telegram_user_id, and the drop test
covers 0 and -5 alongside @alice.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add host-bound Agent run control
* fix(extensions): preserve internal launches and scope run control
* docs(extensions): use a plugin-local run operation key
* fix(channels): run Discord connection-repository work on the Gateway loop
discord.py delivers messages on a private loop in the client thread, and
the Discord adapter awaited the channel-connection repository there even
though its SQLAlchemy engine and pool belong to the Gateway loop. On
PostgreSQL the first Discord message after the Gateway used the pool
failed with "got Future attached to a different loop" and was dropped; on
SQLite a pool-exhausting burst failed with "Queue is bound to a different
event loop" and left the pool's wait queue bound to the Discord loop, so
the Gateway's own queries then failed the same way.
Resolve the identity together with the intake commit, and run /connect
binding, on the Gateway loop through _submit_threadsafe_coroutine, as
Telegram, Feishu and DingTalk already do. Bind replies return through
_run_on_discord_loop, and start()/stop() now open and drain the
cross-thread submissions.
* docs(changelog): reference #6214 for the Discord connection-loop fix
* fix(channels): correct the unscheduled Discord bind comment and pin it
When _submit_threadsafe_coroutine refuses the bind, base.py closes the
coroutine, so consume_oauth_state never runs and the one-time code stays
valid; only the Discord message is swallowed. Say that instead of
claiming the code is consumed, and pin both halves with a test: the
attempt is handled (never a chat turn) and the code still binds later.
* fix(channels): undo Discord typing when the Gateway hand-off fails
Since identity resolution moved to the Gateway loop, both inbound paths
started typing and the ack reaction as soon as the hand-off was
scheduled. A lookup that then raised (database outage, pool-checkout
timeout) dropped the message but left an endless typing loop and a
success reaction behind.
The hand-off now returns its completion future (new
Channel._submit_threadsafe_coroutine_future, which the bool helper wraps)
and the Discord side waits on it through asyncio.shield. Typing still
registers before the hand-off settles, so a fast reply's _stop_typing
cannot run first; the ack reaction waits for the commit. On failure, or
when stop() drains the hand-off, the indicator this message started is
stopped (never one another in-flight message owns) and no ack is added.
* fix(channels): keep shared Discord typing alive for dependent messages
Two messages to the same target share one typing task: the first starts
it, a later one reuses it. If the creator's hand-off failed after a
follower had committed, or while the follower was still pending, the
creator stopped the indicator the follower relied on, and nothing
restored it while the follower's request ran.
Count the hand-offs that depend on each typing task. A pending or
committed hand-off holds the task (a commit keeps its hold until a reply
stops the indicator), a failed one releases its hold, and the indicator
is stopped only when the failed message started it and no hold remains.
The count is dropped when the task ends or the channel cancels typing.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): drain the config read-modify-write across cancellation
Route the five MCP config endpoints through the canonical
run_drained_write helper and fold the tools-cache reset into the drained
unit, so a cancelled caller cannot detach from the worker holding the
extensions-config locks or skip the cache invalidation behind a settled
write. Caller-facing HTTPException contracts re-raise unlogged.
Fixes#6209
* fix(mcp): type the drain helper against the worker signature and unlist drain adopters
- _run_drained_mcp_apply now mirrors the canonical helper's [**P, T]
Callable[P, T] typing so each call site's argument list is checked;
- AGENTS.md stops enumerating the adopted routers, so the line stops
needing an edit on every new adopter.
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* fix(browser): pin browser connections to the SSRF-vetted addresses
The navigate screen and the per-request guard resolve a hostname to vet
it, but Chromium resolved the name again to open the connection. A
rebinding DNS server could answer both checks with a public address and
the connection with a private or cloud-metadata one, and the browser
followed it.
Launch each private browser through a per-session loopback SOCKS5 proxy.
Over SOCKS5 Chromium sends the hostname itself, so the proxy resolves it
exactly once through the session's egress resolver and connects only to
the addresses that resolver vetted. resolve_browser_egress applies the
same allow_private_addresses policy as validate_browser_url, on top of a
new url_safety.resolve_public_addresses helper that
validate_public_http_url now shares. Loopback is proxied explicitly with
bypass "<-loopback>". The tools, navigate_and_capture and the Live
WebSocket all pass the resolver, because a thread's first caller fixes
its launch options.
CDP-attached Chrome is unchanged. Delegated fetch services (crawl4ai,
Browserless, fastcrw) still resolve on their own side, which the Gateway
cannot pin.
* docs(changelog): reference #6201 in the browser egress pinning entry
* fix(browser): bound egress proxy faults and close, and record the WebRTC gap
Review follow-ups for the pinned browser egress proxy:
- A resolver exception other than ValueError escaped the connection
handler as an unretrieved task error. Log it with its traceback and
reply SOCKS5 general failure (0x01), so a resolver fault stays
distinguishable from a policy refusal (0x02).
- Bound the resolve with its own timeout. The handshake timeout did not
cover it, so a wedged getaddrinfo held the half-open tunnel until
Chromium gave up. A timeout replies host unreachable.
- Bound close(): Server.wait_closed() also waits for a connection whose
handler registered after the writer sweep, which could stall session
close and eviction for the 15 s handshake. A handler that starts after
close now drops its connection without opening a tunnel.
- Docs said a launched browser connects "solely" through the proxy.
WebRTC UDP does not traverse a SOCKS proxy, so AGENTS.md, README (en/zh)
and both changelogs now scope the guarantee to TCP and name the gap.
* fix(channels): send Discord typing indicators through the discord.py 2.x API
_start_typing() called channel.trigger_typing(), which discord.py removed
in 2.0; the project requires discord.py>=2.7.0. The typing loop swallowed
every exception, so each tick raised AttributeError and no indicator was
ever sent, and the tests never noticed because they mocked the removed
method on a SimpleNamespace.
Await channel.typing() instead (awaiting the returned Typing sends one
indicator), and log a failed tick at DEBUG rather than discarding it. The
existing typing-lifecycle tests now mock typing(), and new tests drive
_start_typing() against a fake shaped like the 2.x Messageable, which has
no trigger_typing.
* docs(changelog): reference #6138
* fix(channels): warn on a Discord typing loop's first failed indicator
Logging every failed typing tick at DEBUG left a permanently dead
indicator invisible at the default INFO level, the same silence that hid
the trigger_typing AttributeError. A bot that lost Send Messages in a
channel (403 on every tick) or is being rate limited could not be told
apart from a working one without enabling DEBUG.
Log the first failure of each typing loop at WARNING with the traceback,
and later ticks of that loop at DEBUG so a retrying loop cannot flood the
log. The level is per loop, so the next reply's loop warns again.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(blocking-io): catch socket.getaddrinfo on the event loop
Blockbuster wraps socket.socket methods but not the module-level
resolver, so a synchronous hostname lookup on the loop passes the strict
gate. The SSRF guard in deerflow.community.url_safety resolves with
socket.getaddrinfo from async web tools, and the anchors added next stay
green against that unfixed code without this rule.
Add socket.getaddrinfo to _PROJECT_BLOCKING_RULES, scoped to app.* and
deerflow.* callers. loop.getaddrinfo and asyncio.to_thread resolve in a
worker thread and are not flagged. The full make test-blocking-io suite
stays green with the rule active (187 passed).
* fix(community): resolve SSRF-guarded hostnames off the event loop
validate_public_http_url resolves hostnames with blocking
socket.getaddrinfo. The crawl4ai and Browserless web_fetch, web_capture,
browser_navigate, the Gateway browser navigate route, and the Live
stream's navigate input and seed all called it directly from async code,
so a slow DNS answer for a model- or user-chosen URL stalled the Gateway
event loop for the whole lookup.
Run the guard through asyncio.to_thread at each of those call sites, the
same way personal_network already offloads it. The screening result is
unchanged. fastcrw's web_fetch is a sync tool and personal MCP validation
already runs in a worker, so neither needed a change.
* docs(changelog): reference #6140 in the SSRF guard event-loop entry
* fix(browser): run the Playwright request guard off the shared browser loop
BrowserSession._install_request_guard screens every redirect hop, iframe
and subresource by calling the SSRF guard inside the async route
callback. The guard resolves hostnames with blocking socket.getaddrinfo,
and that callback runs on the process-wide Playwright loop, which also
pumps Live frames and input for every browser session, so one slow
lookup stalled all of them.
Run the guard through asyncio.to_thread there as well, and pin it with a
strict-gate anchor that drives the real route handler. Also note in the
url_safety docstring that the screen is check-time only: a caller that
re-resolves at connect time is not pinned to the vetted IPs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`_can_send_rich` checked `len(text)`, a code-point count, against the 32768
rich-message cap, while Telegram measures that cap in UTF-16 code units like
every other limit it enforces. An emoji-heavy reply that fits by code points
is still over the cap by code units, so the rich send was attempted, rejected
by the API, and the reply degraded to chunked plain text.
The check now reuses `_first_utf16_chunk`, the bounded scan the plain-text path
already uses, so eligibility costs the limit rather than the whole reply.
Memory router kept its own bare drain wrapper (_run_memory_mutation,
five call sites) after #6151 hoisted the superset into
app.gateway.persistent_writes.run_drained_write - this is the remaining
per-router fork called out in that review.
Each call site passes the domain errors its handler already maps
(NotImplementedError 501s, ValueError/KeyError 400/404/409 mappings,
MemoryConflictError/MemoryCorruptionError, OSError 500s), so the helper
stays silent on them exactly as the removed wrapper did; unexpected
failures gain the type-only lost-failure log line per the AGENTS.md
rule. Behavior is otherwise unchanged: same off-thread execution and
cancel-drain semantics.
The router test's drain stub moves to the shared name and pins the
clear-memory error tuple.
Signed-off-by: yetuge <2219677952@qq.com>
Checkpoint durable refs for host-externalized tool results and materialize verified bytes before model calls on another Gateway.
Refs: #4189
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
* refactor(gateway): hoist the persistent-write drain into one shared helper
Each persistent-write surface forked its own per-router drain wrapper:
_drained_write in routers/agents.py (action label, expected_errors
pass-through, lost-failure type-only logging), _run_store_mutation in
routers/subagents.py (bare drain), and the inline await_drained call in
routers/managed_models.py. Hoist the superset into
app.gateway.persistent_writes.run_drained_write and point all three
routers at it, per the review suggestion on #6087 (its family
predecessors #6078/#6092/#6093 have since landed).
The subagents and managed-models call sites now also emit the type-only
lost-failure log line on a drained-then-failed write (previously
silent), matching the tradeoff acknowledged in the #6087 review.
artifacts.py and personal_mcp.py keep their direct await_drained forms:
the former drains a coroutine with its own rollback flow, the latter
needs no label or expected-error handling.
Router tests move their monkeypatch targets to the shared name; the
lost-failure assertion pins the helper's logger. AGENTS.md points the
next persistent-write surface at the canonical implementation.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(gateway): quiet expected domain errors and scope the AGENTS.md drain claim
Review follow-ups on #6151:
- The subagents create/update call sites pass (ManagedSubagentExistsError,)
and (FileNotFoundError,) so the shared helper stays silent on routine
4xx outcomes (duplicate-name 409, stale-store 404 race), matching
create_agent's (AgentExistsError,).
- The managed-models save passes (HTTPException,), so the helper stops
double-logging failures that _save_with_failure_logging already
reports and stays silent on its routine 4xx outcomes.
- The AGENTS.md sentence now names the three converted routers instead
of implying every persistent-write surface uses the helper; the
shortened form also keeps the file under the guidance byte budget
that agent-guidance enforces.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* feat(persistence): add notification delivery outbox for scheduled task IM push
New notification_deliveries table and repository (migration 0012) backing
the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id
plus event plus provider plus target, atomic claim of due rows, and
exponential-backoff retry state moving pending to sending to sent or failed.
Execution status stays in scheduled_task_runs; delivery status lives here.
* feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels
Closes the loop for issue #4254. On run completion the scheduler writes
outbox rows for the task owner's connected channel identities (success and
failure events; interrupts stay silent), and a new NotificationDeliveryWorker
polls due rows and pushes them through the channel's proactive send path.
Channel gains send_notification; WeCom implements it via the WS client's
send_message with errcode checking, the base class raises for providers
without proactive push.
Delivery is best-effort and never shadows execution status: enqueue and send
failures are logged and recorded in the outbox, which retries with backoff.
Channel liveness is re-checked at delivery time, and the worker stops before
the channel service on shutdown. Feature activates only when channel
connections are enabled; no new dependencies.
Verified end to end against a live WeCom bot: scheduled run completion
produced an outbox row that was delivered in under one second.
* feat(scheduler): include scheduled run result summary in IM notifications
The v1 notification only carried the run outcome skeleton (status, task id,
run id). Completed runs now include the agent's final answer so the IM push
is actually useful headlessly.
- The delivery worker accepts an optional resolve_run_summary(run_id,
user_id) resolver; for run_completed events it enriches the rendered text
with a "Result:" line (bounded to 1000 chars). The lookup is best-effort:
resolver failures or missing summaries degrade to the skeleton text and
never block or fail the delivery.
- The summary is read at delivery time from runs.last_ai_message (the
convenience column written on run completion), not snapshotted into the
outbox payload; retried deliveries re-read the freshest value and the
outbox row payload stays skeleton-only. The claimed outbox row is never
mutated in place.
- Gateway wiring scopes the read to the outbox row's owner_user_id so a
stale row cannot pull another user's run content.
- Failed runs keep the error-only skeleton: a partial answer on a failed
run would be misleading.
Tests: 9 new worker cases (rendering incl. truncation, sync/async
resolvers, failure/missing fallbacks, no resolver call for failed runs).
Full scheduled + outbox + channels regression green (441 passed), ruff
check + format clean.
* fix(scheduler): address review feedback on notification delivery reliability
Review follow-ups for the scheduled-task IM notification outbox:
1. Crash recovery for "sending" rows: claim now stamps updated_at and the
worker reconciles rows stuck in "sending" past a 10-minute lease back to
"pending" before every claim. Per-row try/except isolates a crashing
delivery from the rest of the claimed batch.
2. Double-send race under multiple workers: claim returns exactly the rows
its own UPDATE...RETURNING flipped (guarded by status='pending'), so a
rival worker's concurrent claim can no longer be handed out twice.
3. Restore non-ASCII comments and drop the BOM accidentally introduced in
test_scheduled_task_service.py.
4. Manual "run now" completions no longer enqueue IM notifications; the
user is already watching the UI. Missing trigger metadata fails safe
towards notifying.
5. Channel-outage failures no longer consume the retry budget: they are
parked with count_attempt=False on a flat 15-minute backoff so an
hours-long outage cannot exhaust retries and drop the notification.
Tests: repository 16, worker 21, scheduled-task service 31; targeted
regression across notification/scheduler/channel suites: 511 passed.
* fix(scheduler): address round-2 review on IM notification delivery
Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(scheduler): complete IM notification docs and drop unused list API
Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): address round-3 review on IM notification delivery
Cap channel-down parking by row age and parked attempts, redact IM egress
content, narrow WeCom transport exceptions, add lifespan wiring tests, and
run ruff format so lint-backend passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): redact entire PEM blocks in IM egress text
The previous BEGIN-only pattern left key material and the END marker in notifications.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): align 0014 parked_attempts with bootstrap schema tests
Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): format notification tests and shorten alembic revision id
ruff format --check failed on test_notification_delivery_worker.py.
0015_notification_delivery_parked_attempts exceeds alembic_version.version_num
(VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead.
* test: fix post-merge CI failures after migration renumber
Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config.
* fix(persistence): chain the notification outbox migrations after 0026
Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two
outbox revisions are renumbered and re-parented:
- 0017_notification_deliveries -> 0027_notification_deliveries
- 0018_parked_attempts -> 0028_parked_attempts
Only the revision ids, parents and the migration guide change.
* fix(scheduler): adapt the notification hook and its docs to main
- tests: the completion tests assert on the atomic complete_run record;
new cases cover a completion that was not recorded, a task deleted
meanwhile, a failed task read, and the outbox being off.
- AGENTS.md: drop the root note and shorten the backend note to a pointer;
the two notes pushed two effective guidance chains past the 96 KiB limit.
- CHANGELOG: move the entry from the released 2.1.0 section to Unreleased
and add its link references.
* test(persistence): move the chain-head pin to the outbox migrations
test_migration_0026 pinned 0026 as the head, so it failed once the outbox
revisions chained after it. It now checks chain membership like the older
migration tests, and a new test file owns the head pin, checks the parents
and the 32-character revision id limit, and runs the upgrade and downgrade.
* fix(gateway): wire scheduled-task notifications after the channel service starts
The outbox wiring sat in the scheduler block and asked for the running
channel service there. #5035 moved the channel service start to after the
scheduler, and the two changes merged without a conflict, so on every
startup the lookup returned None and the feature disabled itself with the
"write-only outbox" warning. The lifespan tests patched the lookup to a
constant and kept passing.
The wiring now runs in its own step after the channel service start. The
scheduler is built as on main, and ScheduledTaskService gains
attach_notification_outbox() so the enqueue side is enabled once the
delivery worker is running. The guard, the worker and the shutdown order
are unchanged.
Tests: the lifespan helper can model the real accessor (None until the
channel service has started); a new case fails without this fix.
* fix(scheduler): enqueue from scheduler start and keep the push when the title read fails
Follow-up to the startup-order fix, from review.
- The enqueue side is wired with the scheduler again (it only needs the
durable table), and only the delivery worker waits for the channel
service. Attaching it later left a window after the scheduler started in
which a finished run wrote no outbox row. If the channel service is
missing or the worker cannot start, the enqueue side is detached, so the
outbox still never becomes write-only.
- A failed task read after complete_run no longer drops the notification:
the title is optional and the message falls back to the task id. A task
deleted meanwhile still stays silent.
* fix(channels): park WeCom notifications while the SDK socket is closed
The WeCom SDK raises a plain RuntimeError when its socket is not open, and
send_notification only mapped ConnectionError, TimeoutError and OSError to
ChannelUnavailable. A disconnect therefore spent the counted retry budget
and settled the row as failed after about 15 minutes, although the docs
say transport outages park without consuming retries.
send_notification now asks the client's is_connected first. The new test
uses a real, never-connected SDK client: it raised RuntimeError after three
retries before, and parks now.
* docs(scheduler): describe which outcomes notify; mirror the changelog entry
- README (en/zh) and CONFIGURATION.md: only runs that finish as success or
failed notify; occurrences that end without a finished run (launch error,
queue timeout, restart recovery) do not. Parking lasts up to about a day.
- CHANGELOG: cite the issue as text like the rest of the file, and add the
entry and link reference to CHANGELOG_zh.md.
- AGENTS.md: drop the backend pointer so this branch adds nothing to the
guidance chains closest to their size limit; the scheduler-side rules
and the lifespan order now live in app/channels/AGENTS.md.
* docs(changelog): cite #6135 next to #4843 for the notification outbox entry
* fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections
Review round 1 on #6135.
The delivery worker now takes resolve_connections (the Gateway wires
ChannelConnectionRepository.list_connections) and, before the channel
liveness check and the summary read, requires the row's owner to still
hold a connected (provider, external_account_id). A disconnected target
is finalized with mark_failed(terminal=True); a lookup failure or an
unexpected result shape parks the row. Terminal outbox rows ignore
later status writes.
WeCom: the SDK raises RuntimeError from send_message for a socket that
is not open and for a dropped connection, but also for a platform
rejection ("Reply ack error: errcode=..."). Only that prefix keeps
counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and
websockets' ConnectionClosed park the row as ChannelUnavailable.
* fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape
---------
Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* test(scripts): cover discord extra auto-detection
The discord mapping in scripts/detect_uv_extras.py worked but was neither
listed in the "currently maps" docstring nor covered by a test, unlike the
buzz mapping that grew up beside it.
* fix(channels): point the Discord missing-dependency hint at the discord extra
discord.py is the only IM transport that is not a core dependency — it ships in
the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and
deploy.sh already expand. Its ImportError hint told users to run `uv add
discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every
restart syncs with `--locked`. Mirror the buzz hint wording and note the extra
in config.example.yaml next to the buzz block.
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
* fix(skills): drain custom-skill mutation tails across cancellation
* fix(skills): log drained skill mutation failures and drain state/install tails
Address review feedback on #6078: record cancelled-then-failed mutation
tails inside the drained unit (exception type only, per the managed-models
precedent), and extend the drained tail to the remaining write-to-cache
shapes in update_skill and the skill install path.
* test(skills): pin the lock contract under the drained cancellation timing
The cancelled update_skill caller now drains its mutation tail, so
CancelledError surfaces only after the tail settles. Start the MCP
writer without awaiting the cancelled task first, assert the section
stays owned while the drained tail is in flight, then await both and
assert the final order.
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* fix(channels): split Telegram messages by UTF-16 code units, not code points
* perf(telegram): clip stream previews without walking the whole reply
Every cumulative stream update measured the full answer and then split the full
answer to keep only its first chunk, so a long reply cost two complete passes
per update on the async channel path. _first_utf16_chunk stops at the first
character that does not fit, and the preview is built from that bounded prefix.
The tests now measure chunk length with the UTF-16 codec instead of a copy of
the production formula, cover the width helper and the bounded clip directly,
and fail if the preview stops being bounded by the limit.
* fix(uploads): reject filenames reserved for temporary staging
Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace.
* fix(uploads): report reserved names before HTTP batch ingestion
Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation.
* fix(uploads): reject Windows-style staging basenames on POSIX
Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation.
Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep blocking SDK teardown off the event loop in stop()
SlackChannel.stop() called SocketModeClient.close() inline. close() joins
the SDK's message-processor thread (~0.7s every call, measured) and waits
for in-flight listeners, whose blocking Web API calls can run up to the
WebClient timeout, so every run, stream and channel on the Gateway stalled
during shutdown and POST /api/channels/{name}/restart.
Feishu and DingTalk had the same shape: they joined their SDK threads
inline with a 5s timeout, and since those threads only return on a fatal
error, the join normally froze the loop for the full 5s.
The teardown now runs through asyncio.to_thread. Slack detaches the client
first so a still-queued connect skips it, and shields and tracks the close
task so a cancelled stop() leaves it running and a retried stop() awaits
the same close instead of closing twice.
tests/blocking_io/test_channel_stop_offloop.py anchors all three stop()
paths to the strict Blockbuster gate, driving a real slack-sdk client.
* docs(changelog): reference #6134
* fix(channels): join the Discord client thread off the event loop in stop()
DiscordChannel.stop() still joined its client thread inline with a 10s
timeout. The thread usually exits right after the cross-loop client close,
but a timed-out close or a slow _run_client() drain keeps it alive for the
whole timeout, stalling the shared Gateway loop the same way the Slack,
Feishu and DingTalk teardown did. Run the join through asyncio.to_thread so
stop() matches the channels/AGENTS.md teardown rule, and add Discord to the
strict Blockbuster anchor.
* fix(auth): reject a non-ASCII ID token nonce instead of raising
The nonce comparison in validate_id_token still used the local
secrets.compare_digest on str operands, which raises TypeError when either
side holds non-ASCII characters. The nonce claim is provider-controlled
text, so such a provider response turned the callback's sso_failed redirect
(OIDCValidationError) into an unhandled 500.
Route it through the shared constant_time_equals helper #6076 introduced
for the same five compare sites, and drop the now-unused local helper.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(auth): reject a non-string ID token nonce claim like a mismatch
A provider returning a truthy non-string nonce claim (e.g. an int)
slipped past the missing-claim guard and crashed the comparison helper
with AttributeError, which the callback turned into a 500 instead of
the sso_failed redirect. Guard the claim type before comparing so every
malformed nonce takes the same OIDCValidationError path.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): drain agent and user-profile writes across cancellation
A client that disconnects mid-request cancels the handler task. On the four
persistent-write endpoints of /api/agents - create, update, delete, and the
USER.md write - a bare asyncio.to_thread then either cancels a still-queued
worker (the write silently never happens) or detaches from a running one and
drops its failure, because the handler's except never runs.
Route the four writes through await_drained like the managed-subagent (#6023),
managed-model (#6024), and custom-skill (#6078) mutations, logging a lost
worker failure with the exception type only before the drain consumes it.
Expected domain errors (AgentExistsError on create) re-raise unlogged; the
caller still maps them to a 409 while connected. Reads stay bare: abandoning
them loses nothing.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(gateway): address review nits on the agents write drain
Make expected_errors a positional-only parameter before *args so the
ParamSpec signature is checker-clean under PEP 612 (the one call site with
extra worker arguments passes () explicitly), acknowledge that
non-cancelled failures log twice like artifacts.py's drained commit does,
and drop the unused request stub helper from the tests.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): anchor agent-name validation against a trailing newline
Two copies of the shared ^[A-Za-z0-9-]+$ grammar still used re.match,
whose $ also matches before a final newline, while every other copy uses
fullmatch. POST /api/agents {"name": "reviewer\n"} therefore passed the
router guard and died inside the strict file store as an HTTP 500, and
DeerMem accepted the same name as an agents/{name}/facts directory.
* test(agents): state which trailing-newline params the regression actually covered
.match only accepted "reviewer\n" on main; "reviewer\n\n" and "reviewer \n"
were already rejected, so the docstrings should not read as if every param in
these parametrizations was broken before the fix.
* fix(gateway): reject non-ASCII tokens instead of raising in constant-time compares
hmac.compare_digest raises TypeError when a str operand holds non-ASCII
characters. Starlette decodes header bytes as latin-1 and query values as
UTF-8, so a single 0xE9 byte in a CSRF token, GitHub webhook signature,
internal auth token, or OIDC state turned the expected 403/401 into a 500.
Route the five client-facing comparisons through a new
app.gateway.utils.constant_time_equals helper that compares the UTF-8
encodings ("surrogatepass" so even a lone surrogate cannot raise). ASCII
results are unchanged, and no bypass was possible before the fix.
* fix(provisioner): reject non-ASCII API keys instead of raising; drive OIDC test over HTTP
The standalone provisioner compared X-API-Key with secrets.compare_digest on
str operands, so a non-ASCII header raised TypeError (500) the same way the
Gateway sites did. It cannot import app.gateway, so encode inline.
The OIDC state regression test now mounts the auth router and sends the
percent-encoded state through a real signed state cookie instead of stubbing
get_state_cookie and calling the handler directly.
* docs(changelog): reference #6076
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): expose DEERFLOW_USER_ID to bash commands
A skill's SKILL.md can trigger scripts, but those scripts run as bash
subprocesses inside the sandbox and cannot reach the Gateway auth context,
so get_effective_user_id() is unavailable to them and user-scoped skills
are impossible to write. The maintainer settled the direction on #3919:
inject the id as DEERFLOW_USER_ID.
Every bash command now begins with 'export DEERFLOW_USER_ID=<resolved>; '
(or 'unset' when the resolved id is unusable), mirroring the existing
DEERFLOW_CHANNEL_USER_ID channel-identity prefix (#3914). The id rides the
command string rather than execute_command(env=...), because a non-empty env
switches AioSandbox to the bash.exec API reserved for request-scoped secrets.
The earlier attempt (#4029) was closed for making every command carry the
new export without updating the exact-command assertions in
tests/test_channel_user_id_env.py. Those assertions are updated here, and the
composition of the two prefixes is pinned by a dedicated test.
* fix(sandbox): fix the DEERFLOW_USER_ID unit test and cover review findings
Follow-up to the review on this PR.
Unit test (CI `backend-unit-tests (2)`): the change moved the command
contract but left the exact-command assertion in
tests/test_skill_request_scoped_secrets.py::test_local_bash_forwards_env_and_timeout
expecting the bare `echo hi`. The local validation run used a `-k` filter that
did not select that file, so the breakage was invisible there. The assertion
now pins the composed command, and `_is_windows` is patched explicitly so it no
longer depends on the host OS.
[P2] Windows local sandbox: `export` is invalid under the PowerShell/cmd.exe
fallback, but skipping the prefix outright left the promise unmet on Windows.
The id is published through `execute_command(env=...)` there instead:
LocalSandbox layers `env` into the per-process environment and, unlike
AioSandbox, has no persistent session that could carry a stale value. It is
deliberately kept out of `injected_env` so the identifier does not enter the
secret-redaction set.
Docstring: `_user_identity_prefix` now carries the same "informational, not
authenticated identity" caveat the channel variable has in
backend/app/channels/AGENTS.md. The value is authorization-grade at the Gateway
boundary, but any bash command can overwrite its own environment, so skills
must not treat the shell variable as proof of identity.
Docs: the en/zh sandbox.mdx identity sections and the channels/AGENTS.md
identity paragraph now describe the second, outermost prefix, its ordering
against the channel prefix, and the Windows behaviour.
* fix(goal): show the goal evaluator the tool calls and results
The evaluator read only the text of user and assistant messages and was told
to fail closed without explicit evidence, so it stood down with
missing_evidence on most completed file tasks: the tool calls and tool
results that the web UI shows never reached it.
Include the assistant's tool calls and the tools' results inside the
existing message window, shortened (200 characters per argument value, 600
per step, line breaks kept), and tell the evaluator to treat tool results as
data, that a successful tool call alone does not meet a goal, and that a
guessed input means needs_user_input. The no-progress and final-clear
signatures are unchanged.
* fix(goal): tighten the evaluator's tool evidence after review
- Pair each tool result with the latest call of its id before it. Some
providers reuse ids such as call_0 across turns, and the window-wide
map labelled an earlier read with a later write's name.
- Keep model-supplied tool names on one line in calls and result labels,
and escape U+0085, U+2028 and U+2029 in arguments and results, which
json.dumps(ensure_ascii=False) leaves raw, so no tool step can start a
new evidence line.
- Leave out the results of a hidden assistant message's calls. The web UI
renders a result only under its call, so neither shows there either.
A clarification prompt stays: the UI shows it as its own card.
- Over the 12,000-char cap, keep whole lines from the end instead of
slicing mid-line. Keep the latest user message that the cut drops at
the top, whole when it and the lines after it fit, else shortened to
2,000 chars. If the next assistant message back does not fit whole,
keep its end in the room left. A marker counts the lines left out.
- Docs: the Gateway AGENTS.md and a worker docstring no longer say the
evaluator never sees tool messages.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(gateway): drain managed model saves on cancellation
* test(gateway): pin managed model save cancellation drain
* fix(gateway): log failed drained managed model saves
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): keep guidance within hard size limit
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): reject SQLite file: URIs instead of opening them as filenames
resolve_sqlite_conn_str passed a file: connection string through
unchanged, but LangGraph's SQLite checkpointer and Store factories
(SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect
without uri=True, so SQLite treated the whole URI as a literal filename:
file:checkpoints.db?mode=rwc created a file with that exact name in the
working directory, file::memory:?cache=shared persisted to disk, and a
file:/// URI failed to open. The readiness probe did open URIs with
uri=True, so it checked a different file than the runtime used and
reported in-memory URIs as not_configured.
Reject file: URIs in resolve_sqlite_conn_str, the one helper all four
factories go through, with an error that names
checkpointer.connection_string. The probe drops its URI parsing and
reports a rejected connection string as unreachable.
* docs(changelog): reference #6069
* test(persistence): pin the case-sensitive file: URI check
* fix(channels): decode Slack entity escapes in inbound message text
Slack sends a user-typed &, < and > as &, < and > in event text, so raw <...> always marks a control sequence. The Slack channel passed that text to the agent as-is, so code, comparisons and names like R&D reached the agent as 'a < b', 'R&D'. Decode the three entities after bot-mention stripping (amp last, so an escaped entity is decoded once) and keep mentions and links untouched.
* test(channels): cover Slack entities inside link labels
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): render the bundled interpreter repair path without backslash escapes
The repair hint used {sys.executable!r}, whose repr() escapes Windows
backslashes - users saw 'D:\college\...\python.exe' in the
copy-paste guidance, and
test_stale_bundled_interpreter_can_be_repaired_without_reinstall fails
3/3 on Windows because the raw path never appears in the detail. The
!r was written for the surrounding quotes; on POSIX paths that is its
only effect, which is why CI never saw it.
Quote the raw path explicitly instead. On Windows the three test params
now pass (22/22 in the file); POSIX output is unchanged apart from the
quoting coming from the literal instead of repr().
* fix(gateway): json-escape the interpreter command in the repair hint
Review follow-up (P2): the raw Windows path in the hint cannot be
pasted into the server's JSON ('\U' is not a JSON escape), so the
prescribed repair could not be saved. Show the command value via
json.dumps(sys.executable) - pasteable JSON on every platform - and
keep the raw path in a parenthetical for terminal use, per review.
