* fix(serper): report a format error for results with no usable objects
`_response_items` skipped non-object entries but returned the filtered
list even when it was empty, so a non-empty `organic`/`images` array
holding no objects fell through to "No results found" / "No images
found" -- telling the agent the query legitimately matched nothing
instead of surfacing the malformed payload.
Treat a non-empty container with no object entries as malformed, matching
the containment rule #6603 documents for this provider family (brave
#6550, groundroute #6603, tavily #6645, webz). `web_search` and
`image_search` share this helper, so both now report the provider's
format error. Missing, null, and empty containers keep reading as an
empty search, and mixed lists still keep their valid entries in order.
* fix: address review and validation findings for PR 6779
* fix(opensandbox): report the command deadline as Exit Code: 124
`execute_command` had no timeout branch, so a command that hit its deadline
was reported as whatever the SDK happened to surface. OpenSandbox is the last
provider without the `Exit Code: 124` marker that Local/AIO/Tenki/E2B/BoxLite
emit, and which #6847 established the evidence chain parses to recover the
real shell status.
The deadline is enforced by execd, not by the client: the timeout travels to
the server in `RunCommandOpts`, the SSE read timeout is disabled, and the
process group is reaped with SIGKILL. So a timed-out command does not raise --
it comes back as a `CommandExecError` result with the signal death reported as
exit code -1, which the adapter rendered as `Exit Code: -1`.
Normalize that pairing to the 124 marker, keeping the partial output the
command produced before the kill. The guard is deliberately narrow: a command
that exits non-zero on its own never carries the error event, and a failed
launch shares the same error event with no usable code, so neither is
relabelled a timeout.
Tests: three cases pin the deadline paths (client-side TimeoutError from a
bridge, the server-side -1 kill with output, and the free-text shape that must
stay on the existing missing-exit-code path as a start failure), plus a case
pinning that ordinary exits keep their own code.
RED/GREEN: the deadline cases fail on the unmodified source and pass with it;
test_opensandbox_provider.py 114 passed (110 pre-existing unchanged); the four
sandbox provider suites 601 passed; ruff check and format clean.
* fix(opensandbox): recognize explicit deadline results
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(sandbox): truncate quarantine path keys to stay under Windows MAX_PATH
The quarantine nests three hashed path components (scope, sandbox id,
container id). Three full 64-char SHA-256 hexdigests exceed Windows'
260-char MAX_PATH once the quarantine root lives under a long path, so
mark() fails with ENOENT creating the generation record. Truncate the
digest to 16 hex chars, matching the sandbox-naming convention used
elsewhere in this module.
* test(sandbox): pin quarantine key truncation to 16 hex chars
Guard against a regression that reverts the key digest to a full 64-char
SHA-256 hexdigest: three nested 64-char components exceed Windows' 260-char
MAX_PATH and re-break record creation, while Linux CI still passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): preserve text appended before the first state snapshot
* test(client): exercise first-snapshot reconciliation with stable IDs
---------
Co-authored-by: BaiHeng Peng <191328900+Aldenshile@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): route gpt-6 tool calls through the OpenAI Responses API by default
The locked langchain-openai==1.2.1 routes gpt-6 models configured with
langchain_openai:ChatOpenAI through /v1/chat/completions, where OpenAI
rejects function tools for the gpt-6 family with an HTTP 400. Upstream
only added the gpt-6 + tools -> Responses API branch in 1.7.x, so
backport that routing decision at the model factory instead of bumping
the dependency (large uv.lock churn, wider regression surface).
_apply_openai_responses_api_default() injects use_responses_api: true +
output_version: responses/v1 for BaseChatOpenAI subclasses whose model
starts with gpt-6, unless the operator configured an explicit value.
Fixes#6841
* fix(models): narrow gpt-6 Responses API default per review
Addresses review findings on #6863:
- Custom endpoints (base_url/openai_api_base) are never touched, so
OpenAI-compatible gateways/proxies (LiteLLM, vLLM, one-api) keep their
chat-completions routing instead of being silently re-pointed at /responses.
- AzureChatOpenAI is skipped too (MRO-based, so env-configured Azure and any
subclass are covered): Azure's Responses API requires api-version >=
2025-04-01-preview.
- An explicit use_responses_api still always wins; a lone explicit
output_version is preserved while use_responses_api: true is still injected,
so gpt-6 no longer silently falls back to /v1/chat/completions.
- The default injection is logged so operators can trace the /responses switch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(models): treat env-var-wired gateways as custom endpoints for gpt-6
Addresses the remaining review findings on #6863:
- A gateway wired purely via OPENAI_BASE_URL/OPENAI_API_BASE (the openai SDK
fallback when no base_url is passed) is now treated like an explicit
endpoint, so a gpt-6 model behind an env-var-wired gateway is not silently
re-pointed at <gateway>/responses. The known api_base-subclass blind spot is
documented in the comment block.
- Regression tests pin the env-var endpoint gate and the explicit-wins
precedence (an explicit use_responses_api still wins on a custom endpoint).
* test(models): isolate Responses routing from endpoint environment
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): skip tasks a cancel released while the poll claim was in flight
claim_due_tasks selected due rows with FOR UPDATE SKIP LOCKED, then set
the lease through an ORM flush that updates by id alone. FOR UPDATE is a
no-op on SQLite, so a request_cancel that committed between the SELECT
and the flush had its cleared poll lease written back. The cancellation
worker only claims rows without a live lease, so the remote cancel waited
for mcp_tasks.lease_seconds (120 s by default), and the task was polled
once more for a result apply_snapshot then discarded.
The claim now selects candidate ids and leases each one with a
conditional UPDATE that re-checks the claim predicates (pollable status,
no cancel request, due next_poll_at, no live lease), returning only the
rows it actually leased, the same atomic fence release_claim already
uses. On Postgres the row lock still orders the two writers.
* docs(changelog): reference #6859 for the MCP task claim fence
* docs(mcp): note that a short poll claim batch is not topped up
* fix(knowledge): scope ragflow link cleanup to its native citation namespace
budget_source_artifact cleaned a delegated task summary with two regexes that
matched any #knowledge-... destination. When a task result carried both the
ragflow-native contract (#knowledge-<id>-<n>) and the unified contract from the
retrieval refactor (#knowledge-source-<id>), those regexes also flattened the
unified citation to plain text. budget_source_artifact only rebuilds the native
knowledge_sources, so the now-linkless unified evidence then failed the
prune_sources substring check and retrieval_sources was emptied. The native
citations in the same result survived, so a mixed result lost attribution for
its unified sources while keeping it for ragflow ones, even with budget headroom
to spare.
Anchor both regexes to the native namespace with a (?!source-) lookahead so they
leave #knowledge-source-<id> for prune_sources to reconcile downstream. Native
ids are 32-hex and never begin with "source-", so native link cleanup is
unchanged.
Add a regression that drives the real middleware entry point on a mixed-contract
task result and asserts the unified citation stays clickable and its
retrieval_sources record survives alongside the native one.
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
* fix(knowledge): remove dangling citation fragments from task summaries
---------
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(llm-error): retry provider errors reported inside a 200 stream
An OpenAI-compatible gateway that answers 200 early, to keep a slow
request alive with SSE keepalives, reports a later provider failure (a
504, say) as an SSE error event. The openai SDK raises that as the base
APIError, with no status code and no response, so _classify_error found
no status, no transient class name and no busy text, and called it
generic: the run failed on its first attempt.
An error with no status now takes an effective status from its own
contents, and the first source that yields one decides: a numeric code
on the error, or a numeric code, status or status_code in its body or
the body's error object; an explicit status in its message ([504],
HTTP 502, status 503); or type server_error (500). Its busy check also
reads each run of _, - and whitespace as one space, so rate-limit and
RATE_LIMIT_EXECUTION_TIMEOUT match "rate limit". Retry budget, backoff
and circuit breaker then treat it as the equivalent HTTP error. Errors
that carry a status are classified exactly as before, and the quota,
auth and burst-rate checks still run first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): note how status-less errors are classified
Add one sentence to item 9 of the middlewares guide, and an llm entry
under Fixed in both changelogs, for the retry of provider errors
reported inside a 200 stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the in-band status source order and leftmost match
Two rules of _extract_in_band_status had no test: a numeric status
field decides before the message is read, and within the message the
leftmost in-range status wins. Either rule could be inverted with every
test still green. Add one stays-generic case for each: code 400 with
"[503]" in the message, and "HTTP 400 after [503]".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): do not read an array index as an in-band status
The bracketed alternative of the in-band status regex had no lookbehind,
so `messages[502]` or `tools[0][503]` matched and their number decided
the error's status. The bracketed form now starts with the lookbehind
`(?<![\w\]])`: a `[` directly after a word character or a `]` opens an
index, not a status. The `http` and `status` alternatives are unchanged;
`[504]: ...` in running text still counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): let an in-band status preempt the separator busy check
The separator-normalised busy check ran for every status-less error,
even when the in-band lookup had already found a status: "rate-limit
header malformed" with `code: 400` was retried as busy, while HTTP 400
is not. It now runs only when no source yields a status, so an in-band
4xx decides like an HTTP one. The same text with no status anywhere
still counts as busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): let only 4xx and 5xx values decide the in-band status
The in-band lookup accepted 100-599 values from its first two sources,
so a 1xx-3xx status could decide the classification: `HTTP 200` in a
message whose server_error type means 500, or `code: 200` alongside
`status: 503`. Only 400-599 now count: `_as_http_status` skips a
smaller value as it skips a non-numeric one, so the candidate loop
moves on, and an out-of-range message match is skipped so the scan
continues. An error that carries a status stays as it was, and the
server_error source is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): describe the narrowed in-band status rules
The guide, the CHANGELOG and the CHANGELOG_zh now describe the three
narrowed rules: only a 4xx or 5xx value decides, an index such as
`messages[502]` is not a status, and the separator-matched busy check
runs only when no source yields a status. Only the sentences that
describe the affected forms changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the remaining in-band status guards
Cover the nested error.status_code field, a code above 599 that yields
to a later status field, a code written in non-ASCII digits, and an
in-band 4xx whose plain busy text still decides, as it does for an HTTP
4xx. These pin current behaviour; no code changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(llm-error): say which in-band fields are read on the error itself
Only `code` is read on the exception; `status` and `status_code` are
read in its body and in the body's `error` object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the in-stream retry entry to #6897
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(llm-error): require a separator and ASCII digits in an in-band status
The message regex read `status503` and `statuscode503` as a status
because the separator after `status` was optional, and it read
Arabic-Indic or fullwidth digits because `\d` and `int()` accept every
Unicode decimal digit. A status-less error with such text was retried
as a 503.
The `status` form now needs spaces or tabs, or a `:` or `=`, before
the number, and all three forms read only ASCII digits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(llm-error): pin the retry of an in-stream error after partial output
An `error` event that follows content chunks is retried like one that
comes first: the whole call re-runs and its result holds only the new
answer, while the tokens the first attempt already streamed to the
callbacks stay delivered, as with a mid-stream `ReadError`. The new
test pins that outcome, and a comment in `_classify_error` notes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Handle transient errors based on status codes
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(slack): accept file_share events and download inbound files
Slack was dropping every Events API message with a subtype, including
user uploads (subtype=file_share). Allowlist file_share, attach event
files to the inbound message, and download private URLs in receive_file
with the bot Bearer token (bytes via INBOUND_FILE_CONTENT_KEY).
Fixes#6853
* fix(slack): tighten inbound file download guards
Address review on #6855: allowlist https files.slack.com before Bearer
fetch, cap aggregate in-memory bytes per file_share message, report
Content-Encoding refusals as download failed, and share one
_web_client_for_connection helper for inbound/outbound.
* fix(slack): satisfy ruff format and AGENTS.md size gate
Reformat test_slack_file_share.py and drop the AGENTS.md file_share
note so channels/AGENTS.md stays under the 49152-byte hard limit
(file was already near the ceiling without it).
* fix(lark): retry the Windows install lock instead of failing after ~10s
msvcrt.LK_LOCK gives up after roughly ten seconds and surfaces contention
as Permission denied, while a real install (download + setup of the
lark/sandbox CLI) can hold the lock far longer. Retry the non-blocking
acquisition on the Windows fallback branch so concurrent Gateway workers
serialize instead of failing, mirroring the extensions manager's lock
introduced in #6020. The POSIX flock branch is unchanged.
Closes#6879
* test(lark): inject msvcrt stub with raising=False for Linux runners
The regression test patches the module-level ``msvcrt`` attribute, which only
exists on Windows (the POSIX branch imports ``fcntl`` instead), so
``monkeypatch.setattr`` with the default ``raising=True`` raised AttributeError
on Linux CI (backend-unit-tests shard 4).
Pass ``raising=False`` so the stub is injected on POSIX runners: ``fcntl`` is
patched to ``None`` first, and ``_exclusive_install_lock`` then takes the
Windows fallback branch against the fake. Windows behavior is unchanged.
Scope: backend/tests/test_lark_cli_install_lock.py, 1 line.
Verification: pytest tests/test_lark_cli_install_lock.py on Windows
(Python 3.12, pytest 9.0.3) -- 3 passed incl. the real-contention case;
Linux CI rerun is the authoritative check.
* fix(lark): harden and share advisory lock retries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`web_search_tool` skipped non-object entries but returned the filtered list
even when it was empty, so a non-empty `results`/`entries`/`articles`
container holding no objects fell through to "No results found". That tells
the agent the query legitimately matched nothing instead of surfacing the
malformed payload -- the failure mode the sibling providers already contain
(brave #6550, groundroute #6603, tavily #6645, sofya #6719): a 200 body
wrapped around an upstream error page ends a research loop as "no hits".
Extract `_response_rows`, which treats a missing or null container as an
empty search and a container that is not a list, or a non-empty list holding
no objects, as the provider's format error, keeping non-object entries
skipped in order so valid rows survive. Both call sites (all three verticals
share one helper) now return the existing structured error.
RED/GREEN: the 7 new cases fail on main with "No results found" and pass
with the fix; test_serply_tools.py 34 passed, the five provider suites
(serply/serper/tavily/sofya/tencent_wsa) 284 passed, ruff check and format
clean.
* fix(discord): accept inbound attachments and file-only messages
Empty content no longer drops messages that have attachments. Capture
CDN descriptors on inbound, download in receive_file via Bot auth, and
hand bytes through INBOUND_FILE_CONTENT_KEY like Slack/Telegram.
Closes#6856
* fix(discord): gate bot auth and share inbound file helpers
Address review on #6857: attach Bot Authorization only for Discord CDN
hosts, report Content-Encoding refusals as download failed, and hoist
filename/size helpers into app.channels.inbound_files (Telegram delegates
to the same module).
* fix(discord): sort telegram imports for ruff I001
CI lint-backend failed on unsorted imports in telegram.py after the
shared inbound_files helper was introduced.
* style: ruff-format discord inbound attachments test
* fix(discord): show the enforced attachment download limit
---------
Co-authored-by: Li-john1021 <Li-john1021@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM runs alive when the creating request disconnects
The channel manager holds a `runs.wait` / `runs.stream` request open for
the whole run, and that request dies with the Gateway instance that
received the IM message: a leader hand-off, a rolling restart, a crash.
The Gateway cancels a run whose creating request disconnects unless the
request asked for `on_disconnect: continue`, so every IM run in flight
was cancelled on failover.
Channel runs now ask for `continue`, like scheduled-task and MCP-task
runs already do. The run finishes on the Gateway executing it and its
thread state is persisted, so the answer stays readable in the web UI
even though the IM reply is lost with the old transport. The key is set
after the fire-and-forget branch because the SDK's `runs.create` has no
`on_disconnect` parameter.
Docs: CONFIGURATION.md and the Helm/deployment guides no longer list
"failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md
and the channels AGENTS.md describe the policy.
* docs(changelog): link #6876
* fix(channels): keep the inbound dedupe key once a run may be executing
With `on_disconnect: continue` a run outlives the manager worker that
requested it, but the worker loop released the inbound dedupe key on
cancellation, so a provider redelivery on the replacement instance
admitted the same event again and executed the turn twice (review on
#6876).
Each inbound message now carries an admission record: `requested`
flips before the run request leaves the manager, `confirmed` once the
Gateway proved it admitted the run (a run id, a stream frame, a
completed response). The generic release paths (worker cancellation,
worker/handler exceptions, a broken stream) keep the key when the run
may continue: after confirmation, on cancellation after the request
was sent, and on post-request transport failures (read timeout, read
error, protocol error). Pre-request failures, Gateway rejections
(busy thread, stale pin, status errors, connection failures) and the
stream's terminal `error` frame still release it so the redelivery
can retry.
* fix(persistence): compare safe_add_column drift as the database spells it
safe_add_column's drift check compared the model's generic type and
default spellings (String -> STRING, BigInteger -> BIGINTEGER,
server_default="0" -> "DefaultClause('0', ...)", sa.false()) with the
reflected dialect spellings (VARCHAR, BIGINT, Postgres TIMESTAMP, '0').
Only JSON/JSONB ever matched, so every column a revision found already
present was reported as drift:
- any database created before 0023 logged a runs.change_seq warning on
upgrade, because 0025 re-runs 0023's safe_add_column;
- a schema whose head stamp was lost (the empty branch commits create_all
before stamping) logged 29 warnings on SQLite and 36 on Postgres, burying
the real TEXT-vs-JSON drift the check exists to surface.
Render both sides through the migration connection's dialect before
comparing: types via TypeEngine.compile (parameters stripped, the
Postgres time-zone suffix kept), defaults by unwrapping DefaultClause,
compiling SQL expressions and stripping one layer of string quoting.
Types the dialect cannot render (SQLite NullType) stay "missing info".
* docs(changelog): reference #6865 for the drift comparator fix
* fix(persistence): keep an empty-string default distinct from no default
Stripping one layer of quotes turned a reflected DEFAULT '' into "", and
`return text or None` then collapsed it to None. A desired
server_default="" (passed as a bare "") also normalized to None, so
_check_column_drift stopped warning when a column had no default but the
model wanted '' (or the reverse).
Render a plain-string DefaultClause argument as the quoted literal the DDL
emits, and return the unquoted value even when it is empty. None now means
only "no default".
On Windows, two threads/processes racing to create a not-yet-existing
lock file with an append open can transiently fail on open() itself with
PermissionError (EACCES), before the byte-range lock is ever attempted.
The open was outside the retry loop, so the error surfaced from
load()/save() instead of being retried. Move the open inside the loop and
close the handle on any OSError before retrying.
* fix(buzz): page a saturated history instead of skipping the oldest
buzz-relay serves at most RELAY_HISTORY_LIMIT (2000) stored events per
subscription, NEWEST-FIRST, even when the filter carries a `since`. A channel
that banked more than the cap across one disconnect therefore had its OLDEST
unread messages neither sent nor marked: the live page delivered the newest
2000, the watermark advanced past them, and the rest were lost -- the one
remaining path in this connector that could skip a message outright.
A chat/backlog page's EOSE now drives a descending `until` walk. When a page
comes back AT the cap it may have older matches held back, so the connector
re-asks with `until` set to that page's oldest `created_at` -- on a SEPARATE
`buzz-backlog-<uuid>` subscription id, never the live one -- and repeats until
a page comes back under the cap or reaches the `since` the live REQ was sent
with, bounded by MAX_BACKLOG_PAGES (50, warned when hit). Delivery is charged
to the subscription that carried it before any gate, so a page's saturation is
measured against what the relay sent, not what the allowlist/mention gates kept.
The walk is strictly additive: the live subscription, its filter, and the
watermark are never touched; a CLOSED for a backlog page only ends that walk;
re-opening the live subscription closes any superseded page rather than
orphaning it; and every failure simply leaves the pre-existing behaviour for the
next reconnect to retry.
* fix(buzz): skip the backlog walk when the channel has no read floor
Reviewer willem-bd noted the descending-until walk has no semantic stop when
`floor is None`: that is exactly a channel's first connect (no stored
watermark) and again after a MAX_CACHED_CHANNELS eviction of its cursor. On
those connects the walk would fetch up to MAX_BACKLOG_PAGES * RELAY_HISTORY_LIMIT
events per channel instead of the newest 2000 the connector used to take.
Require a floor before opening any backlog page. The backlog-loss case this walk
exists for always has a watermark (the cursor survives reconnects and restarts),
so it always has a floor; first connect and an evicted cursor keep the previous
behaviour. Covers the skip with a new test, retargets the page-bound test to a
real walk above its floor, and documents the floor requirement.
* fix(buzz): stop orphaned and stalled backlog walks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): exec the gateway server so SIGTERM reaches uvicorn
The production compose command, both CMDs in backend/Dockerfile and the
Helm chart start the Gateway as `sh -c "cd backend && ... uv run ..."`.
dash forks uv and stays PID 1 without a SIGTERM handler, so the kernel
drops the stop signal: uvicorn never shuts down, the lifespan teardown
(memory flush, run drain, service stops) never runs, and every stop waits
out the grace period and ends in SIGKILL (exit 137).
exec the server in every production launch path, as
docker/dev-entrypoint.sh already does, and give the compose gateway
`init: true` so orphaned grandchildren are still reaped once sh is no
longer PID 1. A regression test pins exec in each launch path.
Fixes#6835
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the gateway SIGTERM entry to #6836
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(gateway): decouple dependency sync from exec launch form
* docs: compact startup guidance to pass inherited size limits
* docs: trim inherited startup guidance after main merge
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): feed config-upgrade inline python on stdin, not argv
On Windows setups where python/uv resolve through shim layers that
reparse arguments with cmd.exe (e.g. pyenv-win), the multi-line
`python -c "..."` payloads in config-upgrade.sh get mangled mid-flight
(observed as `IndentationError: unexpected indent` on a stray
`|| goto :error` line) and `make dev` fails before the Gateway starts.
Feed both inline scripts on stdin instead, which is safe across every
spawn chain, and load the checkout .env explicitly inside the resolver
probe: the default load_dotenv() discovery walks up from cwd/__file__,
which a stdin-fed script (`python -`) cannot satisfy. Pin both with a
static regression test.
* test(config): isolate upgrade fixtures from inherited dotenv selectors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`_parse_results` skipped malformed page entries but returned the collected
list even when it was empty, so a non-empty `Pages` container holding no
usable page fell through to "No results found". That tells the agent the
query legitimately matched nothing instead of surfacing the malformed
payload -- the failure mode the sibling providers already contain (brave
#6550, groundroute #6603, tavily #6645, sofya #6719), and the rule the
tavily changelog entry already lists `tencent_wsa` as applying.
A missing or null `Pages` stays an empty search, a non-list container keeps
the existing structured format error, and individual malformed entries in an
otherwise valid container are still skipped so the valid rows survive. Only
the non-empty-container-with-no-usable-page case now returns the provider's
format error and logs the container, never the query or payload values.
RED/GREEN: the 5 new cases fail on main with "No results found" and pass with
the fix (stash-compared source-only); test_tencent_wsa_tools.py 33 passed,
the five provider suites 281 passed, ruff check and format clean.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): append the Exit Code: 124 marker on E2B/BoxLite bash timeouts
When a bash command exceeds its timeout on the E2B or BoxLite providers,
the SDK raises (TimeoutException / TimeoutError) and the generic except
returned a bare "Error: <message>" with no exit marker; a bridge-future
timeout on BoxLite stringifies to an empty message, so the model saw just
"Error: ". Local, AIO, and Tenki all end timed-out output with the
"Exit Code: 124" marker that _truncate_bash_output preserves and the
acceptance-checklist harvest parses to recover the real shell status.
Catch the timeout exceptions ahead of the generic handler in both
providers and return the Tenki-style "Error: command timed out" +
"Exit Code: 124" ending. With #6782 forwarding bash_command_timeout to
every provider, this path is now a first-class configured behavior.
OpenSandbox routes timeouts through the same generic except, but its SDK
is an optional extra whose timeout behavior I could not verify locally,
so it is left out of this change.
Fixes#6847
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test(boxlite): clarify bridge timeout coverage and SDK failure limits
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): retry feedback and conversation upserts that lose a race
FeedbackRepository.upsert and ChannelConnectionRepository.set_thread_id
looked up the row and inserted it when missing, with no conflict
handling. Two first writers for one key both missed the lookup, and the
second INSERT violated uq_feedback_thread_run_user or
uq_channel_conversation_connection_external: a 500 for the second of two
concurrent first ratings, and a failed message when a conversation's
first message raced /new.
Both now catch IntegrityError, roll back and retry, so the losing writer
re-reads the committed row and updates it, as upsert_connection already
does. A retry rather than ON CONFLICT keeps the NULL-user feedback path
updating its single row, since a NULL key never conflicts.
* docs(changelog): reference #6852 for the upsert race fix
* test(persistence): pin upsert retry exhaustion and share the miss-select double
Both retry loops now have a test that makes every SELECT miss against a
pre-seeded row, so all attempts hit the unique key: the original
IntegrityError must propagate after exactly _UPSERT_MAX_ATTEMPTS
attempts and the stored row must stay unchanged.
The "first SELECT misses" session double lived in four copies (one per
retry test, including upsert_connection's). It now lives once in
tests/_repository_race_helpers.py, with a misses count so the same
double drives both the single lost race and exhaustion.
* feat(channels): control IM notifications and add WeChat/QQ delivery
* fix(scheduler): skip notification preference locks without recipients
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): parse Slack and DingTalk allowed_users fail-closed
Slack and DingTalk still built their allowlists with their own
_normalize_allowed_users: every entry went through str() without
stripping, only list/tuple/set counted as collections, and the gate was
`if self._allowed_users and ...`. A configured list whose entries are all
blank, such as `allowed_users: [$SLACK_USER_ID]` with the variable set to
an empty string, became an empty set and let every sender through.
" U123 " never matched U123, and a frozenset was stringified into one ID.
Parse both with the shared parse_allowed_users helper that Telegram,
WeChat and QQ use, and gate on `is not None`: unset, null, empty
collections and a blank string allow everyone; a scalar is one entry;
strings are stripped and integers keep matching their string form, as
before; booleans, floats, None and mappings are rejected (a mapping as a
whole); a configured value with no valid ID denies every sender and logs
an error. A scalar containing commas or whitespace warns that it stays
one literal ID.
* refactor(channels): share the Slack and DingTalk allowlist coercer and separator warning
Slack and DingTalk had byte-identical ID coercers and their own copies of
the scalar-separator warning. Move the coercer into allowed_users.py as
parse_string_or_int_user_id and opt both channels into
warn_on_scalar_separators, as QQ and WeChat do. Log messages are unchanged.
Pin that a separator-only string such as "," stays one literal ID
(denying everyone else) and still warns.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): merge runtime channel config saves across Gateway workers
Each Gateway worker builds its own ChannelRuntimeConfigStore, loads
runtime-config.json once, and rewrote the whole file from that copy on
every save. With two workers, a bot saved on one worker was dropped when
another bot was saved on the other, and a disconnected bot came back.
Every save, disconnect and removal now rereads the file while holding the
thread lock and an advisory sidecar file lock shared across processes,
changes only its own provider, and saves. The served snapshot is
refreshed only after a successful save, and an unreadable file still
falls back to this process's copy as before.
* docs(changelog): reference #6844 for the runtime channel config fix
* fix(channels): normalize the runtime config lock path like extensions_config
The sidecar lock was derived from the store path as given, so a worker
configured with a symlinked spelling of runtime-config.json took a
different lock file and no longer excluded peers using the real path.
Resolve a symlinked path, and make any other path absolute, before
deriving the sidecar, as extensions_config_file_lock already does.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): bound viewed image context per model request
Keep recent image payloads within a count and byte budget, and move re-viewed paths to the newest position. Add focused regressions for budget enforcement and state ordering.
* test(agents): verify cumulative image budgets and re-view recovery
* fix(agents): clarify image omission and read-attempt limits
* docs(middleware): shorten image context guidance for CI budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): restore saved thread files when creating E2B sandboxes
After an E2B sandbox expires between turns, a replacement starts without
the thread files already saved by the Gateway, so later reads fail.
Restore owner-scoped workspace, uploads, outputs and ACP files before a
new sandbox becomes available. Keep live sandbox reuse unchanged, share
the existing upload budget, and discard incomplete sandboxes on restore
failure. Skip links and unpublished temporary files during restoration.
Add offline regressions for release, expiry and recreation, sync and async
acquisition, owner isolation, live reuse, upload limits and failure cleanup.
* fix(sandbox): make thread restore failures actionable
* fix(sandbox): use neutral upload limit diagnostics
---------
Co-authored-by: conghua1013 <16472796+conghua1013@users.noreply.github.com>
* fix(subagents): release batch executions the worker stops supervising
_execute_item cleaned up its execution in a finally block, but the
executor registry only removes terminal entries. When stop() cancelled
an item task, the subagent was still running at that point: the
cancellation request only signals it, so cleanup kept the entry, and
nothing removed it once the run turned cancelled. Each in-flight item
left its SubagentResult, transcript included, in the process-wide
registry for every SDK SubagentRuntime start/stop cycle.
A repository error while polling took the same exit without even
requesting cancellation, so the item was failed or requeued while its
subagent kept running unsupervised, and a retry could run it twice.
On either exit, cancel a run that is not terminal yet and remove its
entry from the persistent subagent loop once it stops. Cancellation is
cooperative, so the wait is bounded by the item's lease, after which
another worker owns the item and the entry is removed regardless.
* docs(changelog): reference #6785
* fix(subagents): log a scheduled batch release that fails
A concurrent future never reports an exception nobody retrieved, so a
release that raised on the subagent loop leaked its registry entry with
no log trace. Log it at error level with the execution id, as the task
tool's deferred cleanup already does.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mindie): degrade unserializable tool args to str() in XML replay
The XML parser's literal_eval fallback puts values json.dumps rejects
(e.g. Python sets from set-literal arguments) into AIMessage tool_calls
args. Replaying that message from history hit the unguarded json.dumps
in _fix_messages' args join, raising TypeError and failing the whole
request — and every later turn, since the poisoned message rides the
history. Mirror the tool-result json block handling already in the same
function: degrade the value to str() so one unserializable argument
cannot fail the request.
* fix(mindie): reject non-finite floats in literal sets and tuples
* fix(mindie): validate literal dictionary keys recursively
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): surface missing-file AIO reads as actionable errors
When read_file targets a nonexistent file, the AIO server returns an error
envelope with data: {}, which the agent-sandbox SDK cannot parse into its
required FileReadResult fields. The resulting pydantic ValidationError text
became the model-visible tool result ("Error: 2 validation errors for
ResponseFileReadResult ..."), so the agent could not tell the file simply
does not exist and kept retrying or "fixing" nonexistent problems.
Catch ValidationError in AioSandbox.read_file and return an explicit
"Error: file not found or unreadable: <path>" instead. The branch is logged
at debug level: a missing file is an expected outcome, since
ReadBeforeWriteMiddleware probes every new-file write before allowing it, and
the previous error-level logging produced noisy gateway logs on that benign
path. Other exceptions keep the error log and the generic error string.
Fixesbytedance/deer-flow#6833
* docs(aio): align regression test reference style
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): tolerate null template keys in the legacy thinking disable path
`ModelConfig.when_thinking_enabled` is a plain `dict | None` with no
per-key validation, so a template key left blank in config.yaml
(`extra_body:`, `thinking:`) loads as an explicit None. The legacy
disable path read those keys with `effective_wte.get(key, {})`, whose
default only applies to an absent key — so it called `.get` on None and
every create_chat_model for the profile crashed with
AttributeError: 'NoneType' object has no attribute 'get', on the
default thinking_enabled=False path (title generation, summarization,
subagent and lead-agent builds).
Guard the three reads with the same `or {}` idiom the sibling accesses
in this module already use (`_infer_dialect`, the contract path), so a
null value falls through the dialect heuristics like an absent one and
the legacy path matches the contract path, which already tolerates the
same profile shape.
* fix(models): tolerate null nested thinking templates
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): add per-server discovery cache scaffolding
Introduce the per-server tool cache entry, canonical parsed-config snapshots on the applied revision, and the shared _binding_matches reuse predicate. Identity reuse is object-identity based on the pool's current binding (no value equality); snapshots and ownership handles stay out of repr. No selection, eviction or publication is wired yet.
* feat(mcp): selectively evict stale per-server tool cache entries
Evict per-server discovery results during selective reconciliation, after the new binding epoch is installed, so unchanged servers keep their exact entry objects while changed, removed or re-bound servers (including lifecycle-token rotations) are dropped. Delegate the binding verdict to _binding_matches so reuse and eviction cannot diverge.
Clear _server_tool_cache on every full reset, and count retained entries in the reconciliation state predicates so a lingering entry can never let a later config change be silently skipped.
* feat(mcp): reuse unchanged per-server results during initialization
Initialize MCP tools from per-server discovery instead of a single full pass: select reusable entries under the claim lock, discover only the missing servers with get_mcp_tools_by_server, then publish atomically after re-validating every candidate binding against the pool's current binding via _binding_matches. Failed servers stay absent (never cached as empty), and superseded or partially bound candidates are discarded whole. Test fixtures now drive the grouped entry point.
* fix(mcp): close T4 review gaps in per-server cache publication
Address four findings from the adversarial review: treat retained entries as unverifiable when the applied baseline is None so an interceptor change still forces a full reset; count retained entries in the claim-time shared-reset predicate; refuse to publish a false-success empty cache when every enabled server failed discovery; and hoist the lazy imports inside the initializer's try/finally so an import failure releases the initialization claim.
* fix(mcp): sanitize cache failure logging and document per-server cache
The outer lazy-initialization handler now logs only the exception type, never the raw message or traceback, so a discovery error cannot print a resolved $VAR credential. The committed-writer planner mirrors the reader's no-baseline guard so retained entries are never reused under an unknown interceptor chain. Document the per-server tool cache (reuse, invalidation, failure handling, remote tool changes) in MCP_SERVER.md and mcp/AGENTS.md, and drop phase-staging wording from the new code and tests.
* docs(mcp): keep the agent-guidance chain under its hard limit
F2's cache notes pushed the effective AGENTS.md instruction chain 905 bytes past its 98304-byte hard limit, failing the agent-guidance CI job. Compress the selective-reconciliation bullet to its invariants (the full cache behaviour lives in docs/MCP_SERVER.md), and reword the total-failure comment so it no longer implies that build-invalid connections were attempted.
* fix(mcp): fence cached tools by interceptor chain
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): stop Windows replacement uploads from being skipped by the multi-link guard
NTFS never decrements st_nlink when one of a file's names is removed, so
every upload published through os.link(staging, dest) + unlink keeps
reading st_nlink == 2 on Windows. The non-exclusive replacement guards
then reject that destination forever, and DeerFlowClient.upload_files
turns the UnsafeUploadPathError into a log line - a replacement upload
is silently skipped (Closes#6820).
- Add _replacement_destination_is_hardlinked: POSIX-authoritative
st_nlink > 1, skipped on Windows, used only on the not exclusive
replacement path. The exclusive (first-claim) path keeps the strict
count check, as does the POSIX O_NOFOLLOW branch.
- Decide "published alias" in cleanup_stale_upload_staging_files by
inode identity against siblings (st_dev/st_ino via os.lstat) instead
of the link count; DirEntry.stat reports zeros for both on Windows,
matching the existing _staging_entry_stat note.
- Guard the POSIX-only st_nlink assertions in the affected tests and
mark the three inbound-attachment mode-bit tests POSIX-only, mirroring
the existing skipif in the same class.
* fix(uploads): restore the exclusive post-open link guard and harden the staging sweep
Address the three review findings on 5353715:
- Restore the strict post-open st_nlink != 1 check for exclusive=True
on the no-O_NOFOLLOW (Windows fallback) branch, which the previous
commit dropped there against what the PR description claimed. Only
the non-exclusive replacement path defers to the Windows-aware guard.
- Never match a degenerate st_ino == 0 in the sibling-inode sweep: on
FAT/exFAT or some SMB volumes every entry would look like a published
alias and in-flight uploads would be deleted at any age. Such volumes
fall back to the min_age guard only.
- Build one {name: stat} map per uploads directory before the entry
loop so the sweep scans each directory once instead of rescanning it
for every staging file (O(N) instead of O(N^2) lstat calls).
Tests: 407 passed / 27 skipped across test_uploads_manager.py,
test_uploads_router.py, test_upload_copy_publication.py,
test_upload_windows_staging_aliases.py, test_channel_file_attachments.py
(Windows 11, Python 3.13 venv). The three new regression tests fail on
5353715 and pass with this change.
* style(uploads): apply Ruff formatting
* fix(uploads): report staging directory scan failures
---------
Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): leave full results out of the polled batch item list
The item list endpoint returns only previews, but list_items selected
whole rows, so every page loaded and decoded each item's full report,
evidence JSON and prompt before discarding them. The workspace polls
this list every 2 seconds while a batch runs; with the default
max_result_chars a 100-item page held about 34 MB and blocked the event
loop for about 70 ms (170 MB and 300 ms at limit=500).
Defer those columns unless the caller asks for them, with raiseload so
an accidental read fails loudly instead of lazy-loading. A 100-item page
now peaks under 1 MB; responses are unchanged.
* test(subagents): pin the item list route to compact columns
Drive list_batch_items against a real SQLite repository so a route that
starts requesting full results fails, not only a repository change.
* docs(changelog): reference #6781
* test(subagents): scope the route test's seed update to its batch
