* fix(searxng): contain malformed search payloads instead of crashing on them
`_search_page` indexed the decoded body directly:
data = resp.json()
return data.get("results") or []
so a self-hosted instance that answers HTTP 200 with something other than a
result page -- an HTML or plain-text error page from a reverse proxy or captive
portal -- reached the model as a raw Python error, or as an empty search:
{"results": "maintenance window"} -> {"error": "'str' object has no attribute 'get'"}
{"results": {"a": 1}} -> {"error": "'str' object has no attribute 'get'"}
{"results": [1, 2, 3]} -> {"error": "'int' object has no attribute 'get'"}
{"results": [None]} -> {"error": "'NoneType' object has no attribute 'get'"}
Worse, a single malformed entry in an otherwise valid page crashed the whole
call and discarded every good row:
{"results": [{"title": "ok", ...}, 7]} -> {"error": "'int' object has no attribute 'get'"}
This is the containment rule the sibling providers already follow (brave #6550,
groundroute #6603, tavily #6645, sofya #6719, serper #6779, serply #6796,
tencent-wsa #6799): a payload that is not a result page is a format error, not an
empty result, because "No results found" tells the agent the query legitimately
matched nothing and ends a research loop as "no hits".
The client now raises `SearxngResponseError` for a payload that is not an object,
a body that is not JSON, a `results` container that is not a list, and a
non-empty list holding no object entries, while still skipping non-object entries
in an otherwise valid page so the valid rows survive. `web_search_tool` reports
the provider's format error instead of leaking the exception text. Missing, null,
and empty `results` keep the existing "No results found" behavior.
Diagnostics log the container field name and its type, or the absence of usable
objects -- never the query, credentials, or payload values.
* test(searxng): pin the mid-walk format-error trade-off and fix the docstring
The paginating client aborts the walk and discards already-collected rows when
a later page comes back malformed, which is a different outcome from the
single-request siblings. Pin that trade-off with a test so the behaviour is a
deliberate decision rather than an accident, and reword the
_unexpected_format_error docstring: it promises the query is not echoed, but
the sibling providers all echo it back on purpose and only withhold the
payload.
* test(searxng): log skipped entries and pin the dedicated format-error handler
Two follow-ups from review, both about making the family contract observable
and pinned:
- Skipped non-object entries now log their index ("Skipping malformed SearXNG
result entry at results[1]"), matching the merged InfoQuest sibling. SearXNG
is the provider where drift is most expected -- a self-hosted instance serves
whatever sits behind its reverse proxy -- and a search spans up to
_MAX_PAGES requests, so rows could otherwise be shed with no trace at all.
- The dedicated `except SearxngResponseError` branch had no test that could
fail: the exception is a ValueError whose message is already the format-error
string, so the generic handler returns byte-identical JSON. The new test
asserts the generic branch's log line is absent, which is what actually
distinguishes the two paths.
* docs(searxng): name the empty result it actually returns, and pin the dedicated except
Two low findings from the follow-up review:
- The README said missing/null/empty `results` keep the existing "No results
found" response. That response does not exist in this provider: all three
produce a JSON empty list, and the string appears only in comments
explaining what must NOT be reported. The family convention distinguishes
(Tavily: "`[]` for search, Error: No results found for fetch"; InfoQuest:
"keep the existing empty result"), so this now reads "keep the existing
empty result (`[]`)".
- The client-level `except SearxngResponseError: raise` was unpinned: deleting
it left the whole file green, because the tool-level test mocks the client
and the payload tests only assert the raise type. The observable regression
is the generic branch logging "An unexpected error occurred" for every
contained page, so the existing payload test now asserts that line is absent
from caplog.
Mutation check: removing the clause now fails 8 parameterised cases (was 33
green); with the clause in place the suite is 33 passed and 91 across the two
provider files, ruff check/format clean.
* fix(tools): make the search coercion guards catch OverflowError
The community search tools coerce max_results (and Sofya's content
limit) from YAML config or tool parameters through guards that catch
only TypeError and ValueError. `int(float('inf'))` raises OverflowError
instead, so a YAML `.inf` (or a JSON `Infinity`) value still crashes
the coercion - SearXNG's search even fails wholesale with
'cannot convert float infinity to integer'.
Catch OverflowError alongside them so an infinite value falls back to
the default like every other invalid input. NaN is already covered by
ValueError.
Signed-off-by: yetuge <2219677952@qq.com>
* test(tools): move the GroundRoute inf test to module scope
The regression test was appended inside TestWebSearchTool, ending the
class early and turning its last two methods into uncollected nested
functions. Move it after TestWebFetchTool so pytest collects all 20
tests again.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(community): fall back to the default SearXNG max_results on an unparseable value
A blank `max_results:` in config.yaml loads as None, and int(None) then fails inside web_search_tool's try block, so every SearXNG search returns an error payload instead of results. Coerce like the sibling providers do: warn and keep the documented default.
* test(community): cover SearXNG max_results that cannot be parsed
* fix(searxng): walk pageno so max_results above one page is honored
The SearXNG search API has no `limit` parameter: `/search` answers with one
page of results (the instance's `results_per_page`, 10 by default) and
ignores a limit it is handed. The client sent `limit=max_results` anyway and
hardcoded `pageno=1`, so any configured `max_results` larger than a page was
silently truncated to whatever the first page held -- an instance could not
tell a truncated response from a complete one.
Collect results by walking `pageno` until `max_results` is reached, a page
adds nothing new, or a page comes back empty. The walk is capped at
`_MAX_PAGES` so an unexpectedly large `max_results` cannot fan out into
unbounded requests. The unsupported `limit` parameter is no longer sent.
Adds four regression tests covering cross-page collection, the single-request
fast path, dedup-driven early stop, and the absence of `limit`.
* Remove redundant condition for limit check
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(community): add SearXNG and Browserless web search/fetch tools
- SearXNG web_search: privacy-focused meta search engine integration
with configurable base_url via config.yaml tool settings
- Browserless web_fetch: headless browser page fetching with
readability article extraction
- Both tools are fully configurable through tool config section
- No external API keys required for basic operation
* fix: address PR review feedback and add unit tests
- Guard config.model_extra against None values (review #1, #2)
- Coerce max_results to int when reading from config (review #2)
- Fix web_fetch_tool to use direct HTTP fetch instead of reusing
the web_search client config (review #3)
- Fix misleading docstring for SearxngClient.fetch (review #4)
- Remove unused target_url variable to pass Ruff lint (review #5)
- Normalize bool config values with _normalize_bool helper to
handle env-resolved string values correctly (review #6)
- Add unit tests for both SearXNG and Browserless client classes
and their tool functions with mocked httpx (review #7, #8)
* fix: convert to async httpx to avoid blocking I/O on event loop
- Replace httpx.Client with httpx.AsyncClient in both client classes
- Convert tool functions to async def
- Wrap readability_extractor calls in asyncio.to_thread()
- Update all tests to use pytest.mark.asyncio and async mocks
- Fix import sorting to pass Ruff lint
* fix(browserless): replace deprecated waitUntil with waitForEvent
The Browserless API has deprecated the waitUntil parameter.
Replace with waitForEvent which accepts values like 'networkidle'.
Default is empty (no wait), configurable via config.yaml.
* fix(browserless): remove deprecated gotoTimeout and bestAttempt params
The Browserless /content API does not accept gotoTimeout or bestAttempt
as top-level payload keys. These were being sent unconditionally,
causing 400 Bad Request errors on current Browserless versions.
Changes:
- Remove goto_timeout_ms parameter and 'gotoTimeout' from payload
- Remove best_attempt parameter and 'bestAttempt' from payload
- Remove _normalize_bool helper (no longer needed)
- Remove goto_timeout_ms and best_attempt config reading in tools.py
- Add tests for waitForSelector and reject params
- Verify no deprecated params are sent in test_fetch_html_success
* refactor(searxng): remove web_fetch_tool, decouple from web_search config
SearXNG is a search engine — it should only provide web_search_tool.
The web_fetch responsibility belongs to Browserless (headless Chrome)
or Jina AI, not SearXNG.
Changes:
- Remove web_fetch_tool from SearXNG tools.py and __init__.py
- Remove SearxngClient.fetch() method (no longer needed)
- Remove unused asyncio/readability imports from SearXNG tools.py
- Add test for max_results string-to-int coercion from config
- Add test for search with categories parameter
- Add test for httpx.RequestError handling
- Apply ruff format fixes to browserless_client.py and test files
