* fix(searxng): walk pageno so max_results above one page is honored
The SearXNG search API has no `limit` parameter: `/search` answers with one
page of results (the instance's `results_per_page`, 10 by default) and
ignores a limit it is handed. The client sent `limit=max_results` anyway and
hardcoded `pageno=1`, so any configured `max_results` larger than a page was
silently truncated to whatever the first page held -- an instance could not
tell a truncated response from a complete one.
Collect results by walking `pageno` until `max_results` is reached, a page
adds nothing new, or a page comes back empty. The walk is capped at
`_MAX_PAGES` so an unexpectedly large `max_results` cannot fan out into
unbounded requests. The unsupported `limit` parameter is no longer sent.
Adds four regression tests covering cross-page collection, the single-request
fast path, dedup-driven early stop, and the absence of `limit`.
* Remove redundant condition for limit check
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(community): add SearXNG and Browserless web search/fetch tools
- SearXNG web_search: privacy-focused meta search engine integration
with configurable base_url via config.yaml tool settings
- Browserless web_fetch: headless browser page fetching with
readability article extraction
- Both tools are fully configurable through tool config section
- No external API keys required for basic operation
* fix: address PR review feedback and add unit tests
- Guard config.model_extra against None values (review #1, #2)
- Coerce max_results to int when reading from config (review #2)
- Fix web_fetch_tool to use direct HTTP fetch instead of reusing
the web_search client config (review #3)
- Fix misleading docstring for SearxngClient.fetch (review #4)
- Remove unused target_url variable to pass Ruff lint (review #5)
- Normalize bool config values with _normalize_bool helper to
handle env-resolved string values correctly (review #6)
- Add unit tests for both SearXNG and Browserless client classes
and their tool functions with mocked httpx (review #7, #8)
* fix: convert to async httpx to avoid blocking I/O on event loop
- Replace httpx.Client with httpx.AsyncClient in both client classes
- Convert tool functions to async def
- Wrap readability_extractor calls in asyncio.to_thread()
- Update all tests to use pytest.mark.asyncio and async mocks
- Fix import sorting to pass Ruff lint
* fix(browserless): replace deprecated waitUntil with waitForEvent
The Browserless API has deprecated the waitUntil parameter.
Replace with waitForEvent which accepts values like 'networkidle'.
Default is empty (no wait), configurable via config.yaml.
* fix(browserless): remove deprecated gotoTimeout and bestAttempt params
The Browserless /content API does not accept gotoTimeout or bestAttempt
as top-level payload keys. These were being sent unconditionally,
causing 400 Bad Request errors on current Browserless versions.
Changes:
- Remove goto_timeout_ms parameter and 'gotoTimeout' from payload
- Remove best_attempt parameter and 'bestAttempt' from payload
- Remove _normalize_bool helper (no longer needed)
- Remove goto_timeout_ms and best_attempt config reading in tools.py
- Add tests for waitForSelector and reject params
- Verify no deprecated params are sent in test_fetch_html_success
* refactor(searxng): remove web_fetch_tool, decouple from web_search config
SearXNG is a search engine — it should only provide web_search_tool.
The web_fetch responsibility belongs to Browserless (headless Chrome)
or Jina AI, not SearXNG.
Changes:
- Remove web_fetch_tool from SearXNG tools.py and __init__.py
- Remove SearxngClient.fetch() method (no longer needed)
- Remove unused asyncio/readability imports from SearXNG tools.py
- Add test for max_results string-to-int coercion from config
- Add test for search with categories parameter
- Add test for httpx.RequestError handling
- Apply ruff format fixes to browserless_client.py and test files
