* fix(browserless): construct the fetch and screenshot clients off the event loop
* fix(browserless): close the client a cancelled construction still builds
Keeping the constructor off the loop is right, but awaiting
`asyncio.to_thread(httpx.AsyncClient, ...)` directly is not cancellation-safe:
the await can be cancelled while the worker thread is still inside the
constructor. The await is abandoned, the thread runs on and hands back a live
client, and execution never enters the `async with` that would have closed it,
so `is_closed` stays False and the connection pool leaks on that cancellation.
Own the construction as a task instead. When the caller is cancelled, wait for
the task with the shield loop this package already uses for cleanup, close
whatever client it produced, and only then let the cancellation propagate. A
failure while closing is logged rather than raised, so it cannot mask the
cancellation the caller asked for.
The new regression test gates the constructor on a thread event, cancels the
operation while it is parked there, and releases the worker afterwards. Against
the previous commit it fails for both fetch_html_with_status and
capture_screenshot with "the client built after the cancellation was leaked".
---------
Co-authored-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
* fix(browserless): send the web_fetch token as a query parameter
fetch_html_with_status put the configured token in the /content JSON body.
Browserless reads the token only from the token query parameter or the
Authorization header, and checks it before reading the body, so with
BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an
instance started without TOKEN the unknown body key failed schema
validation with HTTP 400. capture_screenshot already sent ?token=, so
web_capture worked with the same token.
Both endpoints now take the token from BrowserlessClient._auth_params().
The test that pinned the body token is rewritten, and a stand-in for the
Browserless request gate (query/header auth, no extra body keys) drives
both endpoints through the real httpx request path.
* docs(changelog): reference #6484
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): surface target-page error status in web_fetch_tool
Browserless returns HTTP 200 for the render request itself even when the
target page responded with a 4xx/5xx or served an anti-bot block page,
tagging the real outcome on X-Response-Code/X-Response-Status headers.
capture_screenshot/web_capture_tool already reads these headers and
surfaces a warning via _target_status_warning. fetch_html only logged
them at debug level and web_fetch_tool returned the block/error page's
raw text as if it were a normal successful fetch, with no indication
anything was wrong.
fetch_html now returns a BrowserlessFetchResult carrying the rendered
HTML plus the target-status headers (mirroring BrowserlessScreenshotResult),
and web_fetch_tool appends the same _target_status_warning used by
web_capture_tool when the target page errored. Legitimate 200-target
fetches are unaffected.
* fix(browserless): keep fetch_html() returning a plain string
BrowserlessClient is re-exported from deerflow.community.browserless.__all__,
so fetch_html() is public harness API with an established str-only contract:
the rendered HTML on success, or an "Error: ..." string on failure. Changing
its return type to BrowserlessFetchResult broke that contract for any caller
that treats the result as a string (.lower(), concatenation, passing to a
parser), even when the fetch itself succeeded.
fetch_html() is now a thin wrapper that always unwraps back to the original
str contract. The richer, status-aware result (needed to tell a genuine 200
apart from a render-succeeded-but-target-errored response) moves to a new
fetch_html_with_status() method, which web_fetch_tool now calls instead so it
keeps surfacing the target-page-error warning.
Tests: retarget the tool-level mocks onto fetch_html_with_status, add a direct
regression asserting fetch_html() returns str on success - including when the
target page itself errored under a 200 render response - and keep the
status-aware coverage on the new method.
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
* feat(community): add SearXNG and Browserless web search/fetch tools
- SearXNG web_search: privacy-focused meta search engine integration
with configurable base_url via config.yaml tool settings
- Browserless web_fetch: headless browser page fetching with
readability article extraction
- Both tools are fully configurable through tool config section
- No external API keys required for basic operation
* fix: address PR review feedback and add unit tests
- Guard config.model_extra against None values (review #1, #2)
- Coerce max_results to int when reading from config (review #2)
- Fix web_fetch_tool to use direct HTTP fetch instead of reusing
the web_search client config (review #3)
- Fix misleading docstring for SearxngClient.fetch (review #4)
- Remove unused target_url variable to pass Ruff lint (review #5)
- Normalize bool config values with _normalize_bool helper to
handle env-resolved string values correctly (review #6)
- Add unit tests for both SearXNG and Browserless client classes
and their tool functions with mocked httpx (review #7, #8)
* fix: convert to async httpx to avoid blocking I/O on event loop
- Replace httpx.Client with httpx.AsyncClient in both client classes
- Convert tool functions to async def
- Wrap readability_extractor calls in asyncio.to_thread()
- Update all tests to use pytest.mark.asyncio and async mocks
- Fix import sorting to pass Ruff lint
* fix(browserless): replace deprecated waitUntil with waitForEvent
The Browserless API has deprecated the waitUntil parameter.
Replace with waitForEvent which accepts values like 'networkidle'.
Default is empty (no wait), configurable via config.yaml.
* fix(browserless): remove deprecated gotoTimeout and bestAttempt params
The Browserless /content API does not accept gotoTimeout or bestAttempt
as top-level payload keys. These were being sent unconditionally,
causing 400 Bad Request errors on current Browserless versions.
Changes:
- Remove goto_timeout_ms parameter and 'gotoTimeout' from payload
- Remove best_attempt parameter and 'bestAttempt' from payload
- Remove _normalize_bool helper (no longer needed)
- Remove goto_timeout_ms and best_attempt config reading in tools.py
- Add tests for waitForSelector and reject params
- Verify no deprecated params are sent in test_fetch_html_success
* refactor(searxng): remove web_fetch_tool, decouple from web_search config
SearXNG is a search engine — it should only provide web_search_tool.
The web_fetch responsibility belongs to Browserless (headless Chrome)
or Jina AI, not SearXNG.
Changes:
- Remove web_fetch_tool from SearXNG tools.py and __init__.py
- Remove SearxngClient.fetch() method (no longer needed)
- Remove unused asyncio/readability imports from SearXNG tools.py
- Add test for max_results string-to-int coercion from config
- Add test for search with categories parameter
- Add test for httpx.RequestError handling
- Apply ruff format fixes to browserless_client.py and test files