* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): send the web_fetch token as a query parameter
fetch_html_with_status put the configured token in the /content JSON body.
Browserless reads the token only from the token query parameter or the
Authorization header, and checks it before reading the body, so with
BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an
instance started without TOKEN the unknown body key failed schema
validation with HTTP 400. capture_screenshot already sent ?token=, so
web_capture worked with the same token.
Both endpoints now take the token from BrowserlessClient._auth_params().
The test that pinned the body token is rewritten, and a stand-in for the
Browserless request gate (query/header auth, no extra body keys) drives
both endpoints through the real httpx request path.
* docs(changelog): reference #6484
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browserless): read reject_resource_types / reject_request_pattern from config
web_fetch_tool declared both parameters, initialised them to None, and passed
them to the client -- but never read them from tool config, so they were always
None on the wire. BrowserlessClient implements them (and is tested for it at
`test_fetch_html_with_reject_params`), so the provider's resource blocking was
unreachable without patching the source. Skipping images/media/fonts is the
main lever for render latency and bandwidth against a self-hosted Browserless,
and it could not be turned on from config.yaml.
Read both keys through a new `_as_str_list` helper that accepts a YAML list or
a comma-separated string and yields None for anything unusable, so an unparseable
value omits the parameter instead of reaching the server. Both keys are now
documented on the Browserless web_fetch entry in config.example.yaml.
Default behaviour is unchanged: an unset key is still None, so the payload is
identical to before for every existing config.
Adds four regression tests covering the list form, the comma-separated form,
the unset case, and unusable values.
* fix(browserless): validate reject list entries
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
web_fetch_tool parsed `wait_for_timeout_ms` with a bare int(), so any value
that is not already an int raised ValueError and aborted the whole tool:
a quoted number in YAML, or a typo like `2s`, turned every fetch into
"Error: invalid literal for int() with base 10". web_capture_tool reads the
same documented key through the tolerant `_as_int` helper two functions
above, so the two tools disagreed about the same config key.
While aligning them, web_fetch_tool now also reads
`wait_for_selector_timeout_ms` instead of hardcoding 5000, so a
`wait_for_selector` on a slow page can be given more than five seconds --
web_capture_tool already honours the key and the client has always
accepted it.
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
`browserless` reads `cfg["timeout_s"]`, while its sibling web providers
`crawl4ai` and `jina_ai` read `cfg["timeout"]`. Tool configs allow extra
fields, so the unrecognised spelling is dropped without a diagnostic: someone
adapting one provider's config snippet for another silently gets the 30s
default instead of the timeout they set. (Observed in the other direction, on a
deployment whose crawl4ai entry carried `timeout_s`.)
Accept both keys, preferring the documented `timeout_s` when both are present.
While adding coverage, two pre-existing bugs in the same three lines surfaced,
both already guarded in crawl4ai/jina_ai but not here:
- `timeout_s: "30s"` (or any non-numeric string) raised ValueError out of
`float(raw)` during tool construction rather than falling back.
- `timeout_s: off` -- YAML parses that as `False`, and `float(False)` is
`0.0`, so every request timed out immediately against a healthy server.
`_coerce_timeout` now mirrors the sibling providers: booleans and unparsable
strings fall back to the default, with a warning for the string case.
Tests: five cases in tests/test_browserless_client.py covering both keys, the
precedence order, and both coercion bugs. Verified red before the fix (3 of 5
fail) and green after.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(browserless): surface target-page error status in web_fetch_tool
Browserless returns HTTP 200 for the render request itself even when the
target page responded with a 4xx/5xx or served an anti-bot block page,
tagging the real outcome on X-Response-Code/X-Response-Status headers.
capture_screenshot/web_capture_tool already reads these headers and
surfaces a warning via _target_status_warning. fetch_html only logged
them at debug level and web_fetch_tool returned the block/error page's
raw text as if it were a normal successful fetch, with no indication
anything was wrong.
fetch_html now returns a BrowserlessFetchResult carrying the rendered
HTML plus the target-status headers (mirroring BrowserlessScreenshotResult),
and web_fetch_tool appends the same _target_status_warning used by
web_capture_tool when the target page errored. Legitimate 200-target
fetches are unaffected.
* fix(browserless): keep fetch_html() returning a plain string
BrowserlessClient is re-exported from deerflow.community.browserless.__all__,
so fetch_html() is public harness API with an established str-only contract:
the rendered HTML on success, or an "Error: ..." string on failure. Changing
its return type to BrowserlessFetchResult broke that contract for any caller
that treats the result as a string (.lower(), concatenation, passing to a
parser), even when the fetch itself succeeded.
fetch_html() is now a thin wrapper that always unwraps back to the original
str contract. The richer, status-aware result (needed to tell a genuine 200
apart from a render-succeeded-but-target-errored response) moves to a new
fetch_html_with_status() method, which web_fetch_tool now calls instead so it
keeps surfacing the target-page-error warning.
Tests: retarget the tool-level mocks onto fetch_html_with_status, add a direct
regression asserting fetch_html() returns str on success - including when the
target page itself errored under a 200 render response - and keep the
status-aware coverage on the new method.
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
* feat(community): add SearXNG and Browserless web search/fetch tools
- SearXNG web_search: privacy-focused meta search engine integration
with configurable base_url via config.yaml tool settings
- Browserless web_fetch: headless browser page fetching with
readability article extraction
- Both tools are fully configurable through tool config section
- No external API keys required for basic operation
* fix: address PR review feedback and add unit tests
- Guard config.model_extra against None values (review #1, #2)
- Coerce max_results to int when reading from config (review #2)
- Fix web_fetch_tool to use direct HTTP fetch instead of reusing
the web_search client config (review #3)
- Fix misleading docstring for SearxngClient.fetch (review #4)
- Remove unused target_url variable to pass Ruff lint (review #5)
- Normalize bool config values with _normalize_bool helper to
handle env-resolved string values correctly (review #6)
- Add unit tests for both SearXNG and Browserless client classes
and their tool functions with mocked httpx (review #7, #8)
* fix: convert to async httpx to avoid blocking I/O on event loop
- Replace httpx.Client with httpx.AsyncClient in both client classes
- Convert tool functions to async def
- Wrap readability_extractor calls in asyncio.to_thread()
- Update all tests to use pytest.mark.asyncio and async mocks
- Fix import sorting to pass Ruff lint
* fix(browserless): replace deprecated waitUntil with waitForEvent
The Browserless API has deprecated the waitUntil parameter.
Replace with waitForEvent which accepts values like 'networkidle'.
Default is empty (no wait), configurable via config.yaml.
* fix(browserless): remove deprecated gotoTimeout and bestAttempt params
The Browserless /content API does not accept gotoTimeout or bestAttempt
as top-level payload keys. These were being sent unconditionally,
causing 400 Bad Request errors on current Browserless versions.
Changes:
- Remove goto_timeout_ms parameter and 'gotoTimeout' from payload
- Remove best_attempt parameter and 'bestAttempt' from payload
- Remove _normalize_bool helper (no longer needed)
- Remove goto_timeout_ms and best_attempt config reading in tools.py
- Add tests for waitForSelector and reject params
- Verify no deprecated params are sent in test_fetch_html_success
* refactor(searxng): remove web_fetch_tool, decouple from web_search config
SearXNG is a search engine — it should only provide web_search_tool.
The web_fetch responsibility belongs to Browserless (headless Chrome)
or Jina AI, not SearXNG.
Changes:
- Remove web_fetch_tool from SearXNG tools.py and __init__.py
- Remove SearxngClient.fetch() method (no longer needed)
- Remove unused asyncio/readability imports from SearXNG tools.py
- Add test for max_results string-to-int coercion from config
- Add test for search with categories parameter
- Add test for httpx.RequestError handling
- Apply ruff format fixes to browserless_client.py and test files
