* fix(readability): preserve <a> links when Readability.js silently degrades on Windows
On Windows, simple_json_from_html_string(use_readability=True) invokes the
Readability.js subprocess, which succeeds without error but returns the
unextracted document instead of the extracted article. The pure-Python
fallback (use_readability=False) also drops every <a> destination. The net
effect is that web_fetch returns link-free Markdown on Windows.
Fix: after extraction, check whether the output retained the <a> elements
that the resolved HTML carried. When they are all gone, fall back to
converting the resolved HTML directly with markdownify — which preserves
<a> as [text](href) — so links survive regardless of the Readability.js
outcome.
Measured: 17 of 22 test_web_fetch_relative_links.py failures resolved;
43/48 pass on Windows (5 remaining are pre-existing malformed-markup
extraction tests, unrelated to this change).
Closes#6459
* test(readability): add degraded-extraction and scope-pinning tests
* test(readability): detect unwanted article fallback
* test(readability): classify HTML body in authority-tag census
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate infoquest client to httpx.AsyncClient
Replace blocking sync requests.post calls with httpx.AsyncClient in the
InfoQuest community tool, as tracked in backend/docs/TODO.md planned
features. fetch/web_search/image_search and their raw-results helpers are
now async, keeping the 30s transport timeout and external behavior
(return values, error strings) unchanged.
The langchain tool wrappers in infoquest/tools.py become async functions;
web_fetch offloads readability extraction to a thread (same pattern as
the already-migrated jina_ai tool). Tests are updated to mock
httpx.AsyncClient instead of requests.post.
* test(community): use AsyncMock for infoquest fetch in relative-links test
The infoquest web_fetch_tool now awaits client.fetch, so the test stub
must return an awaitable instead of a sync lambda.
* fix(community): follow InfoQuest API redirects like the sync client did
Address the review note on #5782: requests.post follows redirects by
default while httpx.AsyncClient defaults to follow_redirects=False, so
a 3xx answer from the reader/search endpoints would have surfaced as
'Error: fetch API returned status 30x' (or fallen into the generic
error path after raise_for_status) instead of the final content.
All three AsyncClient call sites now pass follow_redirects=True. Adds
regression tests asserting each client is constructed with
follow_redirects=True, and documents the parity in the test contract.
* test(community): fix InfoQuest mock endpoint typo
* fix(web-fetch): resolve relative URLs in extracted Markdown
Pass the request URL through Jina and Browserless extraction and resolve
link/image destinations before Readability removes document base tags.
Preserve the optional legacy API and fallback text behavior.
Fixes#5307
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): address provider and base URL review feedback
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): preserve HTML source when resolving destinations
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
