5 Commits

Author SHA1 Message Date
kongxin
cee54cdd49
fix(web-fetch): stop sending .env.example placeholder keys to providers (#8206)
* fix(web-fetch): stop sending .env.example placeholder keys to providers

make config and make setup copy .env.example into .env, and the template
set nine search/fetch provider keys to example values such as
JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent
that text as a Bearer key, so every web_fetch on a fresh install failed
with 401 while make doctor reported web fetch as configured.

- .env.example comments those nine lines out, like every other credential
  in the template already is.
- provider_credentials.resolve_api_key reads a your-...-key/-token/-secret
  value as unset, so existing .env files that still hold the placeholders
  recover without editing, for every provider that resolves keys there.

* chore: leave the changelog entry to merge time

Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they
merge a pull request.

* fix(web-search): route every provider key and doctor through one placeholder rule

Review follow-up for #8206:

- provider_credentials exposes is_template_placeholder(), the single
  definition of an .env.example example value. key/token/secret must be
  a whole hyphen-separated word, so a real value such as your-monkey or
  your-passkey is no longer dropped.
- Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA
  and You.com resolve config and environment keys through
  resolve_api_key instead of private strip-and-send copies, so an old
  .env that still holds the template value no longer sends it.
- make doctor asks the same predicate: a placeholder key is reported as
  still holding the .env.example value instead of "set", and a literal
  placeholder in config.yaml falls back to the environment key as the
  runtime does.
- A census test keeps credential environment reads under community/
  wrapped in resolve_api_key.

* fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor

Second review follow-up for #8206:

- is_template_placeholder() matches exactly the your-... values that
  .env.example ships instead of a pattern, so a token an operator chose
  (a self-hosted Browserless TOKEN such as your-review-team-token) is
  sent again. A test keeps the set equal to the template.
- make doctor mirrors Browserless: a configured token field is
  authoritative, blank or not, and BROWSERLESS_TOKEN is read only when
  the field is omitted. A blank or placeholder token now warns instead
  of reporting the variable, cross-checked against the client.
- The keyless Jina and You.com providers stay ok but name JINA_API_KEY
  or YDC_API_KEY when a placeholder there is ignored, and a literal
  placeholder api_key in config.yaml is named when no env key backs it.

* fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs

Found while re-checking #8206 against every provider's runtime:

- Browserless used as web_fetch had no doctor key check; it now gets the
  same authoritative token handling as web_capture.
- InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an
  api_key field in its tool config as configured.
- GroundRoute needs a key for every call but had no doctor entry, so a
  missing key reported ok.
- You.com names a placeholder api_key in config that it ignores.
- The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which
  the tool never reads. With JINA_API_KEY now commented out in
  .env.example, that line made config loading fail on a fresh install.
- The provider_credentials docstring now scopes doctor's use of the
  predicate to web provider keys, and the scripts guide no longer says
  .env.example ships live placeholder keys.
2026-11-13 23:23:29 +08:00
Younes Beriane
1794c40888
fix(community): screen web_search result URLs in every provider (#7037)
* fix(community): screen tavily and exa web_search result URLs

The Tavily and Exa web_search tools returned provider URLs to the model
unfiltered, so a loopback, link-local or metadata-service address in a
result reached the agent. Brave and Serper already pass result URLs
through safe_public_result_url; do the same here, blanking a non-public
URL while keeping the row's title and snippet.

* fix(community): screen result URLs in the remaining web_search providers

youcom, tencent_wsa, serply (url and scholar pdf_url), groundroute, sofya,
ddg_search, fastcrw, firecrawl and searxng handed provider-supplied URLs
to the model unfiltered. Route each through safe_public_result_url, as
brave, serper, tavily and exa do: a non-public URL is blanked and the
row's title and snippet are kept.

The test now drives every listed provider's web_search tool with a
stubbed provider payload carrying a metadata, loopback or public URL.

* fix(community): screen brave, serper and webz web_search result URLs

The remaining text-search providers still returned result URLs verbatim.
Serper's comment documented that policy; it now screens organic links like
its image results. The shared provider test covers all fourteen providers
and asserts the snippet survives screening.

* docs(tests): describe text-search URL screening coverage

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-22 09:23:31 +08:00
D41910
62588f3675
fix(sofya): report a format error for a results list with no usable objects (#6719)
* fix(sofya): report a format error for results with no usable objects

`_response_results` skipped non-object entries but returned the filtered
list even when it was empty, so a non-empty `results` array holding no
objects fell through to "No results found" -- telling the agent the query
legitimately matched nothing instead of surfacing the malformed payload.

Treat a non-empty list with no object entries as malformed, matching the
containment rule #6603 documents for this provider family (brave #6550,
groundroute #6603, tavily #6645, webz). Both `web_search` and
`web_fetch` share this helper, so both now report the provider's format
error. Missing, null, and empty `results` keep returning "No results
found", and mixed lists still keep their valid entries in order.

* test(sofya): cover mixed fetch result entries

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-12 21:00:24 +08:00
yetuge
5092a45943
fix(tools): make the search coercion guards catch OverflowError (#5957)
* fix(tools): make the search coercion guards catch OverflowError

The community search tools coerce max_results (and Sofya's content
limit) from YAML config or tool parameters through guards that catch
only TypeError and ValueError. `int(float('inf'))` raises OverflowError
instead, so a YAML `.inf` (or a JSON `Infinity`) value still crashes
the coercion - SearXNG's search even fails wholesale with
'cannot convert float infinity to integer'.

Catch OverflowError alongside them so an infinite value falls back to
the default like every other invalid input. NaN is already covered by
ValueError.

Signed-off-by: yetuge <2219677952@qq.com>

* test(tools): move the GroundRoute inf test to module scope

The regression test was appended inside TestWebSearchTool, ending the
class early and turning its last two methods into uncollected nested
functions. Move it after TestWebFetchTool so pytest collects all 20
tests again.

Signed-off-by: yetuge <2219677952@qq.com>

---------

Signed-off-by: yetuge <2219677952@qq.com>
2026-09-28 14:35:42 +08:00
Yusuf Gürdoğan
23bd76046a
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider

Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).

Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
  Implements web_search_tool and web_fetch_tool using httpx.
  API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
  env var. Follows the same interface and output shape as the existing
  ddg_search and serper providers, including the max_results parameter with
  config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
  Unit tests covering API key resolution, config overrides, result mapping,
  time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
  examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
  the environment variables

* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test

- Caller-supplied max_results now wins; config is used only when the
  argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
  warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.

* fix(sofya): cap per-result content so a search stays inline

An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): list Sofya in the recency contract, coerce non-string content

_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog

web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".

Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs

The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 10:50:03 +08:00