mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-20 03:10:34 +00:00
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
cee54cdd49
|
fix(web-fetch): stop sending .env.example placeholder keys to providers (#8206)
* fix(web-fetch): stop sending .env.example placeholder keys to providers make config and make setup copy .env.example into .env, and the template set nine search/fetch provider keys to example values such as JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent that text as a Bearer key, so every web_fetch on a fresh install failed with 401 while make doctor reported web fetch as configured. - .env.example comments those nine lines out, like every other credential in the template already is. - provider_credentials.resolve_api_key reads a your-...-key/-token/-secret value as unset, so existing .env files that still hold the placeholders recover without editing, for every provider that resolves keys there. * chore: leave the changelog entry to merge time Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they merge a pull request. * fix(web-search): route every provider key and doctor through one placeholder rule Review follow-up for #8206: - provider_credentials exposes is_template_placeholder(), the single definition of an .env.example example value. key/token/secret must be a whole hyphen-separated word, so a real value such as your-monkey or your-passkey is no longer dropped. - Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA and You.com resolve config and environment keys through resolve_api_key instead of private strip-and-send copies, so an old .env that still holds the template value no longer sends it. - make doctor asks the same predicate: a placeholder key is reported as still holding the .env.example value instead of "set", and a literal placeholder in config.yaml falls back to the environment key as the runtime does. - A census test keeps credential environment reads under community/ wrapped in resolve_api_key. * fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor Second review follow-up for #8206: - is_template_placeholder() matches exactly the your-... values that .env.example ships instead of a pattern, so a token an operator chose (a self-hosted Browserless TOKEN such as your-review-team-token) is sent again. A test keeps the set equal to the template. - make doctor mirrors Browserless: a configured token field is authoritative, blank or not, and BROWSERLESS_TOKEN is read only when the field is omitted. A blank or placeholder token now warns instead of reporting the variable, cross-checked against the client. - The keyless Jina and You.com providers stay ok but name JINA_API_KEY or YDC_API_KEY when a placeholder there is ignored, and a literal placeholder api_key in config.yaml is named when no env key backs it. * fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs Found while re-checking #8206 against every provider's runtime: - Browserless used as web_fetch had no doctor key check; it now gets the same authoritative token handling as web_capture. - InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an api_key field in its tool config as configured. - GroundRoute needs a key for every call but had no doctor entry, so a missing key reported ok. - You.com names a placeholder api_key in config that it ignores. - The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which the tool never reads. With JINA_API_KEY now commented out in .env.example, that line made config loading fail on a fresh install. - The provider_credentials docstring now scopes doctor's use of the predicate to web provider keys, and the scripts guide no longer says .env.example ships live placeholder keys. |
||
|
|
1794c40888
|
fix(community): screen web_search result URLs in every provider (#7037)
* fix(community): screen tavily and exa web_search result URLs The Tavily and Exa web_search tools returned provider URLs to the model unfiltered, so a loopback, link-local or metadata-service address in a result reached the agent. Brave and Serper already pass result URLs through safe_public_result_url; do the same here, blanking a non-public URL while keeping the row's title and snippet. * fix(community): screen result URLs in the remaining web_search providers youcom, tencent_wsa, serply (url and scholar pdf_url), groundroute, sofya, ddg_search, fastcrw, firecrawl and searxng handed provider-supplied URLs to the model unfiltered. Route each through safe_public_result_url, as brave, serper, tavily and exa do: a non-public URL is blanked and the row's title and snippet are kept. The test now drives every listed provider's web_search tool with a stubbed provider payload carrying a metadata, loopback or public URL. * fix(community): screen brave, serper and webz web_search result URLs The remaining text-search providers still returned result URLs verbatim. Serper's comment documented that policy; it now screens organic links like its image results. The shared provider test covers all fourteen providers and asserts the snippet survives screening. * docs(tests): describe text-search URL screening coverage --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
62588f3675
|
fix(sofya): report a format error for a results list with no usable objects (#6719)
* fix(sofya): report a format error for results with no usable objects `_response_results` skipped non-object entries but returned the filtered list even when it was empty, so a non-empty `results` array holding no objects fell through to "No results found" -- telling the agent the query legitimately matched nothing instead of surfacing the malformed payload. Treat a non-empty list with no object entries as malformed, matching the containment rule #6603 documents for this provider family (brave #6550, groundroute #6603, tavily #6645, webz). Both `web_search` and `web_fetch` share this helper, so both now report the provider's format error. Missing, null, and empty `results` keep returning "No results found", and mixed lists still keep their valid entries in order. * test(sofya): cover mixed fetch result entries --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
5092a45943
|
fix(tools): make the search coercion guards catch OverflowError (#5957)
* fix(tools): make the search coercion guards catch OverflowError
The community search tools coerce max_results (and Sofya's content
limit) from YAML config or tool parameters through guards that catch
only TypeError and ValueError. `int(float('inf'))` raises OverflowError
instead, so a YAML `.inf` (or a JSON `Infinity`) value still crashes
the coercion - SearXNG's search even fails wholesale with
'cannot convert float infinity to integer'.
Catch OverflowError alongside them so an infinite value falls back to
the default like every other invalid input. NaN is already covered by
ValueError.
Signed-off-by: yetuge <2219677952@qq.com>
* test(tools): move the GroundRoute inf test to module scope
The regression test was appended inside TestWebSearchTool, ending the
class early and turning its last two methods into uncollected nested
functions. Move it after TestWebFetchTool so pytest collects all 20
tests again.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
|
||
|
|
23bd76046a
|
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider Add a community provider backed by Sofya (https://sofya.co). Its search endpoint returns the content of the result pages, not only their snippets, and its fetch endpoint returns a page as markdown. Both are plain JSON over HTTP, so this needs no extra Python package (uses httpx, already a dependency). Changes: - backend/packages/harness/deerflow/community/sofya/__init__.py - backend/packages/harness/deerflow/community/sofya/tools.py Implements web_search_tool and web_fetch_tool using httpx. API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY env var. Follows the same interface and output shape as the existing ddg_search and serper providers, including the max_results parameter with config override and the structured "No results found" error. - backend/tests/test_sofya_tools.py Unit tests covering API key resolution, config overrides, result mapping, time range, HTTP errors, empty results, and fetch failures. - config.example.yaml: add commented-out Sofya web_search and web_fetch examples alongside the other providers - .env.example: add SOFYA_API_KEY placeholder - backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and the environment variables * fix(sofya): honor caller max_results, validate search_depth, join time_range contract test - Caller-supplied max_results now wins; config is used only when the argument is omitted, matching GroundRoute. - search_depth is clamped to basic/snippets; an unsupported value logs a warning and falls back to basic. - Sofya added to the shared time_range schema contract test. * fix(sofya): cap per-result content so a search stays inline An unbounded search payload (up to 20 read pages) crossed the tool output budget middleware's externalize_min_chars threshold, which replaces the result list with a file reference. Cap each result's content at contents_max_characters (default 2000, 0 disables), matching Exa's config key. Five capped results stay under the 12000 char threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): list Sofya in the recency contract, coerce non-string content _clip subscripted its input, so a non-string content or description from the API raised TypeError instead of degrading. Coerce to text first, the way _sofya_post and _response_results guard the shapes around it. Also add Sofya to the Web Search Recency section in backend/AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog web_fetch sliced its content the same way web_search did before the last push: a truthy non-string from the API passed the falsiness guard and then raised TypeError. Reuse _clip, keeping the `or ""` so empty content still reports "No content found". Also add sofya to the community provider inventory in packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 * docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs The ZH web_fetch tab list named five providers where EN names seven. Both tabs mirror their EN counterparts, so the two locales list the same web_fetch providers again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |