mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
feat: support configurable Serper base URL (#6483)
* feat: support configurable Serper base URL * fix(serper): address configurable endpoint review feedback * fix(serper): address endpoint configuration review * fix(serper): pair provider settings and validate endpoints * test(serper): account for URL redaction in endpoint diagnostics --------- Co-authored-by: litescraper <322634161+litescraper@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
parent
c7f9d35b66
commit
e1458eefee
@ -1,6 +1,10 @@
|
||||
# Serper API Key (Google Search) - https://serper.dev
|
||||
SERPER_API_KEY=your-serper-api-key
|
||||
|
||||
# Optional operator-controlled Serper-compatible API base URL, without /search or /images.
|
||||
# The configured API key is sent in X-API-KEY to this host; use a trusted provider.
|
||||
SERPER_BASE_URL=https://google.serper.dev
|
||||
|
||||
# Serply API Key (Google Search, News and Scholar) - https://serply.io
|
||||
SERPLY_API_KEY=your-serply-api-key
|
||||
|
||||
|
||||
10
README.md
10
README.md
@ -1419,6 +1419,16 @@ These are deployment settings; the model still supplies only `query` and optiona
|
||||
empty list is forwarded and imposes no restriction of that kind. See the
|
||||
[tool configuration example](backend/docs/CONFIGURATION.md#tools).
|
||||
|
||||
Serper web and image search can use another Serper-compatible provider through
|
||||
`SERPER_BASE_URL` and that provider's `SERPER_API_KEY`. Each tool can instead set
|
||||
its own `base_url` and `api_key` in `config.yaml`; each request reads them from
|
||||
one configuration snapshot. Use an absolute HTTP(S) base URL with a host,
|
||||
without `/search`, `/images`, a query string or a fragment. Invalid overrides
|
||||
return a configuration error before HTTP; blank environment values keep
|
||||
Serper's default unless a tool overrides them. See
|
||||
[Serper endpoints](backend/docs/CONFIGURATION.md#serper-endpoints) for precedence
|
||||
and how the provider receives the API key.
|
||||
|
||||
Serper `web_search` supports deployment-level `include_domains` and
|
||||
`exclude_domains` too. It checks returned URL hosts (including subdomains), with
|
||||
exclusion taking precedence. Filters can return fewer results, including zero;
|
||||
|
||||
@ -728,6 +728,52 @@ Serper `web_search` also accepts the optional model argument
|
||||
to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the
|
||||
search request. This option does not change Serper `image_search`.
|
||||
|
||||
#### Serper endpoints
|
||||
|
||||
To use a Serper-compatible provider for web and image search, set
|
||||
`SERPER_BASE_URL` to its base URL without `/search` or `/images`, and set
|
||||
`SERPER_API_KEY` to that provider's key. The default base URL is
|
||||
`https://google.serper.dev`.
|
||||
|
||||
Each Serper tool can override the environment independently in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
tools:
|
||||
- name: web_search
|
||||
group: web
|
||||
use: deerflow.community.serper.tools:web_search_tool
|
||||
base_url: https://proxy.example/api
|
||||
api_key: $SEARCH_PROVIDER_API_KEY
|
||||
- name: image_search
|
||||
group: web
|
||||
use: deerflow.community.serper.tools:image_search_tool
|
||||
base_url: https://images.example/api
|
||||
api_key: $IMAGE_PROVIDER_API_KEY
|
||||
```
|
||||
|
||||
A non-empty string `base_url` in the requested tool's entry takes precedence
|
||||
over `SERPER_BASE_URL`. Missing, non-string or whitespace-only tool values fall
|
||||
back to the environment. An unset, empty or whitespace-only environment value
|
||||
keeps the default endpoint. Leading/trailing whitespace and trailing slashes
|
||||
are removed before appending `/search` or `/images`. A tool's `api_key` similarly
|
||||
overrides `SERPER_API_KEY`; neither tool inherits the other's endpoint or key.
|
||||
Both settings are read from the same captured tool configuration, so a hot
|
||||
reload cannot pair the old provider's endpoint with the new provider's key.
|
||||
|
||||
Overrides must be absolute `http://` or `https://` URLs with a host and valid
|
||||
port. Query strings and fragments (including empty `?`/`#` markers) are rejected
|
||||
before HTTP. Invalid overrides return a configuration error naming
|
||||
`base_url`/`SERPER_BASE_URL` without exposing the configured value; they do not
|
||||
fall back to another host.
|
||||
|
||||
These are operator-controlled settings, not model-supplied arguments. Choose a
|
||||
trusted provider: the key is sent to the configured host in the `X-API-KEY`
|
||||
header, never in a query parameter. Endpoint configuration does not change the
|
||||
existing validation of returned image URLs or web source filters. The endpoint
|
||||
is resolved once before transport so it can remain constant across retry attempts.
|
||||
Override debug diagnostics show the effective endpoint without URL credentials,
|
||||
query or fragment; result-URL guards do not restrict the operator's API host.
|
||||
|
||||
#### Serper source filters
|
||||
|
||||
```yaml
|
||||
@ -1413,6 +1459,7 @@ models:
|
||||
- `TAVILY_API_KEY` - Tavily search API key
|
||||
- `BRAVE_SEARCH_API_KEY` - Brave Search API key for `web_search` and `image_search`
|
||||
- `SERPER_API_KEY` - Serper (Google Search/Images API) key for `web_search` and `image_search`
|
||||
- `SERPER_BASE_URL` - Optional operator-controlled Serper-compatible base URL for both tools; each tool's `base_url` takes precedence. The provider key is sent to the configured host in `X-API-KEY`, never a query parameter. See [Serper endpoints](#serper-endpoints).
|
||||
- `SERPLY_API_KEY` - [Serply](https://serply.io) key for `web_search` (Google Search, plus Google News and Google Scholar via `vertical`)
|
||||
- `GROUNDROUTE_API_KEY` - GroundRoute meta-search API key for `web_search` and `web_fetch` (routes across Serper, Brave, Exa, Tavily, Firecrawl, Perplexity with gain-share pricing)
|
||||
- `SOFYA_API_KEY` - [Sofya](https://sofya.co) key for `web_search` and `web_fetch`
|
||||
|
||||
@ -12,6 +12,23 @@ report the cleaned original query and actual filtered count, including zero.
|
||||
Image search and unconfigured web behavior retain their existing contracts.
|
||||
This is source selection, not a fetch policy or factuality check.
|
||||
|
||||
Tests: `backend/tests/test_serper_domain_filters.py` and `test_serper_tools.py`.
|
||||
Resolve endpoints once in `_serper_post`, before transport setup: a usable
|
||||
per-tool `base_url` takes precedence over `SERPER_BASE_URL`, then the existing
|
||||
Serper endpoint. Strip surrounding whitespace and trailing slashes. Require an
|
||||
absolute HTTP(S) URL with a host and valid port; reject query/fragment markers,
|
||||
including empty ones, before HTTP without echoing configured values.
|
||||
Read each tool's config once and pass captured extras to key resolution,
|
||||
including an empty mapping when config is absent, to keep endpoint/key paired
|
||||
through hot reload. Settings stay operator-only. Send keys
|
||||
only via `X-API-KEY`, and preserve result URL validation and model arguments.
|
||||
Future retries must reuse the resolved endpoint across attempts. Debug endpoint
|
||||
diagnostics omit URL credentials, query and fragment; result-URL guards do not
|
||||
restrict the operator's API host.
|
||||
|
||||
Tests: `backend/tests/test_serper_domain_filters.py`, `test_serper_tools.py` and
|
||||
`test_serper_endpoints.py`.
|
||||
Diagnostic tests cover plain logs and the shared `UrlRedactionFilter`; root
|
||||
handlers can redact paths. Assert full endpoint routing on the captured HTTP
|
||||
request, and verify diagnostic host/port and credential omission in both modes.
|
||||
Mock HTTP; live Serper semantics remain unverified. See
|
||||
`backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract.
|
||||
|
||||
@ -9,6 +9,7 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from ipaddress import IPv4Address, ip_address
|
||||
from urllib.parse import urlparse
|
||||
|
||||
@ -32,12 +33,14 @@ _SERPER_TBS_BY_TIME_RANGE: dict[SearchTimeRange, str] = {
|
||||
_api_key_warned: set[str] = set()
|
||||
|
||||
|
||||
def _get_api_key(tool_name: str) -> str | None:
|
||||
config = get_app_config().get_tool_config(tool_name)
|
||||
if config is not None:
|
||||
api_key = config.model_extra.get("api_key")
|
||||
if isinstance(api_key, str) and api_key.strip():
|
||||
return api_key.strip()
|
||||
def _get_api_key(tool_name: str, *, extras: Mapping[str, object] | None = None) -> str | None:
|
||||
"""Resolve a key using captured tool settings, then the environment."""
|
||||
if extras is None:
|
||||
config = get_app_config().get_tool_config(tool_name)
|
||||
extras = config.model_extra if config is not None else {}
|
||||
api_key = extras.get("api_key")
|
||||
if isinstance(api_key, str) and api_key.strip():
|
||||
return api_key.strip()
|
||||
env_key = os.getenv("SERPER_API_KEY")
|
||||
if isinstance(env_key, str) and env_key.strip():
|
||||
return env_key.strip()
|
||||
@ -246,10 +249,14 @@ def _safe_public_url(value: object) -> str:
|
||||
return url if ip.is_global else ""
|
||||
|
||||
|
||||
def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None) -> tuple[dict | None, str | None]:
|
||||
def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None, base_url: object | None = None) -> tuple[dict | None, str | None]:
|
||||
"""Send a POST request to a Serper endpoint.
|
||||
|
||||
``query`` is expected to already be normalized via :func:`_clean_query`.
|
||||
A non-blank string ``base_url`` overrides ``SERPER_BASE_URL``; other raw
|
||||
tool-config values fall back to the environment and then the default endpoint.
|
||||
Overrides must be absolute HTTP(S) URLs with a host and no query or fragment.
|
||||
The API key is sent only via ``X-API-KEY`` to the resolved endpoint.
|
||||
|
||||
Returns a ``(data, error_json)`` tuple: on success ``data`` is the parsed
|
||||
JSON response and ``error_json`` is ``None``; on failure ``data`` is ``None``
|
||||
@ -264,6 +271,28 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t
|
||||
payload["tbs"] = _SERPER_TBS_BY_TIME_RANGE[time_range]
|
||||
|
||||
try:
|
||||
# Resolve once before transport setup so retries can reuse this endpoint.
|
||||
if not isinstance(base_url, str) or not base_url.strip():
|
||||
base_url = os.getenv("SERPER_BASE_URL") or ""
|
||||
base_url = base_url.strip()
|
||||
if base_url:
|
||||
try:
|
||||
parsed_base = urlparse(base_url)
|
||||
valid_base = parsed_base.scheme in {"http", "https"} and bool(parsed_base.netloc and parsed_base.hostname) and "?" not in base_url and "#" not in base_url
|
||||
# Accessing port also rejects invalid and out-of-range ports.
|
||||
parsed_base.port
|
||||
except ValueError:
|
||||
valid_base = False
|
||||
if not valid_base:
|
||||
raise ValueError("Invalid Serper base_url/SERPER_BASE_URL: use an absolute HTTP(S) URL with a host, a valid port, and no query or fragment")
|
||||
endpoint = base_url.rstrip("/") + "/" + endpoint.rsplit("/", 1)[-1]
|
||||
parsed_endpoint = urlparse(endpoint)
|
||||
logger.debug(
|
||||
"Serper endpoint from base_url/SERPER_BASE_URL: %s://%s%s",
|
||||
parsed_endpoint.scheme,
|
||||
parsed_endpoint.netloc.rsplit("@", 1)[-1],
|
||||
parsed_endpoint.path,
|
||||
)
|
||||
with httpx.Client(timeout=30) as client:
|
||||
response = client.post(endpoint, headers=headers, json=payload)
|
||||
response.raise_for_status()
|
||||
@ -308,11 +337,12 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang
|
||||
logger.error("Invalid Serper domain filters: %s", exc)
|
||||
return json.dumps({"error": str(exc), "query": query}, ensure_ascii=False)
|
||||
|
||||
api_key = _get_api_key("web_search")
|
||||
api_key = _get_api_key("web_search", extras=extra)
|
||||
if not api_key:
|
||||
return _missing_key_error(query, "web_search")
|
||||
|
||||
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range)
|
||||
base_url = extra.get("base_url")
|
||||
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url)
|
||||
if error_json is not None:
|
||||
error = json.loads(error_json)
|
||||
error["query"] = query
|
||||
@ -363,11 +393,13 @@ def image_search_tool(query: str, max_results: int = 5) -> str:
|
||||
max_results = _coerce_max_results(max_results)
|
||||
query = _clean_query(query)
|
||||
|
||||
api_key = _get_api_key("image_search")
|
||||
extra = config.model_extra if config is not None else {}
|
||||
api_key = _get_api_key("image_search", extras=extra)
|
||||
if not api_key:
|
||||
return _missing_key_error(query, "image_search")
|
||||
|
||||
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results)
|
||||
base_url = extra.get("base_url")
|
||||
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url)
|
||||
if error_json is not None:
|
||||
return error_json
|
||||
|
||||
|
||||
130
backend/tests/test_serper_endpoints.py
Normal file
130
backend/tests/test_serper_endpoints.py
Normal file
@ -0,0 +1,130 @@
|
||||
"""Offline endpoint configuration contracts for both Serper tools."""
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.community.serper import tools
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def serper(monkeypatch):
|
||||
configs = {"web_search": {"api_key": "web-key"}, "image_search": {"api_key": "image-key"}}
|
||||
monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda name: SimpleNamespace(model_extra=configs[name])))
|
||||
monkeypatch.delenv("SERPER_BASE_URL", raising=False)
|
||||
response = MagicMock()
|
||||
response.json.return_value = {
|
||||
"organic": [{"title": "Web result", "link": "https://example.com/page", "snippet": "Snippet"}],
|
||||
"images": [{"title": "Image result", "imageUrl": "https://example.com/image.jpg", "thumbnailUrl": "https://example.com/thumb.jpg"}],
|
||||
}
|
||||
client = MagicMock()
|
||||
post = client.return_value.__enter__.return_value.post
|
||||
post.return_value = response
|
||||
monkeypatch.setattr(tools.httpx, "Client", client)
|
||||
return SimpleNamespace(configs=configs, post=post, client=client)
|
||||
|
||||
|
||||
def assert_search(serper, tool_name, endpoint):
|
||||
search_tool = tools.web_search_tool if tool_name == "web_search" else tools.image_search_tool
|
||||
arguments = {"query": " test ", "max_results": 3}
|
||||
payload = {"q": "test", "num": 3}
|
||||
if tool_name == "web_search":
|
||||
arguments["time_range"] = "week"
|
||||
payload["tbs"] = "qdr:w"
|
||||
result = json.loads(search_tool.invoke(arguments))
|
||||
assert result["query"] == "test"
|
||||
assert result["total_results"] == 1
|
||||
serper.post.assert_called_once_with(endpoint, headers={"X-API-KEY": serper.configs[tool_name]["api_key"], "Content-Type": "application/json"}, json=payload)
|
||||
|
||||
|
||||
def test_tool_config_precedes_environment_and_keeps_tool_credentials_paired(serper, monkeypatch):
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://env.example")
|
||||
serper.configs["web_search"]["base_url"] = " https://web.example/api/ "
|
||||
serper.configs["image_search"]["base_url"] = "\thttps://images.example/v1///\n"
|
||||
assert_search(serper, "web_search", "https://web.example/api/search")
|
||||
serper.post.reset_mock()
|
||||
assert_search(serper, "image_search", "https://images.example/v1/images")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
|
||||
@pytest.mark.parametrize("base_url", [None, "", " \t", 42, False, {}])
|
||||
def test_unusable_tool_config_falls_back_to_environment(serper, monkeypatch, tool_name, route, base_url):
|
||||
monkeypatch.setenv("SERPER_BASE_URL", " https://env.example/ ")
|
||||
serper.configs[tool_name]["base_url"] = base_url
|
||||
assert_search(serper, tool_name, f"https://env.example/{route}")
|
||||
|
||||
|
||||
def test_web_endpoint_does_not_leak_to_image_tool(serper):
|
||||
serper.configs["web_search"]["base_url"] = "https://web.example"
|
||||
assert_search(serper, "image_search", "https://google.serper.dev/images")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
|
||||
def test_environment_endpoint_without_tool_config(serper, monkeypatch, tool_name, route):
|
||||
monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: None))
|
||||
monkeypatch.setenv("SERPER_API_KEY", serper.configs[tool_name]["api_key"])
|
||||
monkeypatch.setenv("SERPER_BASE_URL", " https://env.example/ ")
|
||||
assert_search(serper, tool_name, f"https://env.example/{route}")
|
||||
|
||||
|
||||
def test_endpoint_is_operator_configuration_only():
|
||||
assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"}
|
||||
assert set(tools.image_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
|
||||
@pytest.mark.parametrize("has_tool_config", [True, False], ids=["tool-config", "environment-only"])
|
||||
def test_endpoint_and_key_use_one_config_snapshot(serper, monkeypatch, tool_name, route, has_tool_config):
|
||||
serper.configs[tool_name]["base_url"] = "https://first.example/api"
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://first.example/api")
|
||||
monkeypatch.setenv("SERPER_API_KEY", serper.configs[tool_name]["api_key"])
|
||||
initial_tool = SimpleNamespace(model_extra=serper.configs[tool_name]) if has_tool_config else None
|
||||
first = SimpleNamespace(get_tool_config=MagicMock(return_value=initial_tool))
|
||||
reloaded = SimpleNamespace(get_tool_config=MagicMock(return_value=SimpleNamespace(model_extra={"api_key": "other-provider-secret", "base_url": "https://second.example/api"})))
|
||||
get_config = MagicMock(side_effect=[first, reloaded])
|
||||
monkeypatch.setattr(tools, "get_app_config", get_config)
|
||||
|
||||
assert_search(serper, tool_name, f"https://first.example/api/{route}")
|
||||
|
||||
get_config.assert_called_once_with()
|
||||
first.get_tool_config.assert_called_once_with(tool_name)
|
||||
reloaded.get_tool_config.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool_name", ["web_search", "image_search"])
|
||||
@pytest.mark.parametrize("source", ["tool-config", "environment"])
|
||||
@pytest.mark.parametrize(
|
||||
"base_url",
|
||||
[
|
||||
"///",
|
||||
"ftp://proxy.example",
|
||||
"https:///missing-host",
|
||||
"https://:8443",
|
||||
"https://proxy.example/api?token=URL_SECRET",
|
||||
"https://proxy.example/api#URL_SECRET",
|
||||
pytest.param("https://proxy.example/api?", id="empty-query"),
|
||||
pytest.param("https://proxy.example/api#", id="empty-fragment"),
|
||||
"https://[invalid/",
|
||||
"https://proxy.example:invalid/api",
|
||||
],
|
||||
)
|
||||
def test_invalid_base_url_fails_before_transport_without_exposing_value(serper, monkeypatch, caplog, tool_name, source, base_url):
|
||||
if source == "tool-config":
|
||||
serper.configs[tool_name]["base_url"] = base_url
|
||||
# Invalid explicit overrides must not send the key to a fallback host.
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://fallback.example")
|
||||
else:
|
||||
monkeypatch.setenv("SERPER_BASE_URL", base_url)
|
||||
search_tool = tools.web_search_tool if tool_name == "web_search" else tools.image_search_tool
|
||||
|
||||
result = json.loads(search_tool.invoke({"query": " test "}))
|
||||
|
||||
assert "base_url" in result["error"]
|
||||
assert "SERPER_BASE_URL" in result["error"]
|
||||
assert result["query"] == "test"
|
||||
assert "URL_SECRET" not in json.dumps(result)
|
||||
assert "URL_SECRET" not in caplog.text
|
||||
serper.client.assert_not_called()
|
||||
serper.post.assert_not_called()
|
||||
@ -1,6 +1,7 @@
|
||||
"""Unit tests for the Serper community web search tool."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import httpx
|
||||
@ -8,10 +9,11 @@ import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_api_key_warned():
|
||||
def reset_api_key_warned(monkeypatch):
|
||||
"""Reset the module-level warning flag before each test."""
|
||||
import deerflow.community.serper.tools as serper_mod
|
||||
|
||||
monkeypatch.delenv("SERPER_BASE_URL", raising=False)
|
||||
serper_mod._api_key_warned = set()
|
||||
yield
|
||||
serper_mod._api_key_warned = set()
|
||||
@ -49,6 +51,98 @@ def _make_serper_images_response(images: list) -> MagicMock:
|
||||
return mock_resp
|
||||
|
||||
|
||||
class TestSerperBaseUrl:
|
||||
@pytest.mark.parametrize(("tool_name", "route"), [("web_search_tool", "search"), ("image_search_tool", "images")])
|
||||
@pytest.mark.parametrize(
|
||||
("base_url", "expected_base"),
|
||||
[
|
||||
(None, "https://google.serper.dev"),
|
||||
("", "https://google.serper.dev"),
|
||||
(" \t\r\n", "https://google.serper.dev"),
|
||||
("https://proxy.example", "https://proxy.example"),
|
||||
("https://proxy.example/", "https://proxy.example"),
|
||||
(" \thttps://proxy.example/// \r\n", "https://proxy.example"),
|
||||
("https://proxy.example/api/v1/", "https://proxy.example/api/v1"),
|
||||
("http://127.0.0.1:8080/", "http://127.0.0.1:8080"),
|
||||
],
|
||||
)
|
||||
def test_tools_post_to_configured_endpoint(self, monkeypatch, mock_config_with_key, tool_name: str, route: str, base_url: str | None, expected_base: str) -> None:
|
||||
import deerflow.community.serper.tools as serper_mod
|
||||
|
||||
if base_url is not None:
|
||||
monkeypatch.setenv("SERPER_BASE_URL", base_url)
|
||||
response = MagicMock()
|
||||
response.json.return_value = {
|
||||
"organic": [{"title": "Example", "link": "https://example.com", "snippet": "Result"}],
|
||||
"images": [{"title": "Example", "imageUrl": "https://example.com/image.jpg"}],
|
||||
}
|
||||
with patch("deerflow.community.serper.tools.httpx.Client") as client_class:
|
||||
post = client_class.return_value.__enter__.return_value.post
|
||||
post.return_value = response
|
||||
result = json.loads(getattr(serper_mod, tool_name).invoke({"query": "https://model-input.invalid"}))
|
||||
|
||||
assert "error" not in result
|
||||
post.assert_called_once()
|
||||
assert post.call_args.args[0] == f"{expected_base}/{route}"
|
||||
assert post.call_args.kwargs["headers"] == {"X-API-KEY": "test-serper-key", "Content-Type": "application/json"}
|
||||
assert post.call_args.kwargs["json"] == {"q": "https://model-input.invalid", "num": 5}
|
||||
assert "params" not in post.call_args.kwargs
|
||||
assert "test-serper-key" not in post.call_args.args[0]
|
||||
|
||||
@pytest.mark.parametrize("route", ["search", "images"])
|
||||
def test_endpoint_is_resolved_before_transport(self, monkeypatch, route: str) -> None:
|
||||
import deerflow.community.serper.tools as serper_mod
|
||||
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://first.example/")
|
||||
with patch("deerflow.community.serper.tools.httpx.Client") as client_class:
|
||||
client = MagicMock()
|
||||
client.post.return_value = _make_serper_response([])
|
||||
|
||||
def enter_client():
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://later.example/")
|
||||
return client
|
||||
|
||||
client_class.return_value.__enter__.side_effect = enter_client
|
||||
data, error = serper_mod._serper_post(f"https://google.serper.dev/{route}", "test-key", "test", 5, time_range="week")
|
||||
|
||||
assert error is None
|
||||
assert data == {"organic": []}
|
||||
assert client.post.call_args.args[0] == f"https://first.example/{route}"
|
||||
assert client.post.call_args.kwargs["json"] == {"q": "test", "num": 5, "tbs": "qdr:w"}
|
||||
|
||||
@pytest.mark.parametrize("route", ["search", "images"])
|
||||
@pytest.mark.parametrize("redact_urls", [False, True], ids=["plain-log", "redacted-log"])
|
||||
def test_override_diagnostic_omits_credentials(self, monkeypatch, caplog, route: str, redact_urls: bool) -> None:
|
||||
import deerflow.community.serper.tools as serper_mod
|
||||
from deerflow.logging_config import UrlRedactionFilter
|
||||
|
||||
if redact_urls:
|
||||
# Production logging installs this filter on root handlers; another
|
||||
# test may already have enabled it before this test runs in a shard.
|
||||
monkeypatch.setattr(caplog.handler, "filters", [*caplog.handler.filters, UrlRedactionFilter()])
|
||||
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://user:password@proxy.example:8443/api/v1/")
|
||||
with caplog.at_level(logging.DEBUG, logger=serper_mod.__name__), patch("deerflow.community.serper.tools.httpx.Client") as client_class:
|
||||
post = client_class.return_value.__enter__.return_value.post
|
||||
post.return_value = _make_serper_response([])
|
||||
_, error = serper_mod._serper_post(f"https://google.serper.dev/{route}", "test-secret-api-key", "test", 5)
|
||||
|
||||
assert error is None
|
||||
post.assert_called_once()
|
||||
assert post.call_args.args[0] == f"https://user:password@proxy.example:8443/api/v1/{route}"
|
||||
records = [record for record in caplog.records if "SERPER_BASE_URL" in record.getMessage()]
|
||||
assert len(records) == 1
|
||||
diagnostic = records[0].getMessage()
|
||||
assert diagnostic in {
|
||||
f"Serper endpoint from base_url/SERPER_BASE_URL: https://proxy.example:8443/api/v1/{route}",
|
||||
"Serper endpoint from base_url/SERPER_BASE_URL: https://proxy.example:8443/<redacted>",
|
||||
}
|
||||
if redact_urls:
|
||||
assert "<redacted>" in diagnostic
|
||||
assert "user:password" not in caplog.text
|
||||
assert "test-secret-api-key" not in caplog.text
|
||||
|
||||
|
||||
class TestGetApiKey:
|
||||
def test_returns_config_key_when_present(self):
|
||||
with patch("deerflow.community.serper.tools.get_app_config") as mock:
|
||||
|
||||
@ -933,6 +933,7 @@ tools:
|
||||
# group: web
|
||||
# use: deerflow.community.serper.tools:web_search_tool
|
||||
# max_results: 5 # capped at 10 by the Serper provider
|
||||
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /search
|
||||
# include_domains: [] # Optional domain-only lists, at most 10 entries each
|
||||
# exclude_domains: [] # Exclusion wins; see backend/docs/CONFIGURATION.md
|
||||
# # api_key: $SERPER_API_KEY # Optional explicit env-var reference
|
||||
@ -1275,6 +1276,7 @@ tools:
|
||||
# group: web
|
||||
# use: deerflow.community.serper.tools:image_search_tool
|
||||
# max_results: 5 # capped at 10 by the Serper provider
|
||||
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /images
|
||||
# # api_key: $SERPER_API_KEY # Optional explicit env-var reference
|
||||
|
||||
# Image search tool (uses Brave Image Search API, requires BRAVE_SEARCH_API_KEY)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user