feat: support configurable Serper base URL (#6483)

* feat: support configurable Serper base URL

* fix(serper): address configurable endpoint review feedback

* fix(serper): address endpoint configuration review

* fix(serper): pair provider settings and validate endpoints

* test(serper): account for URL redaction in endpoint diagnostics

---------

Co-authored-by: litescraper <322634161+litescraper@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
litescraper 2026-10-08 08:08:03 -05:00 committed by GitHub
parent c7f9d35b66
commit e1458eefee
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 349 additions and 13 deletions

View File

@ -1,6 +1,10 @@
# Serper API Key (Google Search) - https://serper.dev
SERPER_API_KEY=your-serper-api-key
# Optional operator-controlled Serper-compatible API base URL, without /search or /images.
# The configured API key is sent in X-API-KEY to this host; use a trusted provider.
SERPER_BASE_URL=https://google.serper.dev
# Serply API Key (Google Search, News and Scholar) - https://serply.io
SERPLY_API_KEY=your-serply-api-key

View File

@ -1419,6 +1419,16 @@ These are deployment settings; the model still supplies only `query` and optiona
empty list is forwarded and imposes no restriction of that kind. See the
[tool configuration example](backend/docs/CONFIGURATION.md#tools).
Serper web and image search can use another Serper-compatible provider through
`SERPER_BASE_URL` and that provider's `SERPER_API_KEY`. Each tool can instead set
its own `base_url` and `api_key` in `config.yaml`; each request reads them from
one configuration snapshot. Use an absolute HTTP(S) base URL with a host,
without `/search`, `/images`, a query string or a fragment. Invalid overrides
return a configuration error before HTTP; blank environment values keep
Serper's default unless a tool overrides them. See
[Serper endpoints](backend/docs/CONFIGURATION.md#serper-endpoints) for precedence
and how the provider receives the API key.
Serper `web_search` supports deployment-level `include_domains` and
`exclude_domains` too. It checks returned URL hosts (including subdomains), with
exclusion taking precedence. Filters can return fewer results, including zero;

View File

@ -728,6 +728,52 @@ Serper `web_search` also accepts the optional model argument
to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the
search request. This option does not change Serper `image_search`.
#### Serper endpoints
To use a Serper-compatible provider for web and image search, set
`SERPER_BASE_URL` to its base URL without `/search` or `/images`, and set
`SERPER_API_KEY` to that provider's key. The default base URL is
`https://google.serper.dev`.
Each Serper tool can override the environment independently in `config.yaml`:
```yaml
tools:
- name: web_search
group: web
use: deerflow.community.serper.tools:web_search_tool
base_url: https://proxy.example/api
api_key: $SEARCH_PROVIDER_API_KEY
- name: image_search
group: web
use: deerflow.community.serper.tools:image_search_tool
base_url: https://images.example/api
api_key: $IMAGE_PROVIDER_API_KEY
```
A non-empty string `base_url` in the requested tool's entry takes precedence
over `SERPER_BASE_URL`. Missing, non-string or whitespace-only tool values fall
back to the environment. An unset, empty or whitespace-only environment value
keeps the default endpoint. Leading/trailing whitespace and trailing slashes
are removed before appending `/search` or `/images`. A tool's `api_key` similarly
overrides `SERPER_API_KEY`; neither tool inherits the other's endpoint or key.
Both settings are read from the same captured tool configuration, so a hot
reload cannot pair the old provider's endpoint with the new provider's key.
Overrides must be absolute `http://` or `https://` URLs with a host and valid
port. Query strings and fragments (including empty `?`/`#` markers) are rejected
before HTTP. Invalid overrides return a configuration error naming
`base_url`/`SERPER_BASE_URL` without exposing the configured value; they do not
fall back to another host.
These are operator-controlled settings, not model-supplied arguments. Choose a
trusted provider: the key is sent to the configured host in the `X-API-KEY`
header, never in a query parameter. Endpoint configuration does not change the
existing validation of returned image URLs or web source filters. The endpoint
is resolved once before transport so it can remain constant across retry attempts.
Override debug diagnostics show the effective endpoint without URL credentials,
query or fragment; result-URL guards do not restrict the operator's API host.
#### Serper source filters
```yaml
@ -1413,6 +1459,7 @@ models:
- `TAVILY_API_KEY` - Tavily search API key
- `BRAVE_SEARCH_API_KEY` - Brave Search API key for `web_search` and `image_search`
- `SERPER_API_KEY` - Serper (Google Search/Images API) key for `web_search` and `image_search`
- `SERPER_BASE_URL` - Optional operator-controlled Serper-compatible base URL for both tools; each tool's `base_url` takes precedence. The provider key is sent to the configured host in `X-API-KEY`, never a query parameter. See [Serper endpoints](#serper-endpoints).
- `SERPLY_API_KEY` - [Serply](https://serply.io) key for `web_search` (Google Search, plus Google News and Google Scholar via `vertical`)
- `GROUNDROUTE_API_KEY` - GroundRoute meta-search API key for `web_search` and `web_fetch` (routes across Serper, Brave, Exa, Tavily, Firecrawl, Perplexity with gain-share pricing)
- `SOFYA_API_KEY` - [Sofya](https://sofya.co) key for `web_search` and `web_fetch`

View File

@ -12,6 +12,23 @@ report the cleaned original query and actual filtered count, including zero.
Image search and unconfigured web behavior retain their existing contracts.
This is source selection, not a fetch policy or factuality check.
Tests: `backend/tests/test_serper_domain_filters.py` and `test_serper_tools.py`.
Resolve endpoints once in `_serper_post`, before transport setup: a usable
per-tool `base_url` takes precedence over `SERPER_BASE_URL`, then the existing
Serper endpoint. Strip surrounding whitespace and trailing slashes. Require an
absolute HTTP(S) URL with a host and valid port; reject query/fragment markers,
including empty ones, before HTTP without echoing configured values.
Read each tool's config once and pass captured extras to key resolution,
including an empty mapping when config is absent, to keep endpoint/key paired
through hot reload. Settings stay operator-only. Send keys
only via `X-API-KEY`, and preserve result URL validation and model arguments.
Future retries must reuse the resolved endpoint across attempts. Debug endpoint
diagnostics omit URL credentials, query and fragment; result-URL guards do not
restrict the operator's API host.
Tests: `backend/tests/test_serper_domain_filters.py`, `test_serper_tools.py` and
`test_serper_endpoints.py`.
Diagnostic tests cover plain logs and the shared `UrlRedactionFilter`; root
handlers can redact paths. Assert full endpoint routing on the captured HTTP
request, and verify diagnostic host/port and credential omission in both modes.
Mock HTTP; live Serper semantics remain unverified. See
`backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract.

View File

@ -9,6 +9,7 @@ import json
import logging
import os
import re
from collections.abc import Mapping
from ipaddress import IPv4Address, ip_address
from urllib.parse import urlparse
@ -32,12 +33,14 @@ _SERPER_TBS_BY_TIME_RANGE: dict[SearchTimeRange, str] = {
_api_key_warned: set[str] = set()
def _get_api_key(tool_name: str) -> str | None:
config = get_app_config().get_tool_config(tool_name)
if config is not None:
api_key = config.model_extra.get("api_key")
if isinstance(api_key, str) and api_key.strip():
return api_key.strip()
def _get_api_key(tool_name: str, *, extras: Mapping[str, object] | None = None) -> str | None:
"""Resolve a key using captured tool settings, then the environment."""
if extras is None:
config = get_app_config().get_tool_config(tool_name)
extras = config.model_extra if config is not None else {}
api_key = extras.get("api_key")
if isinstance(api_key, str) and api_key.strip():
return api_key.strip()
env_key = os.getenv("SERPER_API_KEY")
if isinstance(env_key, str) and env_key.strip():
return env_key.strip()
@ -246,10 +249,14 @@ def _safe_public_url(value: object) -> str:
return url if ip.is_global else ""
def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None) -> tuple[dict | None, str | None]:
def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None, base_url: object | None = None) -> tuple[dict | None, str | None]:
"""Send a POST request to a Serper endpoint.
``query`` is expected to already be normalized via :func:`_clean_query`.
A non-blank string ``base_url`` overrides ``SERPER_BASE_URL``; other raw
tool-config values fall back to the environment and then the default endpoint.
Overrides must be absolute HTTP(S) URLs with a host and no query or fragment.
The API key is sent only via ``X-API-KEY`` to the resolved endpoint.
Returns a ``(data, error_json)`` tuple: on success ``data`` is the parsed
JSON response and ``error_json`` is ``None``; on failure ``data`` is ``None``
@ -264,6 +271,28 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t
payload["tbs"] = _SERPER_TBS_BY_TIME_RANGE[time_range]
try:
# Resolve once before transport setup so retries can reuse this endpoint.
if not isinstance(base_url, str) or not base_url.strip():
base_url = os.getenv("SERPER_BASE_URL") or ""
base_url = base_url.strip()
if base_url:
try:
parsed_base = urlparse(base_url)
valid_base = parsed_base.scheme in {"http", "https"} and bool(parsed_base.netloc and parsed_base.hostname) and "?" not in base_url and "#" not in base_url
# Accessing port also rejects invalid and out-of-range ports.
parsed_base.port
except ValueError:
valid_base = False
if not valid_base:
raise ValueError("Invalid Serper base_url/SERPER_BASE_URL: use an absolute HTTP(S) URL with a host, a valid port, and no query or fragment")
endpoint = base_url.rstrip("/") + "/" + endpoint.rsplit("/", 1)[-1]
parsed_endpoint = urlparse(endpoint)
logger.debug(
"Serper endpoint from base_url/SERPER_BASE_URL: %s://%s%s",
parsed_endpoint.scheme,
parsed_endpoint.netloc.rsplit("@", 1)[-1],
parsed_endpoint.path,
)
with httpx.Client(timeout=30) as client:
response = client.post(endpoint, headers=headers, json=payload)
response.raise_for_status()
@ -308,11 +337,12 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang
logger.error("Invalid Serper domain filters: %s", exc)
return json.dumps({"error": str(exc), "query": query}, ensure_ascii=False)
api_key = _get_api_key("web_search")
api_key = _get_api_key("web_search", extras=extra)
if not api_key:
return _missing_key_error(query, "web_search")
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range)
base_url = extra.get("base_url")
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url)
if error_json is not None:
error = json.loads(error_json)
error["query"] = query
@ -363,11 +393,13 @@ def image_search_tool(query: str, max_results: int = 5) -> str:
max_results = _coerce_max_results(max_results)
query = _clean_query(query)
api_key = _get_api_key("image_search")
extra = config.model_extra if config is not None else {}
api_key = _get_api_key("image_search", extras=extra)
if not api_key:
return _missing_key_error(query, "image_search")
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results)
base_url = extra.get("base_url")
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url)
if error_json is not None:
return error_json

View File

@ -0,0 +1,130 @@
"""Offline endpoint configuration contracts for both Serper tools."""
import json
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from deerflow.community.serper import tools
@pytest.fixture
def serper(monkeypatch):
configs = {"web_search": {"api_key": "web-key"}, "image_search": {"api_key": "image-key"}}
monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda name: SimpleNamespace(model_extra=configs[name])))
monkeypatch.delenv("SERPER_BASE_URL", raising=False)
response = MagicMock()
response.json.return_value = {
"organic": [{"title": "Web result", "link": "https://example.com/page", "snippet": "Snippet"}],
"images": [{"title": "Image result", "imageUrl": "https://example.com/image.jpg", "thumbnailUrl": "https://example.com/thumb.jpg"}],
}
client = MagicMock()
post = client.return_value.__enter__.return_value.post
post.return_value = response
monkeypatch.setattr(tools.httpx, "Client", client)
return SimpleNamespace(configs=configs, post=post, client=client)
def assert_search(serper, tool_name, endpoint):
search_tool = tools.web_search_tool if tool_name == "web_search" else tools.image_search_tool
arguments = {"query": " test ", "max_results": 3}
payload = {"q": "test", "num": 3}
if tool_name == "web_search":
arguments["time_range"] = "week"
payload["tbs"] = "qdr:w"
result = json.loads(search_tool.invoke(arguments))
assert result["query"] == "test"
assert result["total_results"] == 1
serper.post.assert_called_once_with(endpoint, headers={"X-API-KEY": serper.configs[tool_name]["api_key"], "Content-Type": "application/json"}, json=payload)
def test_tool_config_precedes_environment_and_keeps_tool_credentials_paired(serper, monkeypatch):
monkeypatch.setenv("SERPER_BASE_URL", "https://env.example")
serper.configs["web_search"]["base_url"] = " https://web.example/api/ "
serper.configs["image_search"]["base_url"] = "\thttps://images.example/v1///\n"
assert_search(serper, "web_search", "https://web.example/api/search")
serper.post.reset_mock()
assert_search(serper, "image_search", "https://images.example/v1/images")
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
@pytest.mark.parametrize("base_url", [None, "", " \t", 42, False, {}])
def test_unusable_tool_config_falls_back_to_environment(serper, monkeypatch, tool_name, route, base_url):
monkeypatch.setenv("SERPER_BASE_URL", " https://env.example/ ")
serper.configs[tool_name]["base_url"] = base_url
assert_search(serper, tool_name, f"https://env.example/{route}")
def test_web_endpoint_does_not_leak_to_image_tool(serper):
serper.configs["web_search"]["base_url"] = "https://web.example"
assert_search(serper, "image_search", "https://google.serper.dev/images")
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
def test_environment_endpoint_without_tool_config(serper, monkeypatch, tool_name, route):
monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: None))
monkeypatch.setenv("SERPER_API_KEY", serper.configs[tool_name]["api_key"])
monkeypatch.setenv("SERPER_BASE_URL", " https://env.example/ ")
assert_search(serper, tool_name, f"https://env.example/{route}")
def test_endpoint_is_operator_configuration_only():
assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"}
assert set(tools.image_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results"}
@pytest.mark.parametrize("tool_name,route", [("web_search", "search"), ("image_search", "images")])
@pytest.mark.parametrize("has_tool_config", [True, False], ids=["tool-config", "environment-only"])
def test_endpoint_and_key_use_one_config_snapshot(serper, monkeypatch, tool_name, route, has_tool_config):
serper.configs[tool_name]["base_url"] = "https://first.example/api"
monkeypatch.setenv("SERPER_BASE_URL", "https://first.example/api")
monkeypatch.setenv("SERPER_API_KEY", serper.configs[tool_name]["api_key"])
initial_tool = SimpleNamespace(model_extra=serper.configs[tool_name]) if has_tool_config else None
first = SimpleNamespace(get_tool_config=MagicMock(return_value=initial_tool))
reloaded = SimpleNamespace(get_tool_config=MagicMock(return_value=SimpleNamespace(model_extra={"api_key": "other-provider-secret", "base_url": "https://second.example/api"})))
get_config = MagicMock(side_effect=[first, reloaded])
monkeypatch.setattr(tools, "get_app_config", get_config)
assert_search(serper, tool_name, f"https://first.example/api/{route}")
get_config.assert_called_once_with()
first.get_tool_config.assert_called_once_with(tool_name)
reloaded.get_tool_config.assert_not_called()
@pytest.mark.parametrize("tool_name", ["web_search", "image_search"])
@pytest.mark.parametrize("source", ["tool-config", "environment"])
@pytest.mark.parametrize(
"base_url",
[
"///",
"ftp://proxy.example",
"https:///missing-host",
"https://:8443",
"https://proxy.example/api?token=URL_SECRET",
"https://proxy.example/api#URL_SECRET",
pytest.param("https://proxy.example/api?", id="empty-query"),
pytest.param("https://proxy.example/api#", id="empty-fragment"),
"https://[invalid/",
"https://proxy.example:invalid/api",
],
)
def test_invalid_base_url_fails_before_transport_without_exposing_value(serper, monkeypatch, caplog, tool_name, source, base_url):
if source == "tool-config":
serper.configs[tool_name]["base_url"] = base_url
# Invalid explicit overrides must not send the key to a fallback host.
monkeypatch.setenv("SERPER_BASE_URL", "https://fallback.example")
else:
monkeypatch.setenv("SERPER_BASE_URL", base_url)
search_tool = tools.web_search_tool if tool_name == "web_search" else tools.image_search_tool
result = json.loads(search_tool.invoke({"query": " test "}))
assert "base_url" in result["error"]
assert "SERPER_BASE_URL" in result["error"]
assert result["query"] == "test"
assert "URL_SECRET" not in json.dumps(result)
assert "URL_SECRET" not in caplog.text
serper.client.assert_not_called()
serper.post.assert_not_called()

View File

@ -1,6 +1,7 @@
"""Unit tests for the Serper community web search tool."""
import json
import logging
from unittest.mock import MagicMock, patch
import httpx
@ -8,10 +9,11 @@ import pytest
@pytest.fixture(autouse=True)
def reset_api_key_warned():
def reset_api_key_warned(monkeypatch):
"""Reset the module-level warning flag before each test."""
import deerflow.community.serper.tools as serper_mod
monkeypatch.delenv("SERPER_BASE_URL", raising=False)
serper_mod._api_key_warned = set()
yield
serper_mod._api_key_warned = set()
@ -49,6 +51,98 @@ def _make_serper_images_response(images: list) -> MagicMock:
return mock_resp
class TestSerperBaseUrl:
@pytest.mark.parametrize(("tool_name", "route"), [("web_search_tool", "search"), ("image_search_tool", "images")])
@pytest.mark.parametrize(
("base_url", "expected_base"),
[
(None, "https://google.serper.dev"),
("", "https://google.serper.dev"),
(" \t\r\n", "https://google.serper.dev"),
("https://proxy.example", "https://proxy.example"),
("https://proxy.example/", "https://proxy.example"),
(" \thttps://proxy.example/// \r\n", "https://proxy.example"),
("https://proxy.example/api/v1/", "https://proxy.example/api/v1"),
("http://127.0.0.1:8080/", "http://127.0.0.1:8080"),
],
)
def test_tools_post_to_configured_endpoint(self, monkeypatch, mock_config_with_key, tool_name: str, route: str, base_url: str | None, expected_base: str) -> None:
import deerflow.community.serper.tools as serper_mod
if base_url is not None:
monkeypatch.setenv("SERPER_BASE_URL", base_url)
response = MagicMock()
response.json.return_value = {
"organic": [{"title": "Example", "link": "https://example.com", "snippet": "Result"}],
"images": [{"title": "Example", "imageUrl": "https://example.com/image.jpg"}],
}
with patch("deerflow.community.serper.tools.httpx.Client") as client_class:
post = client_class.return_value.__enter__.return_value.post
post.return_value = response
result = json.loads(getattr(serper_mod, tool_name).invoke({"query": "https://model-input.invalid"}))
assert "error" not in result
post.assert_called_once()
assert post.call_args.args[0] == f"{expected_base}/{route}"
assert post.call_args.kwargs["headers"] == {"X-API-KEY": "test-serper-key", "Content-Type": "application/json"}
assert post.call_args.kwargs["json"] == {"q": "https://model-input.invalid", "num": 5}
assert "params" not in post.call_args.kwargs
assert "test-serper-key" not in post.call_args.args[0]
@pytest.mark.parametrize("route", ["search", "images"])
def test_endpoint_is_resolved_before_transport(self, monkeypatch, route: str) -> None:
import deerflow.community.serper.tools as serper_mod
monkeypatch.setenv("SERPER_BASE_URL", "https://first.example/")
with patch("deerflow.community.serper.tools.httpx.Client") as client_class:
client = MagicMock()
client.post.return_value = _make_serper_response([])
def enter_client():
monkeypatch.setenv("SERPER_BASE_URL", "https://later.example/")
return client
client_class.return_value.__enter__.side_effect = enter_client
data, error = serper_mod._serper_post(f"https://google.serper.dev/{route}", "test-key", "test", 5, time_range="week")
assert error is None
assert data == {"organic": []}
assert client.post.call_args.args[0] == f"https://first.example/{route}"
assert client.post.call_args.kwargs["json"] == {"q": "test", "num": 5, "tbs": "qdr:w"}
@pytest.mark.parametrize("route", ["search", "images"])
@pytest.mark.parametrize("redact_urls", [False, True], ids=["plain-log", "redacted-log"])
def test_override_diagnostic_omits_credentials(self, monkeypatch, caplog, route: str, redact_urls: bool) -> None:
import deerflow.community.serper.tools as serper_mod
from deerflow.logging_config import UrlRedactionFilter
if redact_urls:
# Production logging installs this filter on root handlers; another
# test may already have enabled it before this test runs in a shard.
monkeypatch.setattr(caplog.handler, "filters", [*caplog.handler.filters, UrlRedactionFilter()])
monkeypatch.setenv("SERPER_BASE_URL", "https://user:password@proxy.example:8443/api/v1/")
with caplog.at_level(logging.DEBUG, logger=serper_mod.__name__), patch("deerflow.community.serper.tools.httpx.Client") as client_class:
post = client_class.return_value.__enter__.return_value.post
post.return_value = _make_serper_response([])
_, error = serper_mod._serper_post(f"https://google.serper.dev/{route}", "test-secret-api-key", "test", 5)
assert error is None
post.assert_called_once()
assert post.call_args.args[0] == f"https://user:password@proxy.example:8443/api/v1/{route}"
records = [record for record in caplog.records if "SERPER_BASE_URL" in record.getMessage()]
assert len(records) == 1
diagnostic = records[0].getMessage()
assert diagnostic in {
f"Serper endpoint from base_url/SERPER_BASE_URL: https://proxy.example:8443/api/v1/{route}",
"Serper endpoint from base_url/SERPER_BASE_URL: https://proxy.example:8443/<redacted>",
}
if redact_urls:
assert "<redacted>" in diagnostic
assert "user:password" not in caplog.text
assert "test-secret-api-key" not in caplog.text
class TestGetApiKey:
def test_returns_config_key_when_present(self):
with patch("deerflow.community.serper.tools.get_app_config") as mock:

View File

@ -933,6 +933,7 @@ tools:
# group: web
# use: deerflow.community.serper.tools:web_search_tool
# max_results: 5 # capped at 10 by the Serper provider
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /search
# include_domains: [] # Optional domain-only lists, at most 10 entries each
# exclude_domains: [] # Exclusion wins; see backend/docs/CONFIGURATION.md
# # api_key: $SERPER_API_KEY # Optional explicit env-var reference
@ -1275,6 +1276,7 @@ tools:
# group: web
# use: deerflow.community.serper.tools:image_search_tool
# max_results: 5 # capped at 10 by the Serper provider
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /images
# # api_key: $SERPER_API_KEY # Optional explicit env-var reference
# Image search tool (uses Brave Image Search API, requires BRAVE_SEARCH_API_KEY)