feat(serper): add configured source-domain filters (#6228)

* feat(serper): add configured source-domain filters

Signed-off-by: 97three <2212371308@qq.com>

* fix(helm): align embedded config version with example

Signed-off-by: 97three <2212371308@qq.com>

---------

Signed-off-by: 97three <2212371308@qq.com>

README.md

@@ -1286,6 +1286,14 @@ These are deployment settings; the model still supplies only `query` and optiona
 empty list is forwarded and imposes no restriction of that kind. See the
 [tool configuration example](backend/docs/CONFIGURATION.md#tools).
 
+Serper `web_search` supports deployment-level `include_domains` and
+`exclude_domains` too. It checks returned URL hosts (including subdomains), with
+exclusion taking precedence. Filters can return fewer results, including zero;
+there are no refill requests. This selects sources, not factual accuracy or a
+global URL-access policy. The model arguments and image search are unchanged.
+See [Serper configuration](backend/docs/CONFIGURATION.md#serper-source-filters)
+for validation and query-length limits.
+
 When using Tavily for `web_fetch`, extracted pages without a title use their URL
 as the heading; their content remains available to the agent.
 Chat tool-step titles accept leading blank lines and up to three spaces before

backend/docs/CONFIGURATION.md

@@ -615,9 +615,62 @@ the existing contract.
 Serper `web_search` also accepts the optional model argument
 `time_range: "day" | "week" | "month" | "year"`. For example,
 `{"query": "Python releases", "time_range": "week"}` sends `tbs: "qdr:w"`
-to Serper. Omitting `time_range` or passing `null` keeps the existing unrestricted
+to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the
 search request. This option does not change Serper `image_search`.
 
+#### Serper source filters
+
+```yaml
+tools:
+  - name: web_search
+    group: web
+    use: deerflow.community.serper.tools:web_search_tool
+    max_results: 5
+    include_domains: [example.com, bücher.de]
+    exclude_domains: [ads.example.com]
+```
+
+Each optional list accepts at most 10 entries (before deduplication). Omitted or
+empty lists impose no restriction; explicit `null`, non-lists, or any invalid
+entry return a configuration error before HTTP. Validation failures are also
+logged without query or configured domain values. Entries must be domain names:
+no surrounding whitespace, scheme, path, port, wildcard, IP literal or query
+operator. Names are lowercased, one trailing dot is removed, and Python's IDNA
+codec converts Unicode names to ASCII. DNS labels must be 1–63 characters and
+the normalized domain at most 253 characters, with at least two labels.
+Duplicates are removed after normalization.
+
+Matching uses the exact hostname or a dot-delimited subdomain; `example.com`
+does not match `notexample.com` or `example.com.evil.com`. Exclusion wins over
+inclusion. With either list non-empty, malformed/non-HTTP(S) result URLs,
+credentials in URLs, and invalid hosts or ports are discarded. No DNS lookup
+or redirect resolution is performed. Unconfigured/empty-filter behavior stays
+unchanged, including the legacy query trimming and 500-character truncation.
+
+The adapter appends Google query operators, for example
+`(news) (site:example.com OR site:example.org) -site:ads.example.com`, in the
+existing Serper `q` field; it sends no provider-specific domain JSON fields.
+After the existing query cleanup, the complete filtered query must fit 500
+characters or the tool returns an error before HTTP. Operators are never
+truncated or dropped. Model-supplied operators can affect upstream retrieval,
+so the local hostname check always enforces the configured scope. The returned
+`query` remains the cleaned original query, without appended restrictions,
+including on provider errors.
+
+`time_range` still maps to `tbs`, and `max_results` caps the filtered results.
+`total_results` reports the actual remaining count (zero with `results: []`
+when none survive). One request is made: no refill or relaxed-filter retries.
+These settings do not affect `image_search` or the model-facing tool schema.
+Source selection is neither a factuality guarantee nor a global URL-access
+policy for fetch tools, browsers, or redirects.
+
+Provider validation: [Google documents `site:` and subdomain behavior](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site)
+and [search exclusion syntax](https://support.google.com/websearch/answer/2466433).
+[Serper describes its Google Search API](https://serper.dev/), but live Serper
+operator handling has not been verified here. Upstream operators are best-effort;
+offline mocked tests verify request composition and local filtering, not provider
+retrieval behavior. No paid API calls are needed for the regression suite.
+
 **Built-in Tools**:
 - `web_search` - Search the web (DuckDuckGo, Tavily, Brave, Serper, Serply, Exa, InfoQuest, Tencent Cloud WSA, Firecrawl, fastCRW, GroundRoute, Sofya)
 - `web_fetch` - Fetch web pages (Jina AI, Crawl4AI, Exa, InfoQuest, Firecrawl, fastCRW, GroundRoute, Browserless, Sofya, Unbrowse)

backend/packages/harness/deerflow/community/serper/AGENTS.md

@@ -0,0 +1,17 @@
+# Serper provider
+
+`tools.py` owns web/image search. Keep model-facing arguments unchanged.
+Web-only `include_domains`/`exclude_domains` use at most 10 domain-only entries
+per list; normalize case, one trailing dot and IDNA, reject invalid configuration
+before transport, log validation errors without config/query values, and enforce
+exact-host/dot-subdomain matching with deny precedence.
+Google `site:` query operators are best-effort; local URL-host filtering is
+mandatory even for queries containing operators. Never truncate restrictions,
+refill results, or relax scope. The composed query limit is 500 characters;
+report the cleaned original query and actual filtered count, including zero.
+Image search and unconfigured web behavior retain their existing contracts.
+This is source selection, not a fetch policy or factuality check.
+
+Tests: `backend/tests/test_serper_domain_filters.py` and `test_serper_tools.py`.
+Mock HTTP; live Serper semantics remain unverified. See
+`backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract.

backend/packages/harness/deerflow/community/serper/tools.py

@@ -8,6 +8,7 @@ API. An API key is required. Sign up at https://serper.dev to get one.
 import json
 import logging
 import os
+import re
 from ipaddress import IPv4Address, ip_address
 from urllib.parse import urlparse
 
@@ -91,6 +92,70 @@ def _clean_query(query: str) -> str:
     return query
 
 
+def _normalize_domain(value: object) -> str:
+    """Accept DNS domains only, using Python's IDNA codec consistently."""
+    if not isinstance(value, str) or not value or value != value.strip():
+        raise ValueError("expected a domain string")
+    domain = value.removesuffix(".").lower()
+    try:
+        domain = domain.encode("idna").decode("ascii")
+        # Validate existing A-labels too (the encoder otherwise passes them through).
+        domain.encode("ascii").decode("idna")
+    except UnicodeError as exc:
+        raise ValueError("invalid IDNA domain") from exc
+    labels = domain.split(".")
+    if len(domain) > 253 or len(labels) < 2 or any(not re.fullmatch(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?", label) for label in labels):
+        raise ValueError("expected a DNS domain without scheme, port, path or operators")
+    try:
+        ip_address(domain)
+    except ValueError:
+        if _decode_ipv4(domain) is None:
+            return domain
+    raise ValueError("IP literals are not domains")
+
+
+def _domain_list(extra: dict, name: str) -> list[str]:
+    values = extra.get(name, [])
+    if not isinstance(values, list) or len(values) > 10:
+        raise ValueError(f"{name} must be a list of at most 10 domains")
+    try:
+        return list(dict.fromkeys(_normalize_domain(value) for value in values))
+    except ValueError as exc:
+        raise ValueError(f"Invalid {name}: {exc}") from exc
+
+
+def _domain_query(query: str, include: list[str], exclude: list[str]) -> str:
+    if not include and not exclude:
+        return query
+    parts = [f"({query})"]
+    if include:
+        parts.append("(" + " OR ".join(f"site:{domain}" for domain in include) + ")")
+    parts.extend(f"-site:{domain}" for domain in exclude)
+    scoped = " ".join(parts)
+    if len(scoped) > 500:
+        raise ValueError("Serper query with domain filters exceeds 500 characters; shorten the query or domain lists")
+    return scoped
+
+
+def _matches_domain_scope(value: object, include: list[str], exclude: list[str]) -> bool:
+    # This is source selection, not DNS resolution or a fetch permission check.
+    if not isinstance(value, str) or any(ord(char) <= 32 or ord(char) == 127 for char in value) or "\\" in value:
+        return False
+    try:
+        parsed = urlparse(value)
+        if parsed.scheme not in {"http", "https"} or not parsed.netloc or parsed.username is not None or parsed.password is not None:
+            return False
+        _ = parsed.port  # Reject malformed/out-of-range ports.
+        host = _normalize_domain(parsed.hostname)
+    except ValueError:
+        return False
+
+    def matches(domains: list[str]) -> bool:
+        return any(host == domain or host.endswith("." + domain) for domain in domains)
+
+    return not matches(exclude) and (not include or matches(include))
+
+
 def _decode_ipv4(host: str) -> IPv4Address | None:
     """Decode obfuscated IPv4 literals that ``ip_address`` rejects.
 
@@ -234,20 +299,34 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang
     max_results = _coerce_max_results(max_results)
     query = _clean_query(query)
 
+    try:
+        extra = config.model_extra if config is not None else {}
+        include = _domain_list(extra, "include_domains")
+        exclude = _domain_list(extra, "exclude_domains")
+        search_query = _domain_query(query, include, exclude)
+    except ValueError as exc:
+        logger.error("Invalid Serper domain filters: %s", exc)
+        return json.dumps({"error": str(exc), "query": query}, ensure_ascii=False)
+
     api_key = _get_api_key("web_search")
     if not api_key:
         return _missing_key_error(query, "web_search")
 
-    data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, query, max_results, time_range=time_range)
+    data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range)
     if error_json is not None:
-        return error_json
+        error = json.loads(error_json)
+        error["query"] = query
+        return json.dumps(error, ensure_ascii=False)
 
     organic, error_json = _response_items(data, "organic", query)
     if error_json is not None:
         return error_json
-    if not organic:
+    if not organic and not (include or exclude):
         return json.dumps({"error": "No results found", "query": query}, ensure_ascii=False)
 
+    if include or exclude:
+        organic = [result for result in organic if _matches_domain_scope(result.get("link"), include, exclude)]
+
     # Search result links are returned verbatim (not passed through
     # _safe_public_url): they are surfaced as citations for the model to read,
     # not fetched/downloaded by this tool, unlike image_search image URLs.

backend/tests/test_agent_guidance_check.py

@@ -21,6 +21,7 @@ EXPECTED_GUIDANCE_PATHS = {
     "backend/packages/harness/deerflow/agents/task_continuity/AGENTS.md",
     "backend/packages/harness/deerflow/community/jina_ai/AGENTS.md",
     "backend/packages/harness/deerflow/community/ragflow/AGENTS.md",
+    "backend/packages/harness/deerflow/community/serper/AGENTS.md",
     "backend/packages/harness/deerflow/community/tavily/AGENTS.md",
     "backend/packages/harness/deerflow/community/e2b_sandbox/AGENTS.md",
     "backend/packages/harness/deerflow/community/aio_sandbox/AGENTS.md",

backend/tests/test_serper_domain_filters.py

@@ -0,0 +1,207 @@
+"""Offline source-selection contract; no real Serper requests."""
+
+import json
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import pytest
+
+from deerflow.community.serper import tools
+
+
+@pytest.fixture
+def search(monkeypatch):
+    post = MagicMock()
+    response = post.return_value
+    response.json.return_value = {"organic": []}
+    client = MagicMock()
+    client.return_value.__enter__.return_value.post = post
+    monkeypatch.setattr(tools.httpx, "Client", client)
+
+    def run(config=None, urls=(), query="news", **arguments):
+        extra = {"api_key": "dummy", **(config or {})}
+        monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(model_extra=extra)))
+        response.json.return_value = {"organic": [{"link": url} for url in urls]}
+        return json.loads(tools.web_search_tool.invoke({"query": query, **arguments}))
+
+    run.post = post
+    run.client = client
+    return run
+
+
+@pytest.mark.parametrize("config", [{}, {"include_domains": [], "exclude_domains": []}])
+def test_empty_preserves_behavior(search, config):
+    result = search(config, ["not a URL"], query=" news ")
+    assert result["results"][0]["url"] == "not a URL"
+    assert search.post.call_args.kwargs["json"] == {"q": "news", "num": 5}
+
+
+@pytest.mark.parametrize("field", ["include_domains", "exclude_domains"])
+@pytest.mark.parametrize(
+    "value",
+    [
+        None,
+        "example.com",
+        {},
+        True,
+        [None],
+        [1],
+        [""],
+        [" example.com"],
+        ["https://example.com"],
+        ["*.example.com"],
+        ["example.com/path"],
+        ["example.com:443"],
+        ["a@b.com"],
+        ["a..com"],
+        ["-a.com"],
+        ["a_.com"],
+        ["localhost"],
+        ["127.0.0.1"],
+        ["example.com OR site:evil.com"],
+        ["a" * 64 + ".com"],
+        ["a.com"] * 11,
+    ],
+)
+def test_invalid_fails_before_transport(search, field, value):
+    result = search({field: value}, ["https://example.com"])
+    assert "error" in result
+    assert field in result["error"]
+    search.client.assert_not_called()
+
+
+def test_scope_normalization_boundary_and_precedence(search):
+    urls = ["https://EXAMPLE.com./a", "https://sub.example.com/a", "https://bad.example.com/a", "https://deep.bad.example.com", "https://notexample.com", "https://example.com.evil.com", "https://bücher.de/a", "https://xn--bcher-kva.de/b"]
+    result = search({"include_domains": ["Example.COM.", "bücher.de", "example.com"], "exclude_domains": ["bad.example.com"]}, urls, max_results=10, time_range="week")
+    assert [r["url"] for r in result["results"]] == [urls[i] for i in [0, 1, 6, 7]]
+    assert result["total_results"] == 4
+    assert result["query"] == "news"
+    assert search.post.call_args.kwargs["json"] == {"q": "(news) (site:example.com OR site:xn--bcher-kva.de) -site:bad.example.com", "num": 10, "tbs": "qdr:w"}
+    search.post.assert_called_once()
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        None,
+        42,
+        "",
+        "example.com",
+        "//example.com",
+        "ftp://example.com",
+        "https://[broken",
+        "https://example.com:bad",
+        "https://example.com:99999",
+        "https://user@example.com",
+        "https://example.com\\@evil.com",
+        "https://exa\nmple.com",
+        "https://example.com..",
+        "https://%65xample.com",
+        "https://a_.com",
+    ],
+)
+def test_malformed_urls_removed_even_with_deny_only(search, url):
+    result = search({"exclude_domains": ["blocked.com"]}, [url])
+    assert result == {"query": "news", "total_results": 0, "results": []}
+
+
+def test_deny_only_and_limit_after_filtering(search):
+    result = search({"exclude_domains": ["blocked.com"], "max_results": 1}, ["https://blocked.com", "https://other.com", "https://third.com"])
+    assert result["total_results"] == 1
+    assert result["results"][0]["url"] == "https://other.com"
+    assert search.post.call_args.kwargs["json"] == {"q": "(news) -site:blocked.com", "num": 1}
+    search.post.assert_called_once()
+
+
+@pytest.mark.parametrize("query", ["news OR site:evil.com", "news) OR (site:evil.com", '"unfinished', "-site:example.com"])
+def test_query_operators_cannot_bypass_local_filter(search, query):
+    result = search({"include_domains": ["example.com"]}, ["https://evil.com", "https://example.com"], query=query)
+    assert result["query"] == query
+    assert [r["url"] for r in result["results"]] == ["https://example.com"]
+
+
+def test_query_length_boundary(search):
+    suffix = ") (site:example.com)"
+    query = "x" * (500 - len(suffix) - 1)
+    result = search({"include_domains": ["example.com"]}, ["https://example.com"], query=query)
+    assert result["query"] == query
+    assert len(search.post.call_args.kwargs["json"]["q"]) == 500
+    search.client.reset_mock()
+    result = search({"include_domains": ["example.com"]}, query=query + "x")
+    assert "error" in result
+    search.client.assert_not_called()
+
+
+def test_filters_alone_exceed_query_budget(search):
+    result = search({"include_domains": ["a" * 60 + f".{i}.com" for i in range(10)]})
+    assert "error" in result
+    search.client.assert_not_called()
+
+
+def test_filtered_empty_and_schema(search):
+    result = search({"include_domains": ["example.com"], "exclude_domains": ["example.com"]}, ["https://example.com"])
+    assert result["total_results"] == 0
+    assert result["results"] == []
+    assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"}
+
+
+def test_image_search_ignores_domain_settings(search):
+    search({"include_domains": "invalid"})
+    search.post.reset_mock()
+    tools.image_search_tool.invoke({"query": "images"})
+    assert search.post.call_args.kwargs["json"] == {"q": "images", "num": 5}
+
+
+@pytest.mark.parametrize("time_range,tbs", [(None, None), ("day", "qdr:d"), ("month", "qdr:m"), ("year", "qdr:y")])
+def test_empty_provider_response_and_recency(search, time_range, tbs):
+    result = search({"include_domains": ["example.com"]}, time_range=time_range)
+    assert result == {"query": "news", "total_results": 0, "results": []}
+    payload = search.post.call_args.kwargs["json"]
+    assert payload.get("tbs") == tbs
+    if tbs is None:
+        assert "tbs" not in payload
+
+
+def test_provider_error_preserves_original_query(search):
+    search.post.side_effect = RuntimeError("provider unavailable")
+    result = search({"include_domains": ["example.com"]}, query=" news ")
+    assert result == {"query": "news", "error": "provider unavailable"}
+
+
+@pytest.mark.parametrize("domain", ["example.com..", "xn--.com", "a" * 63 + "." + "b" * 63 + "." + "c" * 63 + "." + "d" * 62])
+def test_invalid_domain_boundaries(search, domain):
+    assert "error" in search({"include_domains": [domain]})
+    search.client.assert_not_called()
+
+
+def test_maximum_domain_length(search):
+    domain = ".".join(["a" * 63, "b" * 63, "c" * 63, "d" * 61])
+    result = search({"include_domains": [domain]}, ["https://" + domain])
+    assert result["total_results"] == 1
+
+
+@pytest.mark.parametrize(
+    "config,query,error_fragment",
+    [
+        ({"include_domains": "private.example"}, "private-search-text", "include_domains"),
+        ({"exclude_domains": ["https://private.example/secret"]}, "private-search-text", "exclude_domains"),
+        ({"include_domains": ["example.com"]}, "private-search-text" * 40, "500 characters"),
+    ],
+)
+def test_domain_validation_errors_are_logged_without_input_values(search, caplog, config, query, error_fragment):
+    with caplog.at_level("ERROR", logger=tools.__name__):
+        result = search(config, query=query)
+    assert error_fragment in result["error"]
+    search.client.assert_not_called()
+    assert len(caplog.records) == 1
+    assert caplog.records[0].levelname == "ERROR"
+    assert error_fragment in caplog.text
+    assert "private.example" not in caplog.text
+    assert "private-search-text" not in caplog.text
+
+
+def test_valid_domain_settings_do_not_log_errors(search, caplog):
+    with caplog.at_level("ERROR", logger=tools.__name__):
+        result = search({"include_domains": ["example.com"]}, ["https://example.com"])
+    assert result["total_results"] == 1
+    assert not caplog.records

config.example.yaml

@@ -20,7 +20,7 @@
 # ============================================================================
 # Bump this number when the config schema changes.
 # Run `make config-upgrade` to merge new fields into your local config.yaml.
-config_version: 51
+config_version: 52
 
 # Optional operator-owned literal prompt extensions (see CONFIGURATION.md):
 # lead_prompt_overlay:
@@ -917,6 +917,8 @@ tools:
   #   group: web
   #   use: deerflow.community.serper.tools:web_search_tool
   #   max_results: 5   # capped at 10 by the Serper provider
+  #   include_domains: []  # Optional domain-only lists, at most 10 entries each
+  #   exclude_domains: []  # Exclusion wins; see backend/docs/CONFIGURATION.md
   #   # api_key: $SERPER_API_KEY  # Optional explicit env-var reference
 
   # Web search tool (uses Serply - Google Search API, requires SERPLY_API_KEY)

deploy/helm/deer-flow/README.md

@@ -135,7 +135,7 @@ they resolve from the `secrets` map):
 
 ```yaml
 config: |
-  config_version: 51
+  config_version: 52
   models:
     - name: gpt-4
       use: langchain_openai:ChatOpenAI

deploy/helm/deer-flow/values.yaml

@@ -263,7 +263,7 @@ ingress:
 # -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never
 #    inline literal secret values here. The default enables provisioner sandbox.
 config: |
-  config_version: 51
+  config_version: 52
   log_level: info
   recursion_limit: 100
   max_recursion_limit: 1000