diff --git a/README.md b/README.md index 3d223f8f1c..01aea9f8e3 100644 --- a/README.md +++ b/README.md @@ -1286,6 +1286,14 @@ These are deployment settings; the model still supplies only `query` and optiona empty list is forwarded and imposes no restriction of that kind. See the [tool configuration example](backend/docs/CONFIGURATION.md#tools). +Serper `web_search` supports deployment-level `include_domains` and +`exclude_domains` too. It checks returned URL hosts (including subdomains), with +exclusion taking precedence. Filters can return fewer results, including zero; +there are no refill requests. This selects sources, not factual accuracy or a +global URL-access policy. The model arguments and image search are unchanged. +See [Serper configuration](backend/docs/CONFIGURATION.md#serper-source-filters) +for validation and query-length limits. + When using Tavily for `web_fetch`, extracted pages without a title use their URL as the heading; their content remains available to the agent. Chat tool-step titles accept leading blank lines and up to three spaces before diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index 7e3fb3aa9f..c1dd4eedda 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -615,9 +615,62 @@ the existing contract. Serper `web_search` also accepts the optional model argument `time_range: "day" | "week" | "month" | "year"`. For example, `{"query": "Python releases", "time_range": "week"}` sends `tbs: "qdr:w"` -to Serper. Omitting `time_range` or passing `null` keeps the existing unrestricted +to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the search request. This option does not change Serper `image_search`. +#### Serper source filters + +```yaml +tools: + - name: web_search + group: web + use: deerflow.community.serper.tools:web_search_tool + max_results: 5 + include_domains: [example.com, bücher.de] + exclude_domains: [ads.example.com] +``` + +Each optional list accepts at most 10 entries (before deduplication). Omitted or +empty lists impose no restriction; explicit `null`, non-lists, or any invalid +entry return a configuration error before HTTP. Validation failures are also +logged without query or configured domain values. Entries must be domain names: +no surrounding whitespace, scheme, path, port, wildcard, IP literal or query +operator. Names are lowercased, one trailing dot is removed, and Python's IDNA +codec converts Unicode names to ASCII. DNS labels must be 1–63 characters and +the normalized domain at most 253 characters, with at least two labels. +Duplicates are removed after normalization. + +Matching uses the exact hostname or a dot-delimited subdomain; `example.com` +does not match `notexample.com` or `example.com.evil.com`. Exclusion wins over +inclusion. With either list non-empty, malformed/non-HTTP(S) result URLs, +credentials in URLs, and invalid hosts or ports are discarded. No DNS lookup +or redirect resolution is performed. Unconfigured/empty-filter behavior stays +unchanged, including the legacy query trimming and 500-character truncation. + +The adapter appends Google query operators, for example +`(news) (site:example.com OR site:example.org) -site:ads.example.com`, in the +existing Serper `q` field; it sends no provider-specific domain JSON fields. +After the existing query cleanup, the complete filtered query must fit 500 +characters or the tool returns an error before HTTP. Operators are never +truncated or dropped. Model-supplied operators can affect upstream retrieval, +so the local hostname check always enforces the configured scope. The returned +`query` remains the cleaned original query, without appended restrictions, +including on provider errors. + +`time_range` still maps to `tbs`, and `max_results` caps the filtered results. +`total_results` reports the actual remaining count (zero with `results: []` +when none survive). One request is made: no refill or relaxed-filter retries. +These settings do not affect `image_search` or the model-facing tool schema. +Source selection is neither a factuality guarantee nor a global URL-access +policy for fetch tools, browsers, or redirects. + +Provider validation: [Google documents `site:` and subdomain behavior](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site) +and [search exclusion syntax](https://support.google.com/websearch/answer/2466433). +[Serper describes its Google Search API](https://serper.dev/), but live Serper +operator handling has not been verified here. Upstream operators are best-effort; +offline mocked tests verify request composition and local filtering, not provider +retrieval behavior. No paid API calls are needed for the regression suite. + **Built-in Tools**: - `web_search` - Search the web (DuckDuckGo, Tavily, Brave, Serper, Serply, Exa, InfoQuest, Tencent Cloud WSA, Firecrawl, fastCRW, GroundRoute, Sofya) - `web_fetch` - Fetch web pages (Jina AI, Crawl4AI, Exa, InfoQuest, Firecrawl, fastCRW, GroundRoute, Browserless, Sofya, Unbrowse) diff --git a/backend/packages/harness/deerflow/community/serper/AGENTS.md b/backend/packages/harness/deerflow/community/serper/AGENTS.md new file mode 100644 index 0000000000..b379d841d1 --- /dev/null +++ b/backend/packages/harness/deerflow/community/serper/AGENTS.md @@ -0,0 +1,17 @@ +# Serper provider + +`tools.py` owns web/image search. Keep model-facing arguments unchanged. +Web-only `include_domains`/`exclude_domains` use at most 10 domain-only entries +per list; normalize case, one trailing dot and IDNA, reject invalid configuration +before transport, log validation errors without config/query values, and enforce +exact-host/dot-subdomain matching with deny precedence. +Google `site:` query operators are best-effort; local URL-host filtering is +mandatory even for queries containing operators. Never truncate restrictions, +refill results, or relax scope. The composed query limit is 500 characters; +report the cleaned original query and actual filtered count, including zero. +Image search and unconfigured web behavior retain their existing contracts. +This is source selection, not a fetch policy or factuality check. + +Tests: `backend/tests/test_serper_domain_filters.py` and `test_serper_tools.py`. +Mock HTTP; live Serper semantics remain unverified. See +`backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract. diff --git a/backend/packages/harness/deerflow/community/serper/tools.py b/backend/packages/harness/deerflow/community/serper/tools.py index bee14d8f14..a041c32330 100644 --- a/backend/packages/harness/deerflow/community/serper/tools.py +++ b/backend/packages/harness/deerflow/community/serper/tools.py @@ -8,6 +8,7 @@ API. An API key is required. Sign up at https://serper.dev to get one. import json import logging import os +import re from ipaddress import IPv4Address, ip_address from urllib.parse import urlparse @@ -91,6 +92,70 @@ def _clean_query(query: str) -> str: return query +def _normalize_domain(value: object) -> str: + """Accept DNS domains only, using Python's IDNA codec consistently.""" + if not isinstance(value, str) or not value or value != value.strip(): + raise ValueError("expected a domain string") + domain = value.removesuffix(".").lower() + try: + domain = domain.encode("idna").decode("ascii") + # Validate existing A-labels too (the encoder otherwise passes them through). + domain.encode("ascii").decode("idna") + except UnicodeError as exc: + raise ValueError("invalid IDNA domain") from exc + labels = domain.split(".") + if len(domain) > 253 or len(labels) < 2 or any(not re.fullmatch(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?", label) for label in labels): + raise ValueError("expected a DNS domain without scheme, port, path or operators") + try: + ip_address(domain) + except ValueError: + if _decode_ipv4(domain) is None: + return domain + raise ValueError("IP literals are not domains") + + +def _domain_list(extra: dict, name: str) -> list[str]: + values = extra.get(name, []) + if not isinstance(values, list) or len(values) > 10: + raise ValueError(f"{name} must be a list of at most 10 domains") + try: + return list(dict.fromkeys(_normalize_domain(value) for value in values)) + except ValueError as exc: + raise ValueError(f"Invalid {name}: {exc}") from exc + + +def _domain_query(query: str, include: list[str], exclude: list[str]) -> str: + if not include and not exclude: + return query + parts = [f"({query})"] + if include: + parts.append("(" + " OR ".join(f"site:{domain}" for domain in include) + ")") + parts.extend(f"-site:{domain}" for domain in exclude) + scoped = " ".join(parts) + if len(scoped) > 500: + raise ValueError("Serper query with domain filters exceeds 500 characters; shorten the query or domain lists") + return scoped + + +def _matches_domain_scope(value: object, include: list[str], exclude: list[str]) -> bool: + # This is source selection, not DNS resolution or a fetch permission check. + if not isinstance(value, str) or any(ord(char) <= 32 or ord(char) == 127 for char in value) or "\\" in value: + return False + try: + parsed = urlparse(value) + if parsed.scheme not in {"http", "https"} or not parsed.netloc or parsed.username is not None or parsed.password is not None: + return False + _ = parsed.port # Reject malformed/out-of-range ports. + host = _normalize_domain(parsed.hostname) + except ValueError: + return False + + def matches(domains: list[str]) -> bool: + return any(host == domain or host.endswith("." + domain) for domain in domains) + + return not matches(exclude) and (not include or matches(include)) + + def _decode_ipv4(host: str) -> IPv4Address | None: """Decode obfuscated IPv4 literals that ``ip_address`` rejects. @@ -234,20 +299,34 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang max_results = _coerce_max_results(max_results) query = _clean_query(query) + try: + extra = config.model_extra if config is not None else {} + include = _domain_list(extra, "include_domains") + exclude = _domain_list(extra, "exclude_domains") + search_query = _domain_query(query, include, exclude) + except ValueError as exc: + logger.error("Invalid Serper domain filters: %s", exc) + return json.dumps({"error": str(exc), "query": query}, ensure_ascii=False) + api_key = _get_api_key("web_search") if not api_key: return _missing_key_error(query, "web_search") - data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, query, max_results, time_range=time_range) + data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range) if error_json is not None: - return error_json + error = json.loads(error_json) + error["query"] = query + return json.dumps(error, ensure_ascii=False) organic, error_json = _response_items(data, "organic", query) if error_json is not None: return error_json - if not organic: + if not organic and not (include or exclude): return json.dumps({"error": "No results found", "query": query}, ensure_ascii=False) + if include or exclude: + organic = [result for result in organic if _matches_domain_scope(result.get("link"), include, exclude)] + # Search result links are returned verbatim (not passed through # _safe_public_url): they are surfaced as citations for the model to read, # not fetched/downloaded by this tool, unlike image_search image URLs. diff --git a/backend/tests/test_agent_guidance_check.py b/backend/tests/test_agent_guidance_check.py index 6213a76fa3..8df3ac72ef 100644 --- a/backend/tests/test_agent_guidance_check.py +++ b/backend/tests/test_agent_guidance_check.py @@ -21,6 +21,7 @@ EXPECTED_GUIDANCE_PATHS = { "backend/packages/harness/deerflow/agents/task_continuity/AGENTS.md", "backend/packages/harness/deerflow/community/jina_ai/AGENTS.md", "backend/packages/harness/deerflow/community/ragflow/AGENTS.md", + "backend/packages/harness/deerflow/community/serper/AGENTS.md", "backend/packages/harness/deerflow/community/tavily/AGENTS.md", "backend/packages/harness/deerflow/community/e2b_sandbox/AGENTS.md", "backend/packages/harness/deerflow/community/aio_sandbox/AGENTS.md", diff --git a/backend/tests/test_serper_domain_filters.py b/backend/tests/test_serper_domain_filters.py new file mode 100644 index 0000000000..9e87b53e82 --- /dev/null +++ b/backend/tests/test_serper_domain_filters.py @@ -0,0 +1,207 @@ +"""Offline source-selection contract; no real Serper requests.""" + +import json +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from deerflow.community.serper import tools + + +@pytest.fixture +def search(monkeypatch): + post = MagicMock() + response = post.return_value + response.json.return_value = {"organic": []} + client = MagicMock() + client.return_value.__enter__.return_value.post = post + monkeypatch.setattr(tools.httpx, "Client", client) + + def run(config=None, urls=(), query="news", **arguments): + extra = {"api_key": "dummy", **(config or {})} + monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(model_extra=extra))) + response.json.return_value = {"organic": [{"link": url} for url in urls]} + return json.loads(tools.web_search_tool.invoke({"query": query, **arguments})) + + run.post = post + run.client = client + return run + + +@pytest.mark.parametrize("config", [{}, {"include_domains": [], "exclude_domains": []}]) +def test_empty_preserves_behavior(search, config): + result = search(config, ["not a URL"], query=" news ") + assert result["results"][0]["url"] == "not a URL" + assert search.post.call_args.kwargs["json"] == {"q": "news", "num": 5} + + +@pytest.mark.parametrize("field", ["include_domains", "exclude_domains"]) +@pytest.mark.parametrize( + "value", + [ + None, + "example.com", + {}, + True, + [None], + [1], + [""], + [" example.com"], + ["https://example.com"], + ["*.example.com"], + ["example.com/path"], + ["example.com:443"], + ["a@b.com"], + ["a..com"], + ["-a.com"], + ["a_.com"], + ["localhost"], + ["127.0.0.1"], + ["example.com OR site:evil.com"], + ["a" * 64 + ".com"], + ["a.com"] * 11, + ], +) +def test_invalid_fails_before_transport(search, field, value): + result = search({field: value}, ["https://example.com"]) + assert "error" in result + assert field in result["error"] + search.client.assert_not_called() + + +def test_scope_normalization_boundary_and_precedence(search): + urls = ["https://EXAMPLE.com./a", "https://sub.example.com/a", "https://bad.example.com/a", "https://deep.bad.example.com", "https://notexample.com", "https://example.com.evil.com", "https://bücher.de/a", "https://xn--bcher-kva.de/b"] + result = search({"include_domains": ["Example.COM.", "bücher.de", "example.com"], "exclude_domains": ["bad.example.com"]}, urls, max_results=10, time_range="week") + assert [r["url"] for r in result["results"]] == [urls[i] for i in [0, 1, 6, 7]] + assert result["total_results"] == 4 + assert result["query"] == "news" + assert search.post.call_args.kwargs["json"] == {"q": "(news) (site:example.com OR site:xn--bcher-kva.de) -site:bad.example.com", "num": 10, "tbs": "qdr:w"} + search.post.assert_called_once() + + +@pytest.mark.parametrize( + "url", + [ + None, + 42, + "", + "example.com", + "//example.com", + "ftp://example.com", + "https://[broken", + "https://example.com:bad", + "https://example.com:99999", + "https://user@example.com", + "https://example.com\\@evil.com", + "https://exa\nmple.com", + "https://example.com..", + "https://%65xample.com", + "https://a_.com", + ], +) +def test_malformed_urls_removed_even_with_deny_only(search, url): + result = search({"exclude_domains": ["blocked.com"]}, [url]) + assert result == {"query": "news", "total_results": 0, "results": []} + + +def test_deny_only_and_limit_after_filtering(search): + result = search({"exclude_domains": ["blocked.com"], "max_results": 1}, ["https://blocked.com", "https://other.com", "https://third.com"]) + assert result["total_results"] == 1 + assert result["results"][0]["url"] == "https://other.com" + assert search.post.call_args.kwargs["json"] == {"q": "(news) -site:blocked.com", "num": 1} + search.post.assert_called_once() + + +@pytest.mark.parametrize("query", ["news OR site:evil.com", "news) OR (site:evil.com", '"unfinished', "-site:example.com"]) +def test_query_operators_cannot_bypass_local_filter(search, query): + result = search({"include_domains": ["example.com"]}, ["https://evil.com", "https://example.com"], query=query) + assert result["query"] == query + assert [r["url"] for r in result["results"]] == ["https://example.com"] + + +def test_query_length_boundary(search): + suffix = ") (site:example.com)" + query = "x" * (500 - len(suffix) - 1) + result = search({"include_domains": ["example.com"]}, ["https://example.com"], query=query) + assert result["query"] == query + assert len(search.post.call_args.kwargs["json"]["q"]) == 500 + search.client.reset_mock() + result = search({"include_domains": ["example.com"]}, query=query + "x") + assert "error" in result + search.client.assert_not_called() + + +def test_filters_alone_exceed_query_budget(search): + result = search({"include_domains": ["a" * 60 + f".{i}.com" for i in range(10)]}) + assert "error" in result + search.client.assert_not_called() + + +def test_filtered_empty_and_schema(search): + result = search({"include_domains": ["example.com"], "exclude_domains": ["example.com"]}, ["https://example.com"]) + assert result["total_results"] == 0 + assert result["results"] == [] + assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"} + + +def test_image_search_ignores_domain_settings(search): + search({"include_domains": "invalid"}) + search.post.reset_mock() + tools.image_search_tool.invoke({"query": "images"}) + assert search.post.call_args.kwargs["json"] == {"q": "images", "num": 5} + + +@pytest.mark.parametrize("time_range,tbs", [(None, None), ("day", "qdr:d"), ("month", "qdr:m"), ("year", "qdr:y")]) +def test_empty_provider_response_and_recency(search, time_range, tbs): + result = search({"include_domains": ["example.com"]}, time_range=time_range) + assert result == {"query": "news", "total_results": 0, "results": []} + payload = search.post.call_args.kwargs["json"] + assert payload.get("tbs") == tbs + if tbs is None: + assert "tbs" not in payload + + +def test_provider_error_preserves_original_query(search): + search.post.side_effect = RuntimeError("provider unavailable") + result = search({"include_domains": ["example.com"]}, query=" news ") + assert result == {"query": "news", "error": "provider unavailable"} + + +@pytest.mark.parametrize("domain", ["example.com..", "xn--.com", "a" * 63 + "." + "b" * 63 + "." + "c" * 63 + "." + "d" * 62]) +def test_invalid_domain_boundaries(search, domain): + assert "error" in search({"include_domains": [domain]}) + search.client.assert_not_called() + + +def test_maximum_domain_length(search): + domain = ".".join(["a" * 63, "b" * 63, "c" * 63, "d" * 61]) + result = search({"include_domains": [domain]}, ["https://" + domain]) + assert result["total_results"] == 1 + + +@pytest.mark.parametrize( + "config,query,error_fragment", + [ + ({"include_domains": "private.example"}, "private-search-text", "include_domains"), + ({"exclude_domains": ["https://private.example/secret"]}, "private-search-text", "exclude_domains"), + ({"include_domains": ["example.com"]}, "private-search-text" * 40, "500 characters"), + ], +) +def test_domain_validation_errors_are_logged_without_input_values(search, caplog, config, query, error_fragment): + with caplog.at_level("ERROR", logger=tools.__name__): + result = search(config, query=query) + assert error_fragment in result["error"] + search.client.assert_not_called() + assert len(caplog.records) == 1 + assert caplog.records[0].levelname == "ERROR" + assert error_fragment in caplog.text + assert "private.example" not in caplog.text + assert "private-search-text" not in caplog.text + + +def test_valid_domain_settings_do_not_log_errors(search, caplog): + with caplog.at_level("ERROR", logger=tools.__name__): + result = search({"include_domains": ["example.com"]}, ["https://example.com"]) + assert result["total_results"] == 1 + assert not caplog.records diff --git a/config.example.yaml b/config.example.yaml index aa6747a4b8..092e218f7d 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -20,7 +20,7 @@ # ============================================================================ # Bump this number when the config schema changes. # Run `make config-upgrade` to merge new fields into your local config.yaml. -config_version: 51 +config_version: 52 # Optional operator-owned literal prompt extensions (see CONFIGURATION.md): # lead_prompt_overlay: @@ -917,6 +917,8 @@ tools: # group: web # use: deerflow.community.serper.tools:web_search_tool # max_results: 5 # capped at 10 by the Serper provider + # include_domains: [] # Optional domain-only lists, at most 10 entries each + # exclude_domains: [] # Exclusion wins; see backend/docs/CONFIGURATION.md # # api_key: $SERPER_API_KEY # Optional explicit env-var reference # Web search tool (uses Serply - Google Search API, requires SERPLY_API_KEY) diff --git a/deploy/helm/deer-flow/README.md b/deploy/helm/deer-flow/README.md index 728b2150d2..0b290d3159 100644 --- a/deploy/helm/deer-flow/README.md +++ b/deploy/helm/deer-flow/README.md @@ -135,7 +135,7 @@ they resolve from the `secrets` map): ```yaml config: | - config_version: 51 + config_version: 52 models: - name: gpt-4 use: langchain_openai:ChatOpenAI diff --git a/deploy/helm/deer-flow/values.yaml b/deploy/helm/deer-flow/values.yaml index b874d3c81a..cc593b032c 100644 --- a/deploy/helm/deer-flow/values.yaml +++ b/deploy/helm/deer-flow/values.yaml @@ -263,7 +263,7 @@ ingress: # -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never # inline literal secret values here. The default enables provisioner sandbox. config: | - config_version: 51 + config_version: 52 log_level: info recursion_limit: 100 max_recursion_limit: 1000