diff --git a/README.md b/README.md index 4b6287b855..7afa038b3c 100644 --- a/README.md +++ b/README.md @@ -1472,6 +1472,13 @@ global URL-access policy. The model arguments and image search are unchanged. See [Serper configuration](backend/docs/CONFIGURATION.md#serper-source-filters) for validation and query-length limits. +Serper web and image search support opt-in transient recovery with +`max_retries: 1` (default `0`, maximum `3`) in each tool's configuration. +`retry_budget_seconds` bounds retry scheduling, not active synchronous requests. +Budget exhaustion is logged separately while the tool returns the last error. +Extra requests may consume quota or incur cost. See +[Serper retries](backend/docs/CONFIGURATION.md#serper-retries). + When using Tavily for `web_fetch`, extracted pages without a title use their URL as the heading; their content remains available to the agent. Chat tool-step titles accept leading blank lines and up to three spaces before diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index fa5ea01ac4..e7971a3992 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -734,6 +734,46 @@ Serper `web_search` also accepts the optional model argument to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the search request. This option does not change Serper `image_search`. +#### Serper retries + +Both Serper `web_search` and `image_search` accept these operator-only settings +in their respective `tools` entries (the model-facing arguments are unchanged): + +```yaml +max_retries: 0 # Default off; integer 0–3 extra requests +retry_budget_seconds: 30 # Default 30; finite number >0 and <=300 +``` + +Set `max_retries: 1` to enable recovery. Booleans, strings, nulls, out-of-range +values, and non-finite budgets return a structured configuration error before +HTTP. The three-retry cap limits additional quota/cost; the 300-second budget +cap limits scheduling waits. No billing or idempotency guarantee is claimed. + +Only connection establishment failures (`ConnectError`/`ConnectTimeout`) and +HTTP 502/503/504 are retried. HTTP 429 requires a valid `Retry-After` hint. +Authentication errors, other statuses, read/write timeouts, malformed successful +JSON and tool/result-processing failures are never retried. + +Backoff uses equal jitter: 0.25–0.5, 0.5–1, then 1–2 seconds. Valid nonnegative +integer seconds or HTTP-date `Retry-After` hints on 429/503 set a minimum wait +(past dates mean zero). HTTP dates include asctime and RFC 850 forms; RFC 850 +two-digit years resolve to the most recent matching year no more than 50 years +in the future. Invalid hints leave 503 on local backoff; +429 stays terminal. A hint or backoff that reaches/exceeds the remaining budget +returns the last error without another request or a shortened wait. +Both budget checks log `Serper retry time budget exhausted` at warning level, +distinguishing budget stops from exhausting all configured attempts without +changing the returned error. + +One monotonic scheduling deadline starts before client creation, counting elapsed +requests and waits; it is checked before waiting and again before a retry. +It does **not** interrupt active synchronous HTTP requests, response streaming, +or worker threads. HTTPX's unchanged 30-second timeout applies per network phase, +so this is **not** a hard end-to-end wall-clock deadline. The initial request is +always allowed. Payload, API key, domain restrictions and time range stay identical +across attempts; errors still report the cleaned original query. No refill, +cache, global limiter or model-level retry is added. + #### Serper endpoints To use a Serper-compatible provider for web and image search, set @@ -821,7 +861,7 @@ including on provider errors. `time_range` still maps to `tbs`, and `max_results` caps the filtered results. `total_results` reports the actual remaining count (zero with `results: []` -when none survive). One request is made: no refill or relaxed-filter retries. +when none survive). No refill or relaxed-filter requests are made. These settings do not affect `image_search` or the model-facing tool schema. Source selection is neither a factuality guarantee nor a global URL-access policy for fetch tools, browsers, or redirects. diff --git a/backend/packages/harness/deerflow/community/serper/AGENTS.md b/backend/packages/harness/deerflow/community/serper/AGENTS.md index 3584520d02..601869af7e 100644 --- a/backend/packages/harness/deerflow/community/serper/AGENTS.md +++ b/backend/packages/harness/deerflow/community/serper/AGENTS.md @@ -21,7 +21,7 @@ Read each tool's config once and pass captured extras to key resolution, including an empty mapping when config is absent, to keep endpoint/key paired through hot reload. Settings stay operator-only. Send keys only via `X-API-KEY`, and preserve result URL validation and model arguments. -Future retries must reuse the resolved endpoint across attempts. Debug endpoint +Retries must reuse the resolved endpoint across attempts. Debug endpoint diagnostics omit URL credentials, query and fragment; result-URL guards do not restrict the operator's API host. @@ -32,3 +32,16 @@ handlers can redact paths. Assert full endpoint routing on the captured HTTP request, and verify diagnostic host/port and credential omission in both modes. Mock HTTP; live Serper semantics remain unverified. See `backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract. + +`_serper_post` owns default-off physical retries (0–3) and one monotonic +scheduling budget (>0–300s), not cancellation of synchronous I/O. Keep config +validation before transport and retry only connect failures, 502/503/504 and +hinted 429. Provider-local hint parsing avoids coupling to Jina. Whole-tool +extension hooks cannot isolate this boundary. Test with real MockTransport +through both tools in `backend/tests/test_serper_retries.py`. +Endpoint/retry regressions change configuration and environment during backoff; +every attempt must keep the original provider URL, API key and request payload. +Budget aborts before and after backoff log a warning identifying the stage, +without query/key values, and preserve the last structured error. Date tests +pin asctime UTC anchoring (under a non-UTC host timezone where supported) and +the RFC 850 50-year cutoff down to the second. diff --git a/backend/packages/harness/deerflow/community/serper/tools.py b/backend/packages/harness/deerflow/community/serper/tools.py index d42ffeecff..92d629ac5d 100644 --- a/backend/packages/harness/deerflow/community/serper/tools.py +++ b/backend/packages/harness/deerflow/community/serper/tools.py @@ -7,9 +7,14 @@ API. An API key is required. Sign up at https://serper.dev to get one. import json import logging +import math import os +import random import re +import time from collections.abc import Mapping +from datetime import UTC, datetime +from email.utils import parsedate_to_datetime from ipaddress import IPv4Address, ip_address from urllib.parse import urlparse @@ -249,9 +254,73 @@ def _safe_public_url(value: object) -> str: return url if ip.is_global else "" -def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None, base_url: object | None = None) -> tuple[dict | None, str | None]: +def _retry_options(config) -> dict: + extra = config.model_extra if config is not None else {} + return {name: extra[name] for name in ("max_retries", "retry_budget_seconds") if name in extra} + + +_DAY = r"(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)" +_MONTH = r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)" +_CLOCK = r"[0-9]{2}:[0-9]{2}:[0-9]{2}" +_HTTP_DATE = re.compile( + rf"(?:{_DAY}, [0-9]{{2}} {_MONTH} [0-9]{{4}} {_CLOCK} GMT" + rf"|(?:Monday|Tuesday|Wednesday|Thursday|Friday|Saturday|Sunday), [0-9]{{2}}-{_MONTH}-[0-9]{{2}} {_CLOCK} GMT" + rf"|{_DAY} {_MONTH} (?:[0-9]{{2}}| [0-9]) {_CLOCK} [0-9]{{4}})" +) + + +def _retry_after(response: httpx.Response) -> float | None: + """Return a server floor, or None for an invalid HTTP Retry-After value.""" + value = response.headers.get("Retry-After") + if value is None: + return None + value = value.strip(" \t") + if value and value.isascii() and value.isdecimal(): + digits = value.lstrip("0") or "0" + # Bound integer conversion work. Infinity is a valid, unfit floor, not + # a parsing failure that could cause an early fallback retry. + if len(digits) > 309: + return math.inf + seconds = int(digits) + try: + floor = float(seconds) + except OverflowError: + return math.inf + return math.nextafter(floor, math.inf) if floor < seconds else floor + if not _HTTP_DATE.fullmatch(value): + return None + try: + date = parsedate_to_datetime(value).replace(tzinfo=UTC) + now = time.time() + if "-" in value: + # RFC 850 two-digit years: choose the most recent matching year + # no more than 50 years in the future (RFC 9110 section 5.6.7). + current = datetime.fromtimestamp(now, UTC) + year = (current.year + 50) // 100 * 100 + date.year % 100 + if (year, date.month, date.day, date.hour, date.minute, date.second) > (current.year + 50, current.month, current.day, current.hour, current.minute, current.second): + year -= 100 + date = date.replace(year=year) + return max(0.0, date.timestamp() - now) + except (ValueError, OverflowError): + return None + + +def _serper_post( + endpoint: str, + api_key: str, + query: str, + max_results: int, + *, + time_range: SearchTimeRange | None = None, + base_url: object | None = None, + max_retries: int = 0, + retry_budget_seconds: float = 30, +) -> tuple[dict | None, str | None]: """Send a POST request to a Serper endpoint. + Retries share a scheduling deadline, not a hard synchronous I/O deadline. + HTTPX timeouts remain per phase; an active request cannot be cancelled here. + ``query`` is expected to already be normalized via :func:`_clean_query`. A non-blank string ``base_url`` overrides ``SERPER_BASE_URL``; other raw tool-config values fall back to the environment and then the default endpoint. @@ -271,6 +340,11 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t payload["tbs"] = _SERPER_TBS_BY_TIME_RANGE[time_range] try: + if isinstance(max_retries, bool) or not isinstance(max_retries, int) or not 0 <= max_retries <= 3: + raise ValueError("max_retries must be an integer between 0 and 3") + if isinstance(retry_budget_seconds, bool) or not isinstance(retry_budget_seconds, (int, float)) or not 0 < retry_budget_seconds <= 300 or not math.isfinite(retry_budget_seconds): + raise ValueError("retry_budget_seconds must be a finite number greater than 0 and at most 300") + deadline = time.monotonic() + retry_budget_seconds # Resolve once before transport setup so retries can reuse this endpoint. if not isinstance(base_url, str) or not base_url.strip(): base_url = os.getenv("SERPER_BASE_URL") or "" @@ -294,8 +368,31 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t parsed_endpoint.path, ) with httpx.Client(timeout=30) as client: - response = client.post(endpoint, headers=headers, json=payload) - response.raise_for_status() + for attempt in range(max_retries + 1): + try: + response = client.post(endpoint, headers=headers, json=payload) + response.raise_for_status() + break + except (httpx.ConnectError, httpx.ConnectTimeout, httpx.HTTPStatusError) as exc: + if attempt >= max_retries: + raise + hint = None + if isinstance(exc, httpx.HTTPStatusError): + status = exc.response.status_code + if status in (429, 503): + hint = _retry_after(exc.response) + if status not in (502, 503, 504) and not (status == 429 and hint is not None): + raise + # Equal jitter retains exponential pacing even for Retry-After: 0. + backoff = min(0.5 * 2**attempt, 2.0) + delay = max(random.uniform(backoff / 2, backoff), hint or 0.0) + if delay >= deadline - time.monotonic(): + logger.warning("Serper retry time budget exhausted before backoff after %d attempt(s)", attempt + 1) + raise + time.sleep(delay) + if time.monotonic() >= deadline: + logger.warning("Serper retry time budget exhausted after backoff after %d attempt(s)", attempt + 1) + raise data = response.json() if not isinstance(data, dict): logger.error("Serper returned an unexpected payload type: %s", type(data).__name__) @@ -342,7 +439,7 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang return _missing_key_error(query, "web_search") base_url = extra.get("base_url") - data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url) + data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url, **_retry_options(config)) if error_json is not None: error = json.loads(error_json) error["query"] = query @@ -399,7 +496,7 @@ def image_search_tool(query: str, max_results: int = 5) -> str: return _missing_key_error(query, "image_search") base_url = extra.get("base_url") - data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url) + data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url, **_retry_options(config)) if error_json is not None: return error_json diff --git a/backend/tests/test_serper_retries.py b/backend/tests/test_serper_retries.py new file mode 100644 index 0000000000..be70bdb24e --- /dev/null +++ b/backend/tests/test_serper_retries.py @@ -0,0 +1,340 @@ +"""Offline recovery through production tools and real HTTPX transports.""" + +import json +import logging +import time +from datetime import UTC, datetime +from types import SimpleNamespace + +import httpx +import pytest + +from deerflow.community.serper import tools + + +@pytest.fixture(params=["web_search", "image_search"]) +def search(request, monkeypatch): + name = request.param + clients = [] + requests = [] + transports = [] + clock = SimpleNamespace(now=0.0, wall=1700000000.0, waits=[], elapsed=0.0, oversleep=0.0, jitter=1.0, ranges=[]) + real_client = httpx.Client + + def sleep(delay): + clock.waits.append(delay) + clock.now += delay + clock.oversleep + + monkeypatch.setattr(tools, "time", SimpleNamespace(monotonic=lambda: clock.now, time=lambda: clock.wall, sleep=sleep), raising=False) + + def uniform(low, high): + clock.ranges.append((low, high)) + return low + (high - low) * clock.jitter + + monkeypatch.setattr(tools, "random", SimpleNamespace(uniform=uniform), raising=False) + + class Transport(httpx.MockTransport): + closed = False + + def close(self): + self.closed = True + super().close() + + def run(outcomes, options=None, configured=True, direct=False): + extra = {"api_key": "dummy-key", **(options or {})} + monkeypatch.setenv("SERPER_API_KEY", "dummy-key") + monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(model_extra=extra) if configured else None)) + + def handle(req): + requests.append(req) + clock.now += clock.elapsed + outcome = outcomes[min(len(requests) - 1, len(outcomes) - 1)] + if isinstance(outcome, Exception): + raise outcome + if isinstance(outcome, tuple): + return httpx.Response(outcome[0], headers={"Retry-After": outcome[1]}) + if outcome == 200: + return httpx.Response(200, json={"organic": [{"link": "https://example.com/a"}, {"link": "https://ads.example.com/a"}], "images": [{"imageUrl": "https://example.com/a"}]}) + if outcome == "bad-json": + return httpx.Response(200, text="broken") + return httpx.Response(outcome) + + def client(**kwargs): + transport = Transport(handle) + transports.append(transport) + result = real_client(transport=transport, **kwargs) + clients.append(result) + return result + + monkeypatch.setattr(tools.httpx, "Client", client) + if direct: + data, error = tools._serper_post("https://google.serper.dev/search", "dummy-key", "news", 5, **(options or {})) + result = json.loads(error) if error else data + else: + arguments = {"query": " news "} + if name == "web_search": + arguments["time_range"] = "week" + result = json.loads(getattr(tools, name + "_tool").invoke(arguments)) + assert all(client.is_closed for client in clients) + assert all(transport.closed for transport in transports) + return result + + run.requests = requests + run.clients = clients + run.clock = clock + run.name = name + return run + + +@pytest.fixture +def non_utc_timezone(monkeypatch): + if not hasattr(time, "tzset"): + yield + return + try: + with monkeypatch.context() as local_timezone: + local_timezone.setenv("TZ", "GMT-8") + time.tzset() + yield + finally: + time.tzset() + + +@pytest.mark.parametrize("outcome", [502, 503, 504, (429, "0"), httpx.ConnectError("connect"), httpx.ConnectTimeout("connect")]) +def test_transient_recovers(search, outcome): + result = search([outcome, 200], {"max_retries": 1}) + assert "error" not in result + assert len(search.requests) == 2 + + +@pytest.mark.parametrize("configured", [True, False]) +@pytest.mark.parametrize("outcome", [503, (429, "0"), httpx.ConnectTimeout("connect")]) +def test_default_single_attempt(search, configured, outcome): + assert "error" in search([outcome, 200], configured=configured) + assert len(search.requests) == 1 + assert search.clock.waits == [] + + +def test_helper_recovery(search): + assert "organic" in search([503, 200], {"max_retries": 1}, direct=True) + assert len(search.requests) == 2 + + +def test_exhaustion(search, caplog): + assert search([503], {"max_retries": 3}) == {"query": "news", "error": "Serper API error: HTTP 503"} + assert len(search.requests) == 4 + assert search.clock.waits == [0.5, 1.0, 2.0] + assert "retry time budget exhausted" not in caplog.text + + +@pytest.mark.parametrize("outcome", [400, 401, 403, 429, 500, "bad-json", httpx.ReadTimeout("read"), httpx.WriteTimeout("write"), httpx.ReadError("read"), httpx.PoolTimeout("pool")]) +def test_terminal(search, outcome): + assert "error" in search([outcome, 200], {"max_retries": 3}) + assert len(search.requests) == 1 + + +@pytest.mark.parametrize("field,value", [("max_retries", v) for v in [-1, 4, True, 1.0, "1", None]] + [("retry_budget_seconds", v) for v in [0, -1, True, "1", None, float("nan"), float("inf"), 301]]) +def test_invalid_configuration(search, field, value): + assert field in search([200], {field: value})["error"] + assert not search.requests + assert not search.clients + + +@pytest.mark.parametrize("status", [429, 503]) +@pytest.mark.parametrize("hint,delay", [("0", 0.5), ("2", 2.0), ("Tue, 14 Nov 2023 22:13:22 GMT", 2.0), ("Tue, 14 Nov 2023 22:13:19 GMT", 0.5)]) +def test_valid_hints(search, status, hint, delay): + assert "error" not in search([(status, hint), 200], {"max_retries": 1}) + assert search.clock.waits == [delay] + + +@pytest.mark.parametrize("hint", ["", "-1", "1.5", "NaN", "inf", "tomorrow", "1, 2"]) +@pytest.mark.parametrize("status", [429, 503]) +def test_invalid_hints(search, hint, status): + result = search([(status, hint), 200], {"max_retries": 1}) + assert len(search.requests) == (1 if status == 429 else 2) + assert ("error" in result) == (status == 429) + + +@pytest.mark.parametrize("status", [429, 503]) +@pytest.mark.parametrize("hint", ["31", "9" * 5000, "Tue, 14 Nov 2023 22:14:00 GMT"]) +def test_hint_exceeds_budget(search, status, hint): + assert "error" in search([(status, hint), 200], {"max_retries": 1}) + assert len(search.requests) == 1 + assert search.clock.waits == [] + + +@pytest.mark.parametrize("elapsed,budget,attempts,waits", [(0, 1.5, 2, [0.5]), (1, 2, 2, [0.5]), (2, 1, 1, [])]) +def test_single_deadline(search, elapsed, budget, attempts, waits): + search.clock.elapsed = elapsed + assert "error" in search([503], {"max_retries": 3, "retry_budget_seconds": budget}) + assert len(search.requests) == attempts + assert search.clock.waits == waits + + +def test_oversleep_prevents_request(search): + search.clock.oversleep = 2 + assert "error" in search([503, 200], {"max_retries": 1, "retry_budget_seconds": 1}) + assert len(search.requests) == 1 + + +@pytest.mark.parametrize("outcome", [503, (429, "0"), httpx.ConnectTimeout("connect")]) +@pytest.mark.parametrize("budget,oversleep,stage,waits", [(0.5, 0, "before backoff", []), (1, 2, "after backoff", [0.5])]) +def test_budget_exhaustion_diagnostic(search, caplog, outcome, budget, oversleep, stage, waits): + search.clock.oversleep = oversleep + result = search([outcome, 200], {"max_retries": 1, "retry_budget_seconds": budget}) + status = outcome[0] if isinstance(outcome, tuple) else outcome + last_error = "connect" if isinstance(outcome, Exception) else f"Serper API error: HTTP {status}" + assert result == {"error": last_error, "query": "news"} + assert len(search.requests) == 1 + assert search.clock.waits == waits + diagnostics = [record for record in caplog.records if record.name == tools.logger.name and "retry time budget exhausted" in record.getMessage()] + assert len(diagnostics) == 1 + assert diagnostics[0].levelno == logging.WARNING + assert stage in diagnostics[0].getMessage() + assert "dummy-key" not in diagnostics[0].getMessage() + assert "news" not in diagnostics[0].getMessage() + + +def test_preserves_request_and_filters(search): + result = search([503, 200], {"max_retries": 1, "max_results": 1, "include_domains": ["example.com"], "exclude_domains": ["ads.example.com"]}) + assert result["query"] == "news" + assert result["total_results"] == 1 + first, second = search.requests + assert first.content == second.content + assert first.headers == second.headers + assert first.headers["X-API-KEY"] == "dummy-key" + expected = {"q": "news", "num": 1} + if search.name == "web_search": + expected.update(q="(news) (site:example.com) -site:ads.example.com", tbs="qdr:w") + assert result["results"][0]["url"] == "https://example.com/a" + assert json.loads(first.content) == expected + + +@pytest.mark.parametrize("endpoint_source", ["tool-config", "environment"]) +def test_retry_keeps_configured_endpoint_and_key_through_reload(search, monkeypatch, endpoint_source): + """A retry must not switch providers when config or environment changes.""" + options = {"max_retries": 1} + if endpoint_source == "tool-config": + options["base_url"] = " https://first.example/api/// " + monkeypatch.setenv("SERPER_BASE_URL", "https://fallback.example") + else: + monkeypatch.setenv("SERPER_BASE_URL", " https://first.example/api/// ") + original_sleep = tools.time.sleep + + def reload_during_backoff(delay): + original_sleep(delay) + monkeypatch.setenv("SERPER_BASE_URL", "https://second.example/api") + monkeypatch.setenv("SERPER_API_KEY", "second-provider-key") + + def reloaded_config(): + pytest.fail("Retries must use the captured tool configuration") + + monkeypatch.setattr(tools, "get_app_config", reloaded_config) + + monkeypatch.setattr(tools.time, "sleep", reload_during_backoff) + result = search([503, 200], options) + + assert "error" not in result + assert len(search.requests) == 2 + route = "search" if search.name == "web_search" else "images" + assert {str(request.url) for request in search.requests} == {f"https://first.example/api/{route}"} + assert {request.headers["X-API-KEY"] for request in search.requests} == {"dummy-key"} + assert search.requests[0].content == search.requests[1].content + + +def test_filtered_error_keeps_original_query(search): + result = search([503], {"max_retries": 1, "include_domains": ["example.com"]}) + assert result == {"query": "news", "error": "Serper API error: HTTP 503"} + assert len(search.requests) == 2 + + +@pytest.mark.parametrize("options", [{"max_retries": 0}, {"retry_budget_seconds": 1}]) +def test_budget_does_not_enable_retries(search, options): + assert "error" in search([503, 200], options) + assert len(search.requests) == 1 + + +def test_jitter_lower_bound(search): + search.clock.jitter = 0 + search([503], {"max_retries": 3}) + assert search.clock.waits == [0.25, 0.5, 1.0] + assert search.clock.ranges == [(0.25, 0.5), (0.5, 1), (1, 2)] + + +def test_direct_invalid_options(search): + assert "max_retries" in search([200], {"max_retries": True}, direct=True)["error"] + assert not search.clients + + +def test_tool_schemas_unchanged(): + assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"} + assert set(tools.image_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results"} + + +@pytest.mark.parametrize("outcome", [httpx.ConnectError("connect"), httpx.ConnectTimeout("connect")]) +def test_connection_exhaustion(search, outcome): + assert search([outcome], {"max_retries": 1}) == {"error": "connect", "query": "news"} + assert len(search.requests) == 2 + + +def test_last_status_on_budget_exhaustion(search): + result = search([503, (429, "30"), 200], {"max_retries": 3}) + assert result == {"error": "Serper API error: HTTP 429", "query": "news"} + assert len(search.requests) == 2 + assert search.clock.waits == [0.5] + + +@pytest.mark.parametrize("hint", ["Sun Nov 6 08:49:37 1994", "Sunday, 06-Nov-94 08:49:37 GMT"]) +def test_legacy_http_date_is_valid(search, hint): + result = search([(429, hint), 200], {"max_retries": 1}) + assert "error" not in result + assert len(search.requests) == 2 + + +@pytest.mark.parametrize("status", [429, 503]) +@pytest.mark.parametrize( + "current,hint,delay", + [ + (datetime(2023, 11, 14, 22, 13, 20, tzinfo=UTC), "Tue Nov 14 22:13:22 2023", 2.0), + (datetime(2023, 11, 14, 22, 13, 20, tzinfo=UTC), "Tue Nov 14 22:13:19 2023", 0.5), + (datetime(2023, 11, 5, 8, 49, 37, tzinfo=UTC), "Sun Nov 5 08:49:39 2023", 2.0), + (datetime(2023, 11, 5, 8, 49, 37, tzinfo=UTC), "Sun Nov 5 08:49:36 2023", 0.5), + ], +) +def test_asctime_hint_uses_utc_and_server_floor(search, non_utc_timezone, status, current, hint, delay): + search.clock.wall = current.timestamp() + assert "error" not in search([(status, hint), 200], {"max_retries": 1}) + assert len(search.requests) == 2 + assert search.clock.waits == [delay] + + +@pytest.mark.parametrize("status", [429, 503]) +@pytest.mark.parametrize( + "hint,resolved", + [ + ("Tuesday, 14-Nov-23 22:13:22 GMT", datetime(2023, 11, 14, 22, 13, 22, tzinfo=UTC)), + ("Saturday, 14-Nov-26 22:13:20 GMT", datetime(2026, 11, 14, 22, 13, 20, tzinfo=UTC)), + ("Sunday, 14-Nov-76 22:13:20 GMT", datetime(1976, 11, 14, 22, 13, 20, tzinfo=UTC)), + ("Tuesday, 14-Nov-73 22:13:20 GMT", datetime(2073, 11, 14, 22, 13, 20, tzinfo=UTC)), + ("Wednesday, 14-Nov-73 22:13:21 GMT", datetime(1973, 11, 14, 22, 13, 21, tzinfo=UTC)), + ], +) +def test_rfc850_hint_year_window_and_scheduling(search, status, hint, resolved): + floor = max(0.0, resolved.timestamp() - search.clock.wall) + assert tools._retry_after(httpx.Response(status, headers={"Retry-After": hint})) == floor + result = search([(status, hint), 200], {"max_retries": 1}) + if floor >= 30: + assert result == {"error": f"Serper API error: HTTP {status}", "query": "news"} + assert len(search.requests) == 1 + assert search.clock.waits == [] + else: + assert "error" not in result + assert len(search.requests) == 2 + assert search.clock.waits == [max(0.5, floor)] + + +@pytest.mark.parametrize("hint", ["Wed, 21 Oct 2015 07:28:00 +0100", "Wed, 21 Oct 2015 07:28:00 UTC", "Wed, 21 Oct 2015 07:28 GMT"]) +def test_email_dates_do_not_enable_429_retry(search, hint): + result = search([(429, hint), 200], {"max_retries": 1}) + assert "error" in result + assert len(search.requests) == 1 diff --git a/config.example.yaml b/config.example.yaml index 4b507746e7..91b8b01423 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -935,6 +935,8 @@ tools: # group: web # use: deerflow.community.serper.tools:web_search_tool # max_results: 5 # capped at 10 by the Serper provider + # max_retries: 0 # Opt in with 1-3 extra requests; may incur cost + # retry_budget_seconds: 30 # >0 to 300; scheduling budget, not an I/O deadline # # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /search # include_domains: [] # Optional domain-only lists, at most 10 entries each # exclude_domains: [] # Exclusion wins; see backend/docs/CONFIGURATION.md @@ -1278,6 +1280,8 @@ tools: # group: web # use: deerflow.community.serper.tools:image_search_tool # max_results: 5 # capped at 10 by the Serper provider + # max_retries: 0 # Opt in with 1-3 extra requests; may incur cost + # retry_budget_seconds: 30 # >0 to 300; scheduling budget, not an I/O deadline # # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /images # # api_key: $SERPER_API_KEY # Optional explicit env-var reference