mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
feat(serper): add opt-in bounded transient retries (#6456)
* feat(serper): add opt-in bounded transient retries Signed-off-by: 97three <2212371308@qq.com> * fix(serper): validate HTTP retry dates precisely Signed-off-by: 97three <2212371308@qq.com> * fix(serper): address retry review feedback --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
creed
GitHub
parent
2d55c75870
commit
be62fff070
@ -1472,6 +1472,13 @@ global URL-access policy. The model arguments and image search are unchanged.
|
||||
See [Serper configuration](backend/docs/CONFIGURATION.md#serper-source-filters)
|
||||
for validation and query-length limits.
|
||||
|
||||
Serper web and image search support opt-in transient recovery with
|
||||
`max_retries: 1` (default `0`, maximum `3`) in each tool's configuration.
|
||||
`retry_budget_seconds` bounds retry scheduling, not active synchronous requests.
|
||||
Budget exhaustion is logged separately while the tool returns the last error.
|
||||
Extra requests may consume quota or incur cost. See
|
||||
[Serper retries](backend/docs/CONFIGURATION.md#serper-retries).
|
||||
|
||||
When using Tavily for `web_fetch`, extracted pages without a title use their URL
|
||||
as the heading; their content remains available to the agent.
|
||||
Chat tool-step titles accept leading blank lines and up to three spaces before
|
||||
|
||||
@ -734,6 +734,46 @@ Serper `web_search` also accepts the optional model argument
|
||||
to Serper. Omitting `time_range` or passing `null` omits the recency constraint from the
|
||||
search request. This option does not change Serper `image_search`.
|
||||
|
||||
#### Serper retries
|
||||
|
||||
Both Serper `web_search` and `image_search` accept these operator-only settings
|
||||
in their respective `tools` entries (the model-facing arguments are unchanged):
|
||||
|
||||
```yaml
|
||||
max_retries: 0 # Default off; integer 0–3 extra requests
|
||||
retry_budget_seconds: 30 # Default 30; finite number >0 and <=300
|
||||
```
|
||||
|
||||
Set `max_retries: 1` to enable recovery. Booleans, strings, nulls, out-of-range
|
||||
values, and non-finite budgets return a structured configuration error before
|
||||
HTTP. The three-retry cap limits additional quota/cost; the 300-second budget
|
||||
cap limits scheduling waits. No billing or idempotency guarantee is claimed.
|
||||
|
||||
Only connection establishment failures (`ConnectError`/`ConnectTimeout`) and
|
||||
HTTP 502/503/504 are retried. HTTP 429 requires a valid `Retry-After` hint.
|
||||
Authentication errors, other statuses, read/write timeouts, malformed successful
|
||||
JSON and tool/result-processing failures are never retried.
|
||||
|
||||
Backoff uses equal jitter: 0.25–0.5, 0.5–1, then 1–2 seconds. Valid nonnegative
|
||||
integer seconds or HTTP-date `Retry-After` hints on 429/503 set a minimum wait
|
||||
(past dates mean zero). HTTP dates include asctime and RFC 850 forms; RFC 850
|
||||
two-digit years resolve to the most recent matching year no more than 50 years
|
||||
in the future. Invalid hints leave 503 on local backoff;
|
||||
429 stays terminal. A hint or backoff that reaches/exceeds the remaining budget
|
||||
returns the last error without another request or a shortened wait.
|
||||
Both budget checks log `Serper retry time budget exhausted` at warning level,
|
||||
distinguishing budget stops from exhausting all configured attempts without
|
||||
changing the returned error.
|
||||
|
||||
One monotonic scheduling deadline starts before client creation, counting elapsed
|
||||
requests and waits; it is checked before waiting and again before a retry.
|
||||
It does **not** interrupt active synchronous HTTP requests, response streaming,
|
||||
or worker threads. HTTPX's unchanged 30-second timeout applies per network phase,
|
||||
so this is **not** a hard end-to-end wall-clock deadline. The initial request is
|
||||
always allowed. Payload, API key, domain restrictions and time range stay identical
|
||||
across attempts; errors still report the cleaned original query. No refill,
|
||||
cache, global limiter or model-level retry is added.
|
||||
|
||||
#### Serper endpoints
|
||||
|
||||
To use a Serper-compatible provider for web and image search, set
|
||||
@ -821,7 +861,7 @@ including on provider errors.
|
||||
|
||||
`time_range` still maps to `tbs`, and `max_results` caps the filtered results.
|
||||
`total_results` reports the actual remaining count (zero with `results: []`
|
||||
when none survive). One request is made: no refill or relaxed-filter retries.
|
||||
when none survive). No refill or relaxed-filter requests are made.
|
||||
These settings do not affect `image_search` or the model-facing tool schema.
|
||||
Source selection is neither a factuality guarantee nor a global URL-access
|
||||
policy for fetch tools, browsers, or redirects.
|
||||
|
||||
@ -21,7 +21,7 @@ Read each tool's config once and pass captured extras to key resolution,
|
||||
including an empty mapping when config is absent, to keep endpoint/key paired
|
||||
through hot reload. Settings stay operator-only. Send keys
|
||||
only via `X-API-KEY`, and preserve result URL validation and model arguments.
|
||||
Future retries must reuse the resolved endpoint across attempts. Debug endpoint
|
||||
Retries must reuse the resolved endpoint across attempts. Debug endpoint
|
||||
diagnostics omit URL credentials, query and fragment; result-URL guards do not
|
||||
restrict the operator's API host.
|
||||
|
||||
@ -32,3 +32,16 @@ handlers can redact paths. Assert full endpoint routing on the captured HTTP
|
||||
request, and verify diagnostic host/port and credential omission in both modes.
|
||||
Mock HTTP; live Serper semantics remain unverified. See
|
||||
`backend/docs/CONFIGURATION.md#serper-source-filters` for the operator contract.
|
||||
|
||||
`_serper_post` owns default-off physical retries (0–3) and one monotonic
|
||||
scheduling budget (>0–300s), not cancellation of synchronous I/O. Keep config
|
||||
validation before transport and retry only connect failures, 502/503/504 and
|
||||
hinted 429. Provider-local hint parsing avoids coupling to Jina. Whole-tool
|
||||
extension hooks cannot isolate this boundary. Test with real MockTransport
|
||||
through both tools in `backend/tests/test_serper_retries.py`.
|
||||
Endpoint/retry regressions change configuration and environment during backoff;
|
||||
every attempt must keep the original provider URL, API key and request payload.
|
||||
Budget aborts before and after backoff log a warning identifying the stage,
|
||||
without query/key values, and preserve the last structured error. Date tests
|
||||
pin asctime UTC anchoring (under a non-UTC host timezone where supported) and
|
||||
the RFC 850 50-year cutoff down to the second.
|
||||
|
||||
@ -7,9 +7,14 @@ API. An API key is required. Sign up at https://serper.dev to get one.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Mapping
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
from ipaddress import IPv4Address, ip_address
|
||||
from urllib.parse import urlparse
|
||||
|
||||
@ -249,9 +254,73 @@ def _safe_public_url(value: object) -> str:
|
||||
return url if ip.is_global else ""
|
||||
|
||||
|
||||
def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, time_range: SearchTimeRange | None = None, base_url: object | None = None) -> tuple[dict | None, str | None]:
|
||||
def _retry_options(config) -> dict:
|
||||
extra = config.model_extra if config is not None else {}
|
||||
return {name: extra[name] for name in ("max_retries", "retry_budget_seconds") if name in extra}
|
||||
|
||||
|
||||
_DAY = r"(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)"
|
||||
_MONTH = r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)"
|
||||
_CLOCK = r"[0-9]{2}:[0-9]{2}:[0-9]{2}"
|
||||
_HTTP_DATE = re.compile(
|
||||
rf"(?:{_DAY}, [0-9]{{2}} {_MONTH} [0-9]{{4}} {_CLOCK} GMT"
|
||||
rf"|(?:Monday|Tuesday|Wednesday|Thursday|Friday|Saturday|Sunday), [0-9]{{2}}-{_MONTH}-[0-9]{{2}} {_CLOCK} GMT"
|
||||
rf"|{_DAY} {_MONTH} (?:[0-9]{{2}}| [0-9]) {_CLOCK} [0-9]{{4}})"
|
||||
)
|
||||
|
||||
|
||||
def _retry_after(response: httpx.Response) -> float | None:
|
||||
"""Return a server floor, or None for an invalid HTTP Retry-After value."""
|
||||
value = response.headers.get("Retry-After")
|
||||
if value is None:
|
||||
return None
|
||||
value = value.strip(" \t")
|
||||
if value and value.isascii() and value.isdecimal():
|
||||
digits = value.lstrip("0") or "0"
|
||||
# Bound integer conversion work. Infinity is a valid, unfit floor, not
|
||||
# a parsing failure that could cause an early fallback retry.
|
||||
if len(digits) > 309:
|
||||
return math.inf
|
||||
seconds = int(digits)
|
||||
try:
|
||||
floor = float(seconds)
|
||||
except OverflowError:
|
||||
return math.inf
|
||||
return math.nextafter(floor, math.inf) if floor < seconds else floor
|
||||
if not _HTTP_DATE.fullmatch(value):
|
||||
return None
|
||||
try:
|
||||
date = parsedate_to_datetime(value).replace(tzinfo=UTC)
|
||||
now = time.time()
|
||||
if "-" in value:
|
||||
# RFC 850 two-digit years: choose the most recent matching year
|
||||
# no more than 50 years in the future (RFC 9110 section 5.6.7).
|
||||
current = datetime.fromtimestamp(now, UTC)
|
||||
year = (current.year + 50) // 100 * 100 + date.year % 100
|
||||
if (year, date.month, date.day, date.hour, date.minute, date.second) > (current.year + 50, current.month, current.day, current.hour, current.minute, current.second):
|
||||
year -= 100
|
||||
date = date.replace(year=year)
|
||||
return max(0.0, date.timestamp() - now)
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _serper_post(
|
||||
endpoint: str,
|
||||
api_key: str,
|
||||
query: str,
|
||||
max_results: int,
|
||||
*,
|
||||
time_range: SearchTimeRange | None = None,
|
||||
base_url: object | None = None,
|
||||
max_retries: int = 0,
|
||||
retry_budget_seconds: float = 30,
|
||||
) -> tuple[dict | None, str | None]:
|
||||
"""Send a POST request to a Serper endpoint.
|
||||
|
||||
Retries share a scheduling deadline, not a hard synchronous I/O deadline.
|
||||
HTTPX timeouts remain per phase; an active request cannot be cancelled here.
|
||||
|
||||
``query`` is expected to already be normalized via :func:`_clean_query`.
|
||||
A non-blank string ``base_url`` overrides ``SERPER_BASE_URL``; other raw
|
||||
tool-config values fall back to the environment and then the default endpoint.
|
||||
@ -271,6 +340,11 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t
|
||||
payload["tbs"] = _SERPER_TBS_BY_TIME_RANGE[time_range]
|
||||
|
||||
try:
|
||||
if isinstance(max_retries, bool) or not isinstance(max_retries, int) or not 0 <= max_retries <= 3:
|
||||
raise ValueError("max_retries must be an integer between 0 and 3")
|
||||
if isinstance(retry_budget_seconds, bool) or not isinstance(retry_budget_seconds, (int, float)) or not 0 < retry_budget_seconds <= 300 or not math.isfinite(retry_budget_seconds):
|
||||
raise ValueError("retry_budget_seconds must be a finite number greater than 0 and at most 300")
|
||||
deadline = time.monotonic() + retry_budget_seconds
|
||||
# Resolve once before transport setup so retries can reuse this endpoint.
|
||||
if not isinstance(base_url, str) or not base_url.strip():
|
||||
base_url = os.getenv("SERPER_BASE_URL") or ""
|
||||
@ -294,8 +368,31 @@ def _serper_post(endpoint: str, api_key: str, query: str, max_results: int, *, t
|
||||
parsed_endpoint.path,
|
||||
)
|
||||
with httpx.Client(timeout=30) as client:
|
||||
response = client.post(endpoint, headers=headers, json=payload)
|
||||
response.raise_for_status()
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
response = client.post(endpoint, headers=headers, json=payload)
|
||||
response.raise_for_status()
|
||||
break
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.HTTPStatusError) as exc:
|
||||
if attempt >= max_retries:
|
||||
raise
|
||||
hint = None
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
status = exc.response.status_code
|
||||
if status in (429, 503):
|
||||
hint = _retry_after(exc.response)
|
||||
if status not in (502, 503, 504) and not (status == 429 and hint is not None):
|
||||
raise
|
||||
# Equal jitter retains exponential pacing even for Retry-After: 0.
|
||||
backoff = min(0.5 * 2**attempt, 2.0)
|
||||
delay = max(random.uniform(backoff / 2, backoff), hint or 0.0)
|
||||
if delay >= deadline - time.monotonic():
|
||||
logger.warning("Serper retry time budget exhausted before backoff after %d attempt(s)", attempt + 1)
|
||||
raise
|
||||
time.sleep(delay)
|
||||
if time.monotonic() >= deadline:
|
||||
logger.warning("Serper retry time budget exhausted after backoff after %d attempt(s)", attempt + 1)
|
||||
raise
|
||||
data = response.json()
|
||||
if not isinstance(data, dict):
|
||||
logger.error("Serper returned an unexpected payload type: %s", type(data).__name__)
|
||||
@ -342,7 +439,7 @@ def web_search_tool(query: str, max_results: int = 5, time_range: SearchTimeRang
|
||||
return _missing_key_error(query, "web_search")
|
||||
|
||||
base_url = extra.get("base_url")
|
||||
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url)
|
||||
data, error_json = _serper_post(_SERPER_SEARCH_ENDPOINT, api_key, search_query, max_results, time_range=time_range, base_url=base_url, **_retry_options(config))
|
||||
if error_json is not None:
|
||||
error = json.loads(error_json)
|
||||
error["query"] = query
|
||||
@ -399,7 +496,7 @@ def image_search_tool(query: str, max_results: int = 5) -> str:
|
||||
return _missing_key_error(query, "image_search")
|
||||
|
||||
base_url = extra.get("base_url")
|
||||
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url)
|
||||
data, error_json = _serper_post(_SERPER_IMAGES_ENDPOINT, api_key, query, max_results, base_url=base_url, **_retry_options(config))
|
||||
if error_json is not None:
|
||||
return error_json
|
||||
|
||||
|
||||
340
backend/tests/test_serper_retries.py
Normal file
340
backend/tests/test_serper_retries.py
Normal file
@ -0,0 +1,340 @@
|
||||
"""Offline recovery through production tools and real HTTPX transports."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from deerflow.community.serper import tools
|
||||
|
||||
|
||||
@pytest.fixture(params=["web_search", "image_search"])
|
||||
def search(request, monkeypatch):
|
||||
name = request.param
|
||||
clients = []
|
||||
requests = []
|
||||
transports = []
|
||||
clock = SimpleNamespace(now=0.0, wall=1700000000.0, waits=[], elapsed=0.0, oversleep=0.0, jitter=1.0, ranges=[])
|
||||
real_client = httpx.Client
|
||||
|
||||
def sleep(delay):
|
||||
clock.waits.append(delay)
|
||||
clock.now += delay + clock.oversleep
|
||||
|
||||
monkeypatch.setattr(tools, "time", SimpleNamespace(monotonic=lambda: clock.now, time=lambda: clock.wall, sleep=sleep), raising=False)
|
||||
|
||||
def uniform(low, high):
|
||||
clock.ranges.append((low, high))
|
||||
return low + (high - low) * clock.jitter
|
||||
|
||||
monkeypatch.setattr(tools, "random", SimpleNamespace(uniform=uniform), raising=False)
|
||||
|
||||
class Transport(httpx.MockTransport):
|
||||
closed = False
|
||||
|
||||
def close(self):
|
||||
self.closed = True
|
||||
super().close()
|
||||
|
||||
def run(outcomes, options=None, configured=True, direct=False):
|
||||
extra = {"api_key": "dummy-key", **(options or {})}
|
||||
monkeypatch.setenv("SERPER_API_KEY", "dummy-key")
|
||||
monkeypatch.setattr(tools, "get_app_config", lambda: SimpleNamespace(get_tool_config=lambda _: SimpleNamespace(model_extra=extra) if configured else None))
|
||||
|
||||
def handle(req):
|
||||
requests.append(req)
|
||||
clock.now += clock.elapsed
|
||||
outcome = outcomes[min(len(requests) - 1, len(outcomes) - 1)]
|
||||
if isinstance(outcome, Exception):
|
||||
raise outcome
|
||||
if isinstance(outcome, tuple):
|
||||
return httpx.Response(outcome[0], headers={"Retry-After": outcome[1]})
|
||||
if outcome == 200:
|
||||
return httpx.Response(200, json={"organic": [{"link": "https://example.com/a"}, {"link": "https://ads.example.com/a"}], "images": [{"imageUrl": "https://example.com/a"}]})
|
||||
if outcome == "bad-json":
|
||||
return httpx.Response(200, text="broken")
|
||||
return httpx.Response(outcome)
|
||||
|
||||
def client(**kwargs):
|
||||
transport = Transport(handle)
|
||||
transports.append(transport)
|
||||
result = real_client(transport=transport, **kwargs)
|
||||
clients.append(result)
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(tools.httpx, "Client", client)
|
||||
if direct:
|
||||
data, error = tools._serper_post("https://google.serper.dev/search", "dummy-key", "news", 5, **(options or {}))
|
||||
result = json.loads(error) if error else data
|
||||
else:
|
||||
arguments = {"query": " news "}
|
||||
if name == "web_search":
|
||||
arguments["time_range"] = "week"
|
||||
result = json.loads(getattr(tools, name + "_tool").invoke(arguments))
|
||||
assert all(client.is_closed for client in clients)
|
||||
assert all(transport.closed for transport in transports)
|
||||
return result
|
||||
|
||||
run.requests = requests
|
||||
run.clients = clients
|
||||
run.clock = clock
|
||||
run.name = name
|
||||
return run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def non_utc_timezone(monkeypatch):
|
||||
if not hasattr(time, "tzset"):
|
||||
yield
|
||||
return
|
||||
try:
|
||||
with monkeypatch.context() as local_timezone:
|
||||
local_timezone.setenv("TZ", "GMT-8")
|
||||
time.tzset()
|
||||
yield
|
||||
finally:
|
||||
time.tzset()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("outcome", [502, 503, 504, (429, "0"), httpx.ConnectError("connect"), httpx.ConnectTimeout("connect")])
|
||||
def test_transient_recovers(search, outcome):
|
||||
result = search([outcome, 200], {"max_retries": 1})
|
||||
assert "error" not in result
|
||||
assert len(search.requests) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("configured", [True, False])
|
||||
@pytest.mark.parametrize("outcome", [503, (429, "0"), httpx.ConnectTimeout("connect")])
|
||||
def test_default_single_attempt(search, configured, outcome):
|
||||
assert "error" in search([outcome, 200], configured=configured)
|
||||
assert len(search.requests) == 1
|
||||
assert search.clock.waits == []
|
||||
|
||||
|
||||
def test_helper_recovery(search):
|
||||
assert "organic" in search([503, 200], {"max_retries": 1}, direct=True)
|
||||
assert len(search.requests) == 2
|
||||
|
||||
|
||||
def test_exhaustion(search, caplog):
|
||||
assert search([503], {"max_retries": 3}) == {"query": "news", "error": "Serper API error: HTTP 503"}
|
||||
assert len(search.requests) == 4
|
||||
assert search.clock.waits == [0.5, 1.0, 2.0]
|
||||
assert "retry time budget exhausted" not in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("outcome", [400, 401, 403, 429, 500, "bad-json", httpx.ReadTimeout("read"), httpx.WriteTimeout("write"), httpx.ReadError("read"), httpx.PoolTimeout("pool")])
|
||||
def test_terminal(search, outcome):
|
||||
assert "error" in search([outcome, 200], {"max_retries": 3})
|
||||
assert len(search.requests) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [("max_retries", v) for v in [-1, 4, True, 1.0, "1", None]] + [("retry_budget_seconds", v) for v in [0, -1, True, "1", None, float("nan"), float("inf"), 301]])
|
||||
def test_invalid_configuration(search, field, value):
|
||||
assert field in search([200], {field: value})["error"]
|
||||
assert not search.requests
|
||||
assert not search.clients
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [429, 503])
|
||||
@pytest.mark.parametrize("hint,delay", [("0", 0.5), ("2", 2.0), ("Tue, 14 Nov 2023 22:13:22 GMT", 2.0), ("Tue, 14 Nov 2023 22:13:19 GMT", 0.5)])
|
||||
def test_valid_hints(search, status, hint, delay):
|
||||
assert "error" not in search([(status, hint), 200], {"max_retries": 1})
|
||||
assert search.clock.waits == [delay]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hint", ["", "-1", "1.5", "NaN", "inf", "tomorrow", "1, 2"])
|
||||
@pytest.mark.parametrize("status", [429, 503])
|
||||
def test_invalid_hints(search, hint, status):
|
||||
result = search([(status, hint), 200], {"max_retries": 1})
|
||||
assert len(search.requests) == (1 if status == 429 else 2)
|
||||
assert ("error" in result) == (status == 429)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [429, 503])
|
||||
@pytest.mark.parametrize("hint", ["31", "9" * 5000, "Tue, 14 Nov 2023 22:14:00 GMT"])
|
||||
def test_hint_exceeds_budget(search, status, hint):
|
||||
assert "error" in search([(status, hint), 200], {"max_retries": 1})
|
||||
assert len(search.requests) == 1
|
||||
assert search.clock.waits == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("elapsed,budget,attempts,waits", [(0, 1.5, 2, [0.5]), (1, 2, 2, [0.5]), (2, 1, 1, [])])
|
||||
def test_single_deadline(search, elapsed, budget, attempts, waits):
|
||||
search.clock.elapsed = elapsed
|
||||
assert "error" in search([503], {"max_retries": 3, "retry_budget_seconds": budget})
|
||||
assert len(search.requests) == attempts
|
||||
assert search.clock.waits == waits
|
||||
|
||||
|
||||
def test_oversleep_prevents_request(search):
|
||||
search.clock.oversleep = 2
|
||||
assert "error" in search([503, 200], {"max_retries": 1, "retry_budget_seconds": 1})
|
||||
assert len(search.requests) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("outcome", [503, (429, "0"), httpx.ConnectTimeout("connect")])
|
||||
@pytest.mark.parametrize("budget,oversleep,stage,waits", [(0.5, 0, "before backoff", []), (1, 2, "after backoff", [0.5])])
|
||||
def test_budget_exhaustion_diagnostic(search, caplog, outcome, budget, oversleep, stage, waits):
|
||||
search.clock.oversleep = oversleep
|
||||
result = search([outcome, 200], {"max_retries": 1, "retry_budget_seconds": budget})
|
||||
status = outcome[0] if isinstance(outcome, tuple) else outcome
|
||||
last_error = "connect" if isinstance(outcome, Exception) else f"Serper API error: HTTP {status}"
|
||||
assert result == {"error": last_error, "query": "news"}
|
||||
assert len(search.requests) == 1
|
||||
assert search.clock.waits == waits
|
||||
diagnostics = [record for record in caplog.records if record.name == tools.logger.name and "retry time budget exhausted" in record.getMessage()]
|
||||
assert len(diagnostics) == 1
|
||||
assert diagnostics[0].levelno == logging.WARNING
|
||||
assert stage in diagnostics[0].getMessage()
|
||||
assert "dummy-key" not in diagnostics[0].getMessage()
|
||||
assert "news" not in diagnostics[0].getMessage()
|
||||
|
||||
|
||||
def test_preserves_request_and_filters(search):
|
||||
result = search([503, 200], {"max_retries": 1, "max_results": 1, "include_domains": ["example.com"], "exclude_domains": ["ads.example.com"]})
|
||||
assert result["query"] == "news"
|
||||
assert result["total_results"] == 1
|
||||
first, second = search.requests
|
||||
assert first.content == second.content
|
||||
assert first.headers == second.headers
|
||||
assert first.headers["X-API-KEY"] == "dummy-key"
|
||||
expected = {"q": "news", "num": 1}
|
||||
if search.name == "web_search":
|
||||
expected.update(q="(news) (site:example.com) -site:ads.example.com", tbs="qdr:w")
|
||||
assert result["results"][0]["url"] == "https://example.com/a"
|
||||
assert json.loads(first.content) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint_source", ["tool-config", "environment"])
|
||||
def test_retry_keeps_configured_endpoint_and_key_through_reload(search, monkeypatch, endpoint_source):
|
||||
"""A retry must not switch providers when config or environment changes."""
|
||||
options = {"max_retries": 1}
|
||||
if endpoint_source == "tool-config":
|
||||
options["base_url"] = " https://first.example/api/// "
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://fallback.example")
|
||||
else:
|
||||
monkeypatch.setenv("SERPER_BASE_URL", " https://first.example/api/// ")
|
||||
original_sleep = tools.time.sleep
|
||||
|
||||
def reload_during_backoff(delay):
|
||||
original_sleep(delay)
|
||||
monkeypatch.setenv("SERPER_BASE_URL", "https://second.example/api")
|
||||
monkeypatch.setenv("SERPER_API_KEY", "second-provider-key")
|
||||
|
||||
def reloaded_config():
|
||||
pytest.fail("Retries must use the captured tool configuration")
|
||||
|
||||
monkeypatch.setattr(tools, "get_app_config", reloaded_config)
|
||||
|
||||
monkeypatch.setattr(tools.time, "sleep", reload_during_backoff)
|
||||
result = search([503, 200], options)
|
||||
|
||||
assert "error" not in result
|
||||
assert len(search.requests) == 2
|
||||
route = "search" if search.name == "web_search" else "images"
|
||||
assert {str(request.url) for request in search.requests} == {f"https://first.example/api/{route}"}
|
||||
assert {request.headers["X-API-KEY"] for request in search.requests} == {"dummy-key"}
|
||||
assert search.requests[0].content == search.requests[1].content
|
||||
|
||||
|
||||
def test_filtered_error_keeps_original_query(search):
|
||||
result = search([503], {"max_retries": 1, "include_domains": ["example.com"]})
|
||||
assert result == {"query": "news", "error": "Serper API error: HTTP 503"}
|
||||
assert len(search.requests) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("options", [{"max_retries": 0}, {"retry_budget_seconds": 1}])
|
||||
def test_budget_does_not_enable_retries(search, options):
|
||||
assert "error" in search([503, 200], options)
|
||||
assert len(search.requests) == 1
|
||||
|
||||
|
||||
def test_jitter_lower_bound(search):
|
||||
search.clock.jitter = 0
|
||||
search([503], {"max_retries": 3})
|
||||
assert search.clock.waits == [0.25, 0.5, 1.0]
|
||||
assert search.clock.ranges == [(0.25, 0.5), (0.5, 1), (1, 2)]
|
||||
|
||||
|
||||
def test_direct_invalid_options(search):
|
||||
assert "max_retries" in search([200], {"max_retries": True}, direct=True)["error"]
|
||||
assert not search.clients
|
||||
|
||||
|
||||
def test_tool_schemas_unchanged():
|
||||
assert set(tools.web_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results", "time_range"}
|
||||
assert set(tools.image_search_tool.args_schema.model_json_schema()["properties"]) == {"query", "max_results"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("outcome", [httpx.ConnectError("connect"), httpx.ConnectTimeout("connect")])
|
||||
def test_connection_exhaustion(search, outcome):
|
||||
assert search([outcome], {"max_retries": 1}) == {"error": "connect", "query": "news"}
|
||||
assert len(search.requests) == 2
|
||||
|
||||
|
||||
def test_last_status_on_budget_exhaustion(search):
|
||||
result = search([503, (429, "30"), 200], {"max_retries": 3})
|
||||
assert result == {"error": "Serper API error: HTTP 429", "query": "news"}
|
||||
assert len(search.requests) == 2
|
||||
assert search.clock.waits == [0.5]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hint", ["Sun Nov 6 08:49:37 1994", "Sunday, 06-Nov-94 08:49:37 GMT"])
|
||||
def test_legacy_http_date_is_valid(search, hint):
|
||||
result = search([(429, hint), 200], {"max_retries": 1})
|
||||
assert "error" not in result
|
||||
assert len(search.requests) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [429, 503])
|
||||
@pytest.mark.parametrize(
|
||||
"current,hint,delay",
|
||||
[
|
||||
(datetime(2023, 11, 14, 22, 13, 20, tzinfo=UTC), "Tue Nov 14 22:13:22 2023", 2.0),
|
||||
(datetime(2023, 11, 14, 22, 13, 20, tzinfo=UTC), "Tue Nov 14 22:13:19 2023", 0.5),
|
||||
(datetime(2023, 11, 5, 8, 49, 37, tzinfo=UTC), "Sun Nov 5 08:49:39 2023", 2.0),
|
||||
(datetime(2023, 11, 5, 8, 49, 37, tzinfo=UTC), "Sun Nov 5 08:49:36 2023", 0.5),
|
||||
],
|
||||
)
|
||||
def test_asctime_hint_uses_utc_and_server_floor(search, non_utc_timezone, status, current, hint, delay):
|
||||
search.clock.wall = current.timestamp()
|
||||
assert "error" not in search([(status, hint), 200], {"max_retries": 1})
|
||||
assert len(search.requests) == 2
|
||||
assert search.clock.waits == [delay]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [429, 503])
|
||||
@pytest.mark.parametrize(
|
||||
"hint,resolved",
|
||||
[
|
||||
("Tuesday, 14-Nov-23 22:13:22 GMT", datetime(2023, 11, 14, 22, 13, 22, tzinfo=UTC)),
|
||||
("Saturday, 14-Nov-26 22:13:20 GMT", datetime(2026, 11, 14, 22, 13, 20, tzinfo=UTC)),
|
||||
("Sunday, 14-Nov-76 22:13:20 GMT", datetime(1976, 11, 14, 22, 13, 20, tzinfo=UTC)),
|
||||
("Tuesday, 14-Nov-73 22:13:20 GMT", datetime(2073, 11, 14, 22, 13, 20, tzinfo=UTC)),
|
||||
("Wednesday, 14-Nov-73 22:13:21 GMT", datetime(1973, 11, 14, 22, 13, 21, tzinfo=UTC)),
|
||||
],
|
||||
)
|
||||
def test_rfc850_hint_year_window_and_scheduling(search, status, hint, resolved):
|
||||
floor = max(0.0, resolved.timestamp() - search.clock.wall)
|
||||
assert tools._retry_after(httpx.Response(status, headers={"Retry-After": hint})) == floor
|
||||
result = search([(status, hint), 200], {"max_retries": 1})
|
||||
if floor >= 30:
|
||||
assert result == {"error": f"Serper API error: HTTP {status}", "query": "news"}
|
||||
assert len(search.requests) == 1
|
||||
assert search.clock.waits == []
|
||||
else:
|
||||
assert "error" not in result
|
||||
assert len(search.requests) == 2
|
||||
assert search.clock.waits == [max(0.5, floor)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hint", ["Wed, 21 Oct 2015 07:28:00 +0100", "Wed, 21 Oct 2015 07:28:00 UTC", "Wed, 21 Oct 2015 07:28 GMT"])
|
||||
def test_email_dates_do_not_enable_429_retry(search, hint):
|
||||
result = search([(429, hint), 200], {"max_retries": 1})
|
||||
assert "error" in result
|
||||
assert len(search.requests) == 1
|
||||
@ -935,6 +935,8 @@ tools:
|
||||
# group: web
|
||||
# use: deerflow.community.serper.tools:web_search_tool
|
||||
# max_results: 5 # capped at 10 by the Serper provider
|
||||
# max_retries: 0 # Opt in with 1-3 extra requests; may incur cost
|
||||
# retry_budget_seconds: 30 # >0 to 300; scheduling budget, not an I/O deadline
|
||||
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /search
|
||||
# include_domains: [] # Optional domain-only lists, at most 10 entries each
|
||||
# exclude_domains: [] # Exclusion wins; see backend/docs/CONFIGURATION.md
|
||||
@ -1278,6 +1280,8 @@ tools:
|
||||
# group: web
|
||||
# use: deerflow.community.serper.tools:image_search_tool
|
||||
# max_results: 5 # capped at 10 by the Serper provider
|
||||
# max_retries: 0 # Opt in with 1-3 extra requests; may incur cost
|
||||
# retry_budget_seconds: 30 # >0 to 300; scheduling budget, not an I/O deadline
|
||||
# # base_url: https://google.serper.dev # Overrides SERPER_BASE_URL; omit /images
|
||||
# # api_key: $SERPER_API_KEY # Optional explicit env-var reference
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user