8 Commits

Author SHA1 Message Date
kongxin
cee54cdd49
fix(web-fetch): stop sending .env.example placeholder keys to providers (#8206)
* fix(web-fetch): stop sending .env.example placeholder keys to providers

make config and make setup copy .env.example into .env, and the template
set nine search/fetch provider keys to example values such as
JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent
that text as a Bearer key, so every web_fetch on a fresh install failed
with 401 while make doctor reported web fetch as configured.

- .env.example comments those nine lines out, like every other credential
  in the template already is.
- provider_credentials.resolve_api_key reads a your-...-key/-token/-secret
  value as unset, so existing .env files that still hold the placeholders
  recover without editing, for every provider that resolves keys there.

* chore: leave the changelog entry to merge time

Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they
merge a pull request.

* fix(web-search): route every provider key and doctor through one placeholder rule

Review follow-up for #8206:

- provider_credentials exposes is_template_placeholder(), the single
  definition of an .env.example example value. key/token/secret must be
  a whole hyphen-separated word, so a real value such as your-monkey or
  your-passkey is no longer dropped.
- Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA
  and You.com resolve config and environment keys through
  resolve_api_key instead of private strip-and-send copies, so an old
  .env that still holds the template value no longer sends it.
- make doctor asks the same predicate: a placeholder key is reported as
  still holding the .env.example value instead of "set", and a literal
  placeholder in config.yaml falls back to the environment key as the
  runtime does.
- A census test keeps credential environment reads under community/
  wrapped in resolve_api_key.

* fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor

Second review follow-up for #8206:

- is_template_placeholder() matches exactly the your-... values that
  .env.example ships instead of a pattern, so a token an operator chose
  (a self-hosted Browserless TOKEN such as your-review-team-token) is
  sent again. A test keeps the set equal to the template.
- make doctor mirrors Browserless: a configured token field is
  authoritative, blank or not, and BROWSERLESS_TOKEN is read only when
  the field is omitted. A blank or placeholder token now warns instead
  of reporting the variable, cross-checked against the client.
- The keyless Jina and You.com providers stay ok but name JINA_API_KEY
  or YDC_API_KEY when a placeholder there is ignored, and a literal
  placeholder api_key in config.yaml is named when no env key backs it.

* fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs

Found while re-checking #8206 against every provider's runtime:

- Browserless used as web_fetch had no doctor key check; it now gets the
  same authoritative token handling as web_capture.
- InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an
  api_key field in its tool config as configured.
- GroundRoute needs a key for every call but had no doctor entry, so a
  missing key reported ok.
- You.com names a placeholder api_key in config that it ignores.
- The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which
  the tool never reads. With JINA_API_KEY now commented out in
  .env.example, that line made config loading fail on a fresh install.
- The provider_credentials docstring now scopes doctor's use of the
  predicate to web provider keys, and the scripts guide no longer says
  .env.example ships live placeholder keys.
2026-11-13 23:23:29 +08:00
creed
3c482e0f58
feat(jina): add bounded crawl completion diagnostics (#6504)
* feat(jina): add bounded crawl completion diagnostics

Signed-off-by: 97three <2212371308@qq.com>

* test(jina): cover diagnostics with request admission

Signed-off-by: 97three <2212371308@qq.com>

---------

Signed-off-by: 97three <2212371308@qq.com>
2026-10-27 10:10:26 +08:00
yc2bgr8
127c2c220c
fix(community): follow Jina API redirects during web fetch (#5808)
* fix(community): 跟随 Jina API 重定向

* test(community): 覆盖 Jina 跨域重定向凭据边界

* test(community): assert Jina redirects retain request body
2026-10-09 19:23:51 +08:00
Daoyuan Li
3ac40bf9bd
test: isolate Jina timeout logging from missing-key warning (#5139) 2026-09-02 14:19:37 +08:00
Ryker_Feng
f92a26d56f
fix(web_fetch): support proxy for Jina reader in restricted networks (#3418) (#3430)
* fix(web_fetch): support proxy for Jina reader in restricted networks

The web_fetch tool built a bare httpx.AsyncClient() with no proxy
awareness, so users behind a corporate proxy / in Docker / WSL could
not reach https://r.jina.ai and web_fetch timed out.

- Add optional `proxy` / `trust_env` params to JinaClient.crawl and
  wire them from the `web_fetch` tool config (with type coercion for
  YAML string values).
- Pass internal service hostnames through NO_PROXY in both compose
  files so proxy env inherited via env_file does not break in-cluster
  calls (gateway/provisioner/etc).
- Load proxy vars from .env into the shell in scripts/docker.sh so the
  NO_PROXY interpolation can merge user-provided values on `make` path.
- Document proxy/trust_env options in config.example.yaml.

Closes #3418

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-06-08 23:25:29 +08:00
d 🔹
e8572b9d0c
fix(jina): log transient failures at WARNING without traceback (#2484) (#2485)
The exception handler in JinaClient.crawl used logger.exception, which
emits an ERROR-level record with the full httpx/httpcore/anyio traceback
for every transient network failure (timeout, connection refused). Other
search/crawl providers in the project log the same class of recoverable
failures as a single line. One offline/slow-network session could produce
dozens of multi-frame ERROR stack traces, drowning out real problems.

Switch to logger.warning with a concise message that includes the
exception type and its str, matching the style used elsewhere for
recoverable transient failures (aio_sandbox, ddg, etc.). The exception
type now also surfaces into the returned "Error: ..." string so callers
retain diagnostic signal.

Adds a regression test that asserts the log record is WARNING, carries
no exc_info, and includes the exception class name.

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-24 16:00:14 +08:00
lesliewangwyc-dev
1df389b9d0
fix: wrap blocking readability call with asyncio.to_thread in web_fetch (#2157)
* fix: wrap blocking readability call with asyncio.to_thread in web_fetch

The readability extractor internally spawns a Node.js subprocess via
readabilipy, which blocks the async event loop and causes a
BlockingError when web_fetch is invoked inside LangGraph's async
runtime.

Wrap the synchronous extract_article call with asyncio.to_thread to
offload it to a thread pool, unblocking the event loop.

Note: community/infoquest/tools.py has the same latent issue and
should be addressed in a follow-up PR.

Closes #2152

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: verify web_fetch offloads extraction via asyncio.to_thread

Add a regression test that monkeypatches asyncio.to_thread to confirm
readability extraction is offloaded to a worker thread, preventing
future refactors from reintroducing the blocking call.

Addresses Copilot review feedback on #2157.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-13 21:15:24 +08:00
Shengyuan Wang
2f3744f807
refactor: replace sync requests with async httpx in Jina AI client (#1603)
* refactor: replace sync requests with async httpx in Jina AI client

Replace synchronous `requests.post()` with `httpx.AsyncClient` in
JinaClient.crawl() and make web_fetch_tool async. This is part of the
planned async concurrency optimization for the agent hot path
(see docs/TODO.md).

* fix: address Copilot review feedback on async Jina client

- Short-circuit error strings in web_fetch_tool before passing to
  ReadabilityExtractor, preventing misleading extraction results
- Log missing JINA_API_KEY warning only once per process to reduce
  noise under concurrent async fetching
- Use logger.exception instead of logger.error in crawl exception
  handler to preserve stack traces for debugging
- Add async web_fetch_tool tests and warn-once coverage

* fix: mock get_app_config in web_fetch_tool tests for CI

The web_fetch_tool tests failed in CI because get_app_config requires
a config.yaml file that isn't present in the test environment. Mock
the config loader to remove the filesystem dependency.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 17:02:39 +08:00