mirror of https://github.com/bytedance/deer-flow.git synced 2026-11-15 07:41:20 +00:00

fix(web-fetch): stop sending .env.example placeholder keys to providers (#8206 )

* fix(web-fetch): stop sending .env.example placeholder keys to providers

make config and make setup copy .env.example into .env, and the template
set nine search/fetch provider keys to example values such as
JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent
that text as a Bearer key, so every web_fetch on a fresh install failed
with 401 while make doctor reported web fetch as configured.

- .env.example comments those nine lines out, like every other credential
in the template already is.
- provider_credentials.resolve_api_key reads a your-...-key/-token/-secret
value as unset, so existing .env files that still hold the placeholders
recover without editing, for every provider that resolves keys there.

* chore: leave the changelog entry to merge time

Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they
merge a pull request.

* fix(web-search): route every provider key and doctor through one placeholder rule

Review follow-up for #8206:

- provider_credentials exposes is_template_placeholder(), the single
definition of an .env.example example value. key/token/secret must be
a whole hyphen-separated word, so a real value such as your-monkey or
your-passkey is no longer dropped.
- Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA
and You.com resolve config and environment keys through
resolve_api_key instead of private strip-and-send copies, so an old
.env that still holds the template value no longer sends it.
- make doctor asks the same predicate: a placeholder key is reported as
still holding the .env.example value instead of "set", and a literal
placeholder in config.yaml falls back to the environment key as the
runtime does.
- A census test keeps credential environment reads under community/
wrapped in resolve_api_key.

* fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor

Second review follow-up for #8206:

- is_template_placeholder() matches exactly the your-... values that
.env.example ships instead of a pattern, so a token an operator chose
(a self-hosted Browserless TOKEN such as your-review-team-token) is
sent again. A test keeps the set equal to the template.
- make doctor mirrors Browserless: a configured token field is
authoritative, blank or not, and BROWSERLESS_TOKEN is read only when
the field is omitted. A blank or placeholder token now warns instead
of reporting the variable, cross-checked against the client.
- The keyless Jina and You.com providers stay ok but name JINA_API_KEY
or YDC_API_KEY when a placeholder there is ignored, and a literal
placeholder api_key in config.yaml is named when no env key backs it.

* fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs

Found while re-checking #8206 against every provider's runtime:

- Browserless used as web_fetch had no doctor key check; it now gets the
same authoritative token handling as web_capture.
- InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an
api_key field in its tool config as configured.
- GroundRoute needs a key for every call but had no doctor entry, so a
missing key reported ok.
- You.com names a placeholder api_key in config that it ignores.
- The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which
the tool never reads. With JINA_API_KEY now commented out in
.env.example, that line made config loading fail on a fresh install.
- The provider_credentials docstring now scopes doctor's use of the
predicate to web provider keys, and the scripts guide no longer says
.env.example ships live placeholder keys.

2026-11-13 23:23:29 +08:00

4.8 KiB

Raw Blame History

Orchestration scripts (`scripts/`)

Scripts behind the root Makefile: local start and stop (serve.sh), Docker development (docker.sh), production deployment (deploy.sh), setup and diagnosis (setup_wizard.py, doctor.py, support_bundle.py), static analysis and repository checks. Each script documents its own flags; run it with --help or read its header.

Verify

cd backend
python -m pytest tests -q -k "serve or deploy or compose or doctor or dotenv or script or wizard or support_bundle"

The shell suites run the real scripts against stub docker and uv binaries. Cases marked for real Compose run only when a docker CLI is installed.

Portability

Make recipes call .sh files through RUN_SHELL_SCRIPT, and a script that calls a sibling script prefixes it with bash. A source archive, core.fileMode=false or a non-POSIX filesystem does not keep executable bits.
Host pnpm calls go through scripts/pnpm.py. It chooses the launcher by Python's os.name, not by the invoking shell, so native Windows, MSYS and POSIX each get the right one.
Shell filters stay compatible with GNU and BSD sed, because macOS ships the BSD tools. sandbox-mode.sh uses Bash and awk only.
Lookups that run before the environment exists use the standard library only. Config detection, label checks and the dotenv helpers run before uv sync.
Python entry points guard their stream encoding. Windows consoles and redirected CI logs are not UTF-8 by default, and embedded or captured streams have no reconfigure. Read config and credential JSON with or without a BOM, as the runtime loader does.

Config and dotenv

Act on the file the Gateway would load. doctor.py and config-upgrade.sh ask the harness for the resolved config path. A fixed config.yaml path diagnoses or upgrades the wrong file when an override is set.
Never source a dotenv file. Parse it and quote path arguments. A dotenv value overrides nothing that the shell exported, including an exported empty value.
deploy.sh lets Compose parse .env. Compose accepts forms that a KEY=VALUE grep misses, so the script asks docker compose config what a key resolves to. Do not export a value the script read from .env: that shadows Compose's own parsing, and make up then replaces the operator's secret with a generated one.
Secrets resolve shell, then .env, then the persisted file, then a new value. A failing probe stops the script; it does not guess.
One dotenv selection per deployment. DEER_FLOW_COMPOSE_ENV_FILE is resolved once, and Compose interpolation, service env files, secret probes and extras detection all use that path.
Do not seed production .env from the examples. They hold development defaults and sample keys, not values an operator chose for production. Both compose files mark the env files optional so a fresh checkout still starts and stops.
Support bundles export no secrets and show an external runtime home as {DEER_FLOW_HOME}.
Doctor does not echo model credentials. Obvious sample key values fail the model readiness check even when the environment variable is nonempty.

Start, stop, clean

Validate before you stop anything. serve.sh resolves the bind host and config first, so a bad value cannot tear down a running stack.
down skips secret and permission checks. Permission damage in the runtime home must not prevent teardown.
make clean deletes the runtime home, which the compose stacks mount. It refuses while the Gateway container runs, and it stops sandbox containers before the delete because their thread mounts live in that tree.
Match container names exactly. Docker's name= filter is a substring match. cleanup-containers.sh verifies the anchored prefix afterwards, so it cannot remove another project's container.
Runtime commands use uv run --no-sync, which keeps installed extras such as database drivers.
Browser sessions need a single Gateway worker process. They are process-local, so launchers keep WEB_CONCURRENCY unset or 1.
The multi-instance dev harness keeps IM channels disabled unless asked. Credentials saved through the UI could otherwise connect a real bot later.

Checks and analysis

Skill review waivers take effect from the base revision only. The head manifest is validated but cannot authorize a finding in its own pull request. Blockers are never waivable.
detect-blocking-io is a static approximation. It resolves calls by name inside one file and over-reports on purpose. The strict gate is the runtime suite make test-blocking-io; scan_changed_blocking_io.py limits the static findings to a diff.
detect-thread-boundaries --runtime-config imports configured tools and models. It does not call models or external services.

4.8 KiB Raw Blame History

Orchestration scripts (scripts/)