mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-15 07:41:20 +00:00
cee54cdd49
* fix(web-fetch): stop sending .env.example placeholder keys to providers make config and make setup copy .env.example into .env, and the template set nine search/fetch provider keys to example values such as JINA_API_KEY=your-jina-api-key. The default keyless Jina Reader sent that text as a Bearer key, so every web_fetch on a fresh install failed with 401 while make doctor reported web fetch as configured. - .env.example comments those nine lines out, like every other credential in the template already is. - provider_credentials.resolve_api_key reads a your-...-key/-token/-secret value as unset, so existing .env files that still hold the placeholders recover without editing, for every provider that resolves keys there. * chore: leave the changelog entry to merge time Per CONTRIBUTING.md (#8202), maintainers add changelog entries when they merge a pull request. * fix(web-search): route every provider key and doctor through one placeholder rule Review follow-up for #8206: - provider_credentials exposes is_template_placeholder(), the single definition of an .env.example example value. key/token/secret must be a whole hyphen-separated word, so a real value such as your-monkey or your-passkey is no longer dropped. - Serper, Serply, Sofya, SpecStream, Unbrowse, Webz, Brave, Tencent WSA and You.com resolve config and environment keys through resolve_api_key instead of private strip-and-send copies, so an old .env that still holds the template value no longer sends it. - make doctor asks the same predicate: a placeholder key is reported as still holding the .env.example value instead of "set", and a literal placeholder in config.yaml falls back to the environment key as the runtime does. - A census test keeps credential environment reads under community/ wrapped in resolve_api_key. * fix(web-search): match shipped placeholders exactly and mirror Browserless precedence in doctor Second review follow-up for #8206: - is_template_placeholder() matches exactly the your-... values that .env.example ships instead of a pattern, so a token an operator chose (a self-hosted Browserless TOKEN such as your-review-team-token) is sent again. A test keeps the set equal to the template. - make doctor mirrors Browserless: a configured token field is authoritative, blank or not, and BROWSERLESS_TOKEN is read only when the field is omitted. A blank or placeholder token now warns instead of reporting the variable, cross-checked against the client. - The keyless Jina and You.com providers stay ok but name JINA_API_KEY or YDC_API_KEY when a placeholder there is ignored, and a literal placeholder api_key in config.yaml is named when no env key backs it. * fix(doctor): match each web provider's key precedence; drop the dead Jina api_key from docs Found while re-checking #8206 against every provider's runtime: - Browserless used as web_fetch had no doctor key check; it now gets the same authoritative token handling as web_capture. - InfoQuest reads only INFOQUEST_API_KEY, so doctor no longer counts an api_key field in its tool config as configured. - GroundRoute needs a key for every call but had no doctor entry, so a missing key reported ok. - You.com names a placeholder api_key in config that it ignores. - The Jina web_fetch docs snippet carried api_key: $JINA_API_KEY, which the tool never reads. With JINA_API_KEY now commented out in .env.example, that line made config loading fail on a fresh install. - The provider_credentials docstring now scopes doctor's use of the predicate to web provider keys, and the scripts guide no longer says .env.example ships live placeholder keys.
94 lines
4.8 KiB
Markdown
94 lines
4.8 KiB
Markdown
# Orchestration scripts (`scripts/`)
|
|
|
|
Scripts behind the root `Makefile`: local start and stop (`serve.sh`), Docker
|
|
development (`docker.sh`), production deployment (`deploy.sh`), setup and
|
|
diagnosis (`setup_wizard.py`, `doctor.py`, `support_bundle.py`), static analysis
|
|
and repository checks. Each script documents its own flags; run it with `--help`
|
|
or read its header.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
cd backend
|
|
python -m pytest tests -q -k "serve or deploy or compose or doctor or dotenv or script or wizard or support_bundle"
|
|
```
|
|
|
|
The shell suites run the real scripts against stub `docker` and `uv` binaries.
|
|
Cases marked for real Compose run only when a `docker` CLI is installed.
|
|
|
|
## Portability
|
|
|
|
- **Make recipes call `.sh` files through `RUN_SHELL_SCRIPT`**, and a script that
|
|
calls a sibling script prefixes it with `bash`. A source archive,
|
|
`core.fileMode=false` or a non-POSIX filesystem does not keep executable bits.
|
|
- **Host pnpm calls go through `scripts/pnpm.py`.** It chooses the launcher by
|
|
Python's `os.name`, not by the invoking shell, so native Windows, MSYS and POSIX
|
|
each get the right one.
|
|
- **Shell filters stay compatible with GNU and BSD sed**, because macOS ships
|
|
the BSD tools. `sandbox-mode.sh` uses Bash and awk only.
|
|
- **Lookups that run before the environment exists use the standard library
|
|
only.** Config detection, label checks and the dotenv helpers run before
|
|
`uv sync`.
|
|
- **Python entry points guard their stream encoding.** Windows consoles and
|
|
redirected CI logs are not UTF-8 by default, and embedded or captured streams
|
|
have no `reconfigure`. Read config and credential JSON with or without a BOM,
|
|
as the runtime loader does.
|
|
|
|
## Config and dotenv
|
|
|
|
- **Act on the file the Gateway would load.** `doctor.py` and `config-upgrade.sh`
|
|
ask the harness for the resolved config path. A fixed `config.yaml` path
|
|
diagnoses or upgrades the wrong file when an override is set.
|
|
- **Never source a dotenv file.** Parse it and quote path arguments. A dotenv
|
|
value overrides nothing that the shell exported, including an exported empty
|
|
value.
|
|
- **`deploy.sh` lets Compose parse `.env`.** Compose accepts forms that a
|
|
`KEY=VALUE` grep misses, so the script asks `docker compose config` what a key
|
|
resolves to. Do not export a value the script read from `.env`: that shadows
|
|
Compose's own parsing, and `make up` then replaces the operator's secret with a
|
|
generated one.
|
|
- **Secrets resolve shell, then `.env`, then the persisted file, then a new
|
|
value.** A failing probe stops the script; it does not guess.
|
|
- **One dotenv selection per deployment.** `DEER_FLOW_COMPOSE_ENV_FILE` is
|
|
resolved once, and Compose interpolation, service env files, secret probes and
|
|
extras detection all use that path.
|
|
- **Do not seed production `.env` from the examples.** They hold development
|
|
defaults and sample keys, not values an operator chose for production. Both
|
|
compose files mark the env files optional so a fresh checkout still starts and
|
|
stops.
|
|
- **Support bundles export no secrets** and show an external runtime home as
|
|
`{DEER_FLOW_HOME}`.
|
|
- **Doctor does not echo model credentials.** Obvious sample key values fail the
|
|
model readiness check even when the environment variable is nonempty.
|
|
|
|
## Start, stop, clean
|
|
|
|
- **Validate before you stop anything.** `serve.sh` resolves the bind host and
|
|
config first, so a bad value cannot tear down a running stack.
|
|
- **`down` skips secret and permission checks.** Permission damage in the runtime
|
|
home must not prevent teardown.
|
|
- **`make clean` deletes the runtime home**, which the compose stacks mount. It
|
|
refuses while the Gateway container runs, and it stops sandbox containers
|
|
before the delete because their thread mounts live in that tree.
|
|
- **Match container names exactly.** Docker's `name=` filter is a substring match.
|
|
`cleanup-containers.sh` verifies the anchored prefix afterwards, so it cannot
|
|
remove another project's container.
|
|
- **Runtime commands use `uv run --no-sync`**, which keeps installed extras such
|
|
as database drivers.
|
|
- **Browser sessions need a single Gateway worker process.** They are
|
|
process-local, so launchers keep `WEB_CONCURRENCY` unset or `1`.
|
|
- **The multi-instance dev harness keeps IM channels disabled** unless asked.
|
|
Credentials saved through the UI could otherwise connect a real bot later.
|
|
|
|
## Checks and analysis
|
|
|
|
- **Skill review waivers take effect from the base revision only.** The head
|
|
manifest is validated but cannot authorize a finding in its own pull request.
|
|
Blockers are never waivable.
|
|
- **`detect-blocking-io` is a static approximation.** It resolves calls by name
|
|
inside one file and over-reports on purpose. The strict gate is the runtime
|
|
suite `make test-blocking-io`; `scan_changed_blocking_io.py` limits the static
|
|
findings to a diff.
|
|
- **`detect-thread-boundaries --runtime-config` imports configured tools and
|
|
models.** It does not call models or external services.
|