diff --git a/CHANGELOG.md b/CHANGELOG.md index fa8a13c36e..b74bce94f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -327,6 +327,18 @@ This release closes that milestone with **181 merged pull requests**. ### Fixed +- **config:** `make config-upgrade` (also run by `make dev` / `make start`) + upgrades the `config.yaml` the Gateway loads. With both + `/config.yaml` and `backend/config.yaml` present, the script + upgraded the `backend/` copy while the Gateway read the checkout copy, so the + file in use stayed outdated even though the upgrade reported success. It also + ignored `DEER_FLOW_PROJECT_ROOT` and a `DEER_FLOW_CONFIG_PATH` set in `.env`, + and fell back to another file when `DEER_FLOW_CONFIG_PATH` named a missing + one. The script now asks the harness resolver + (`AppConfig.resolve_config_path`) for the file, and a missing + `DEER_FLOW_CONFIG_PATH` or invalid `DEER_FLOW_PROJECT_ROOT` fails with the + Gateway's error instead of upgrading a fallback. ([#5991]) + - **sandbox:** Unwrap `Overwrite`-wrapped sandbox state in `ToolOutputBudgetMiddleware` and `ReadBeforeWriteMiddleware`. In delta checkpoint mode, forked or restored threads deliver the `sandbox` channel @@ -6233,5 +6245,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#5981]: https://github.com/bytedance/deer-flow/pull/5981 [#5982]: https://github.com/bytedance/deer-flow/pull/5982 [#5987]: https://github.com/bytedance/deer-flow/pull/5987 +[#5991]: https://github.com/bytedance/deer-flow/pull/5991 [#6015]: https://github.com/bytedance/deer-flow/pull/6015 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index bf60e85c05..de0fc15209 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -332,6 +332,17 @@ ### 修复 +- **配置:** `make config-upgrade`(`make dev` / `make start` 也会执行)现在升级的是 + Gateway 实际加载的 `config.yaml`。当 `/config.yaml` 与 + `backend/config.yaml` 同时存在时,脚本升级的是 `backend/` 下的副本,而 Gateway + 读取的是 checkout 根目录的副本,因此实际使用的文件仍停留在旧版本,升级却显示 + 成功。脚本还会忽略 `DEER_FLOW_PROJECT_ROOT` 以及 `.env` 中设置的 + `DEER_FLOW_CONFIG_PATH`,并在 `DEER_FLOW_CONFIG_PATH` 指向不存在的文件时退回到 + 其他文件。现在脚本通过 harness 的解析器 + (`AppConfig.resolve_config_path`)确定文件;`DEER_FLOW_CONFIG_PATH` 不存在或 + `DEER_FLOW_PROJECT_ROOT` 无效时,会以 Gateway 相同的错误失败,而不是升级 + 回退文件。([#5991]) + - **沙箱:** 在中间件(`ToolOutputBudgetMiddleware` 与 `ReadBeforeWriteMiddleware`) 中解包 `Overwrite` 包装的沙箱状态。在 delta checkpoint 模式下,分叉或回滚 的对话交付的 `sandbox` 通道状态会被 LangGraph 的 `Overwrite` 包装。此前直接判断 @@ -5287,4 +5298,5 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子 [#5981]: https://github.com/bytedance/deer-flow/pull/5981 [#5982]: https://github.com/bytedance/deer-flow/pull/5982 [#5987]: https://github.com/bytedance/deer-flow/pull/5987 +[#5991]: https://github.com/bytedance/deer-flow/pull/5991 [#6015]: https://github.com/bytedance/deer-flow/pull/6015 diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index 13f332fe20..587f29e825 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -109,7 +109,7 @@ Run `make config-upgrade` to merge new fields into your config. ``` - **Missing `config_version`** in your config is treated as version 0. -- Run `make config-upgrade` to auto-merge missing fields (your existing values are preserved, a `.bak` backup is created). +- Run `make config-upgrade` to auto-merge missing fields (your existing values are preserved, a `.bak` backup is created). It upgrades the file the Gateway loads, resolved as in [Configuration Priority](#configuration-priority). - When changing the config schema, bump `config_version` in `config.example.yaml`. ## Configuration Sections diff --git a/backend/packages/harness/deerflow/config/AGENTS.md b/backend/packages/harness/deerflow/config/AGENTS.md index 3f7b3b6e2d..2d42aed8c5 100644 --- a/backend/packages/harness/deerflow/config/AGENTS.md +++ b/backend/packages/harness/deerflow/config/AGENTS.md @@ -57,6 +57,8 @@ Configuration priority: 3. `config.yaml` under `DEER_FLOW_PROJECT_ROOT`, or the current directory when it is unset 4. Legacy `backend/config.yaml`, then repository-root `config.yaml` (project root is the **recommended location**) +`scripts/config-upgrade.sh` calls `AppConfig.resolve_config_path` rather than copying this order. +The legacy locations are anchored to the installed harness source, not to the caller's checkout. `scripts/doctor.py` calls `AppConfig.resolve_config_path` rather than copying this order. Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`). diff --git a/backend/tests/test_config_version.py b/backend/tests/test_config_version.py index 9f1b608243..ddfec03ce0 100644 --- a/backend/tests/test_config_version.py +++ b/backend/tests/test_config_version.py @@ -11,7 +11,7 @@ import pytest import yaml from support.shell import find_script_bash -from deerflow.config.app_config import AppConfig +from deerflow.config.app_config import AppConfig, _legacy_config_candidates from deerflow.config.knowledge_base_config import KnowledgeBaseConfig from deerflow.tools.tools import get_available_tools @@ -625,3 +625,154 @@ def test_version_46_pii_disabled_config_upgrade_skips_token_secret(tmp_path): assert upgraded["config_version"] == expected_version assert "token_secret" not in upgraded["pii_redaction"] AppConfig.model_validate(upgraded) + + +def _run_config_upgrade_in_checkout(checkout: Path, **env_overrides: str): + """Run a copy of scripts/config-upgrade.sh from a throwaway checkout. + + The script resolves the checkout from its own location, so the copy sees + ``checkout`` as the repository root while ``UV_PROJECT`` keeps its + ``uv run`` on the real backend environment. Resolution overrides are + cleared so the developer's shell cannot pick the file under test. + + The harness's legacy fallback is anchored to its install location, which + here is the real repository, so a run may only reach it when no legacy + candidate exists: otherwise the script would upgrade the developer's own + ``config.yaml``. + """ + import shutil + import subprocess + + project_root = Path(env_overrides.get("DEER_FLOW_PROJECT_ROOT", checkout)) + if "DEER_FLOW_CONFIG_PATH" not in env_overrides and project_root.is_dir() and not (project_root / "config.yaml").is_file(): + assert not any(path.exists() for path in _legacy_config_candidates()), "would fall back to the real repository's config.yaml" + + repo_root = Path(__file__).resolve().parents[2] + (checkout / "scripts").mkdir(parents=True, exist_ok=True) + (checkout / "backend").mkdir(exist_ok=True) + shutil.copy2(repo_root / "scripts" / "config-upgrade.sh", checkout / "scripts" / "config-upgrade.sh") + shutil.copy2(repo_root / "config.example.yaml", checkout / "config.example.yaml") + + env = {key: value for key, value in os.environ.items() if key not in {"DEER_FLOW_CONFIG_PATH", "DEER_FLOW_PROJECT_ROOT"}} + env["UV_PROJECT"] = str(repo_root / "backend") + env.update(env_overrides) + return subprocess.run( + [SCRIPT_BASH, str(checkout / "scripts" / "config-upgrade.sh")], + env=env, + capture_output=True, + text=True, + timeout=120, + ) + + +def _write_outdated_config(path: Path) -> str: + path.parent.mkdir(parents=True, exist_ok=True) + text = yaml.dump({"config_version": 1, "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}}) + path.write_text(text, encoding="utf-8") + return text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_targets_checkout_config_over_legacy_backend_copy(tmp_path): + """With both copies present, upgrade the checkout config.yaml that `make dev` loads.""" + checkout = tmp_path / "checkout" + _write_outdated_config(checkout / "config.yaml") + legacy_text = _write_outdated_config(checkout / "backend" / "config.yaml") + + result = _run_config_upgrade_in_checkout(checkout) + assert result.returncode == 0, result.stdout + result.stderr + + expected_version = yaml.safe_load((checkout / "config.example.yaml").read_text(encoding="utf-8"))["config_version"] + upgraded = yaml.safe_load((checkout / "config.yaml").read_text(encoding="utf-8")) + assert upgraded["config_version"] == expected_version + assert (checkout / "backend" / "config.yaml").read_text(encoding="utf-8") == legacy_text + assert not (checkout / "backend" / "config.yaml.bak").exists() + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_honors_deer_flow_project_root(tmp_path): + """An exported DEER_FLOW_PROJECT_ROOT decides the file, as it does for the Gateway.""" + checkout = tmp_path / "checkout" + project_root = tmp_path / "project" + checkout_text = _write_outdated_config(checkout / "config.yaml") + _write_outdated_config(project_root / "config.yaml") + + result = _run_config_upgrade_in_checkout(checkout, DEER_FLOW_PROJECT_ROOT=str(project_root)) + assert result.returncode == 0, result.stdout + result.stderr + + upgraded = yaml.safe_load((project_root / "config.yaml").read_text(encoding="utf-8")) + assert upgraded["config_version"] > 1 + assert (checkout / "config.yaml").read_text(encoding="utf-8") == checkout_text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_fails_on_missing_deer_flow_config_path(tmp_path): + """A missing explicit config stops the Gateway, so no fallback file is upgraded instead.""" + checkout = tmp_path / "checkout" + checkout_text = _write_outdated_config(checkout / "config.yaml") + legacy_text = _write_outdated_config(checkout / "backend" / "config.yaml") + missing = tmp_path / "missing" / "config.yaml" + + result = _run_config_upgrade_in_checkout(checkout, DEER_FLOW_CONFIG_PATH=str(missing)) + assert result.returncode != 0 + assert "DEER_FLOW_CONFIG_PATH" in result.stderr + assert (checkout / "config.yaml").read_text(encoding="utf-8") == checkout_text + assert (checkout / "backend" / "config.yaml").read_text(encoding="utf-8") == legacy_text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_reports_invalid_deer_flow_project_root(tmp_path): + """A project root the Gateway rejects fails with its message, not a traceback.""" + checkout = tmp_path / "checkout" + checkout_text = _write_outdated_config(checkout / "config.yaml") + + result = _run_config_upgrade_in_checkout(checkout, DEER_FLOW_PROJECT_ROOT=str(tmp_path / "missing")) + assert result.returncode != 0 + assert "DEER_FLOW_PROJECT_ROOT is set to" in result.stderr + assert "Traceback" not in result.stderr + assert (checkout / "config.yaml").read_text(encoding="utf-8") == checkout_text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_honors_deer_flow_config_path_from_dotenv(tmp_path): + """`make config-upgrade` does not source .env, but the Gateway loads it; follow the Gateway.""" + checkout = tmp_path / "checkout" + live = tmp_path / "live" / "config.yaml" + checkout_text = _write_outdated_config(checkout / "config.yaml") + _write_outdated_config(live) + (checkout / ".env").write_text(f"DEER_FLOW_CONFIG_PATH={live}\n", encoding="utf-8") + + result = _run_config_upgrade_in_checkout(checkout) + assert result.returncode == 0, result.stdout + result.stderr + + assert yaml.safe_load(live.read_text(encoding="utf-8"))["config_version"] > 1 + assert (checkout / "config.yaml").read_text(encoding="utf-8") == checkout_text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_resolves_relative_deer_flow_config_path_from_backend(tmp_path): + """The Gateway runs from backend/, so a relative DEER_FLOW_CONFIG_PATH is relative to it.""" + checkout = tmp_path / "checkout" + checkout_text = _write_outdated_config(checkout / "config.yaml") + _write_outdated_config(checkout / "backend" / "custom.yaml") + + result = _run_config_upgrade_in_checkout(checkout, DEER_FLOW_CONFIG_PATH="custom.yaml") + assert result.returncode == 0, result.stdout + result.stderr + + upgraded = yaml.safe_load((checkout / "backend" / "custom.yaml").read_text(encoding="utf-8")) + assert upgraded["config_version"] > 1 + assert (checkout / "config.yaml").read_text(encoding="utf-8") == checkout_text + + +@pytest.mark.skipif(SCRIPT_BASH is None, reason="repo shell-script tests need Git Bash on Windows") +def test_config_upgrade_creates_checkout_config_when_none_exists(tmp_path): + """With no config anywhere, the resolver miss is not an error: the example seeds /config.yaml.""" + if any(path.exists() for path in _legacy_config_candidates()): + pytest.skip("the resolver's legacy fallback would find this repository's own config.yaml") + checkout = tmp_path / "checkout" + + result = _run_config_upgrade_in_checkout(checkout) + assert result.returncode == 0, result.stdout + result.stderr + + assert "creating from example" in result.stdout + assert (checkout / "config.yaml").read_bytes() == (checkout / "config.example.yaml").read_bytes() diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index c5888bcbb3..8d2dd6b55f 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -57,6 +57,14 @@ Pinned by `backend/tests/test_doctor.py::TestMainConfigResolution`. Root `make install` runs pre-commit through uv, so uv's tool bin directory need not be on `PATH`. +`config-upgrade.sh` upgrades the file the Gateway loads by asking the harness +(`AppConfig.resolve_config_path`) rather than copying its lookup order. It +defaults `DEER_FLOW_PROJECT_ROOT` to the checkout, as `serve.sh` does, so +`/config.yaml` wins over a legacy `backend/config.yaml`. A missing +`DEER_FLOW_CONFIG_PATH` or invalid project root is an error, never a fallback. +Only "no config anywhere" creates `/config.yaml` from the example. +`backend/tests/test_config_version.py::test_config_upgrade_*` pins this. + ## Shell Script Invocation Contract Root Makefile recipes must invoke repository `.sh` files through diff --git a/scripts/config-upgrade.sh b/scripts/config-upgrade.sh index 625324057a..81e3411e2d 100755 --- a/scripts/config-upgrade.sh +++ b/scripts/config-upgrade.sh @@ -11,17 +11,35 @@ set -e REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" EXAMPLE="$REPO_ROOT/config.example.yaml" -# Resolve config.yaml location: env var > backend/ > repo root -if [ -n "$DEER_FLOW_CONFIG_PATH" ] && [ -f "$DEER_FLOW_CONFIG_PATH" ]; then - CONFIG="$DEER_FLOW_CONFIG_PATH" -elif [ -f "$REPO_ROOT/backend/config.yaml" ]; then - CONFIG="$REPO_ROOT/backend/config.yaml" -elif [ -f "$REPO_ROOT/config.yaml" ]; then - CONFIG="$REPO_ROOT/config.yaml" +if command -v cygpath >/dev/null 2>&1; then + REPO_ROOT_WIN="$(cygpath -w "$REPO_ROOT")" else - CONFIG="" + REPO_ROOT_WIN="$REPO_ROOT" fi +# Upgrade the config.yaml the Gateway loads. Ask the harness resolver rather +# than copying its order: with both /config.yaml and +# backend/config.yaml present, `make dev` reads the checkout copy. The import +# loads .env as the Gateway does; DEER_FLOW_PROJECT_ROOT then defaults to the +# checkout, as in serve.sh. Prints nothing when no config exists yet. +CONFIG="$(cd "$REPO_ROOT/backend" && REPO_ROOT_WIN_PATH="$REPO_ROOT_WIN" uv run python -c " +import os +import sys + +from deerflow.config.app_config import AppConfig + +os.environ.setdefault('DEER_FLOW_PROJECT_ROOT', os.environ['REPO_ROOT_WIN_PATH']) +try: + sys.stdout.write(str(AppConfig.resolve_config_path())) +except FileNotFoundError as exc: + # An explicit DEER_FLOW_CONFIG_PATH that does not exist stops the Gateway + # too; never upgrade a fallback file in its place. + if os.environ.get('DEER_FLOW_CONFIG_PATH'): + sys.exit(f'ERROR {exc}') +except ValueError as exc: + sys.exit(f'ERROR {exc}') +")" + if [ ! -f "$EXAMPLE" ]; then echo "✗ config.example.yaml not found at $EXAMPLE" exit 1