diff --git a/CHANGELOG.md b/CHANGELOG.md index da78341425..f94b814fa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -623,6 +623,12 @@ This release closes that milestone with **439 merged pull requests**. ### Fixed +- **deploy:** `make up`, `make down` and `make prod-logs` no longer stop on a + fresh checkout with `env file .../.env not found`. `.env` and `frontend/.env` are gitignored and + `make up` does not create them, but the production compose file required + both. Its `env_file` entries are now optional, as in the development compose + file: Compose loads them when present and skips them otherwise. This needs + Docker Compose 2.24 or newer, the floor the README already documents. ([#6370]) - **memory:** DeerMem relevance ranking no longer counts punctuation as query terms when jieba is installed (`memory-zh` extra). `tokenize()` dropped only whitespace from `jieba.cut`, which emits `,`, `。`, `,` and `!` as standalone @@ -8869,4 +8875,5 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#6351]: https://github.com/bytedance/deer-flow/pull/6351 [#6354]: https://github.com/bytedance/deer-flow/pull/6354 [#6365]: https://github.com/bytedance/deer-flow/pull/6365 +[#6370]: https://github.com/bytedance/deer-flow/pull/6370 [#6388]: https://github.com/bytedance/deer-flow/pull/6388 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index 4ab3e29c5f..fa87e75b8c 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -558,6 +558,11 @@ ### 修复 +- **部署:** 在全新检出的仓库上,`make up`、`make down` 与 `make prod-logs` + 不再因 `env file .../.env not found` 而中止。`.env` 与 + `frontend/.env` 已被 gitignore,`make up` 也不会创建它们,但生产 compose 文件此前要求两者都存在。 + 现在其 `env_file` 条目与开发 compose 文件一样为可选:文件存在时加载,不存在时跳过。此写法需要 + Docker Compose 2.24 或更高版本,即 README 已注明的最低版本。([#6370]) - **记忆:** 安装 jieba(`memory-zh` 扩展)时,DeerMem 相关度排序不再把标点计为查询词。 `tokenize()` 只过滤了 `jieba.cut` 输出中的空白,而 jieba 会把 `,`、`。`、`,`、`!` 切成独立 token,因此同样含逗号的查询与无关事实也会得到非零分,近似去重的相似度被抬高, @@ -7324,4 +7329,5 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子 [#6350]: https://github.com/bytedance/deer-flow/pull/6350 [#6351]: https://github.com/bytedance/deer-flow/pull/6351 [#6354]: https://github.com/bytedance/deer-flow/pull/6354 +[#6370]: https://github.com/bytedance/deer-flow/pull/6370 [#6388]: https://github.com/bytedance/deer-flow/pull/6388 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8bff80c896..b830776fa8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,8 +13,8 @@ Docker provides a consistent, isolated environment with all dependencies pre-con #### Prerequisites - Docker Desktop or Docker Engine -- Docker Compose **v2.24 or newer** (check with `docker compose version`). The dev - Compose file marks its `env_file` entries optional using the long-form +- Docker Compose **v2.24 or newer** (check with `docker compose version`). Both + Compose files mark their `env_file` entries optional using the long-form `path`/`required` syntax; older clients reject it with `services.gateway.env_file.0 must be a string`. `make docker-start` verifies the version and tells you to upgrade — direct `docker compose` callers get that raw diff --git a/README.md b/README.md index b87d9a3f0f..9cdc5b0bf6 100644 --- a/README.md +++ b/README.md @@ -393,7 +393,7 @@ Use the table below as a practical starting point when choosing how to run DeerF Requires Docker Desktop / Docker Engine and **Docker Compose v2.24+** (`docker compose version`). Older Compose clients cannot parse the optional -`env_file` syntax in `docker/docker-compose-dev.yaml`. +`env_file` syntax in `docker/docker-compose.yaml` and `docker/docker-compose-dev.yaml`. **Development** (hot-reload, source mounts): diff --git a/README_pt.md b/README_pt.md index d6f389f975..7d08a7f9fe 100644 --- a/README_pt.md +++ b/README_pt.md @@ -310,7 +310,7 @@ Use a tabela abaixo como ponto de partida prático para escolher como rodar o De #### Opção 1: Docker (recomendado) -Requer Docker Desktop / Docker Engine e **Docker Compose v2.24+** (`docker compose version`). Clientes Compose mais antigos não conseguem interpretar a sintaxe opcional de `env_file` em `docker/docker-compose-dev.yaml`. +Requer Docker Desktop / Docker Engine e **Docker Compose v2.24+** (`docker compose version`). Clientes Compose mais antigos não conseguem interpretar a sintaxe opcional de `env_file` em `docker/docker-compose.yaml` e `docker/docker-compose-dev.yaml`. **Desenvolvimento** (hot-reload, mounts do código-fonte): diff --git a/README_zh.md b/README_zh.md index bb5b746925..e2fa0050c4 100644 --- a/README_zh.md +++ b/README_zh.md @@ -255,7 +255,7 @@ DeerFlow 新近集成了 BytePlus 自研的智能搜索与抓取工具集——[ 需要 Docker Desktop / Docker Engine,以及 **Docker Compose v2.24+** (`docker compose version`)。更旧的 Compose 客户端无法解析 -`docker/docker-compose-dev.yaml` 里的可选 `env_file` 语法。 +`docker/docker-compose.yaml` 与 `docker/docker-compose-dev.yaml` 里的可选 `env_file` 语法。 **开发模式**(支持热更新,挂载源码): diff --git a/backend/tests/AGENTS.md b/backend/tests/AGENTS.md index 7fd7171666..9e02e2d36c 100644 --- a/backend/tests/AGENTS.md +++ b/backend/tests/AGENTS.md @@ -28,6 +28,14 @@ The local sandbox's UTF-8 subprocess guard inspects each text-mode call with `ast`, checking both `encoding` and `errors`; module-wide literal counts can hide unpinned calls behind unrelated settings. +## Real Compose tests + +`support/compose.py` probes `docker compose version --short` and requires Compose +2.24+ for optional `env_file` syntax. Skip missing, old or unreadable clients with +an actionable reason; cover version detection offline in `test_support_compose.py`. +Real rendering and production entry-point tests use only read-only Compose calls. +Never start or stop a stack from these tests. + ## Claude provider tests `test_claude_provider_prompt_caching.py` exercises real Anthropic SDK serialization diff --git a/backend/tests/support/compose.py b/backend/tests/support/compose.py new file mode 100644 index 0000000000..832975dea8 --- /dev/null +++ b/backend/tests/support/compose.py @@ -0,0 +1,41 @@ +"""Locate a Docker CLI whose Compose plugin can render the repo's compose files. + +Tests that hand a compose file to the real client prove what Compose itself +accepts, which a YAML-shape assertion cannot. They skip without a compatible client. +""" + +from __future__ import annotations + +import re +import shutil +import subprocess + +import pytest + +MIN_COMPOSE_VERSION = (2, 24) + + +def find_docker_compose() -> tuple[str | None, str]: + """Return a compatible ``docker`` executable and an empty or actionable skip reason.""" + docker = shutil.which("docker") + if docker is None: + return None, "no docker compose client installed" + try: + probe = subprocess.run([docker, "compose", "version", "--short"], capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=60, check=False) + except (OSError, subprocess.TimeoutExpired): + return None, "no docker compose client installed" + if probe.returncode != 0: + return None, "no docker compose client installed" + + version = probe.stdout.strip() + requirement = f"Compose {MIN_COMPOSE_VERSION[0]}.{MIN_COMPOSE_VERSION[1]}+ required" + match = re.fullmatch(r"v?(\d+)\.(\d+)(?:\.\d+)?(?:[-+][a-zA-Z0-9.-]+)?", version) + if match is None: + return None, f"{requirement}, could not determine version from {version!r}" + if tuple(int(part) for part in match.groups()) < MIN_COMPOSE_VERSION: + return None, f"{requirement}, found {version}" + return docker, "" + + +DOCKER, _skip_reason = find_docker_compose() +requires_docker_compose = pytest.mark.skipif(DOCKER is None, reason=_skip_reason) diff --git a/backend/tests/test_compose_default_bind_host.py b/backend/tests/test_compose_default_bind_host.py index ef054220e9..d17db87101 100644 --- a/backend/tests/test_compose_default_bind_host.py +++ b/backend/tests/test_compose_default_bind_host.py @@ -17,10 +17,15 @@ intentionally expose the stack behind their own TLS/auth front door. from __future__ import annotations +import json +import os +import shutil +import subprocess from pathlib import Path import pytest import yaml +from support.compose import DOCKER, requires_docker_compose REPO_ROOT = Path(__file__).resolve().parents[2] COMPOSE_PATHS = { @@ -120,14 +125,89 @@ def _bind_address(mapping: str) -> str | None: return segments[0] if len(segments) >= 3 else None -def test_dev_compose_env_files_are_optional(): - """Missing .env files must not fail `docker compose -f docker/docker-compose-dev.yaml`.""" - compose = yaml.safe_load(COMPOSE_PATHS["dev"].read_text(encoding="utf-8")) - expected = { - "provisioner": "../.env", - "frontend": "../frontend/.env", - "gateway": "../.env", - } - for service_name, path in expected.items(): +EXPECTED_ENV_FILES = { + "provisioner": "../.env", + "frontend": "../frontend/.env", + "gateway": "../.env", +} + + +@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS)) +def test_compose_env_files_are_optional(variant: str): + """Missing .env files must not fail either compose file. + + Both are gitignored, so a fresh checkout has neither. ``make up`` seeds + config.yaml but not these, and the short ``- ../.env`` form makes Compose + abort with ``env file ... not found`` before building anything. + """ + compose = yaml.safe_load(COMPOSE_PATHS[variant].read_text(encoding="utf-8")) + for service_name, path in EXPECTED_ENV_FILES.items(): entries = compose["services"][service_name]["env_file"] - assert entries == [{"path": path, "required": False}], f"{service_name} env_file must be optional; got: {entries!r}" + assert entries == [{"path": path, "required": False}], f"{variant} {service_name} env_file must be optional; got: {entries!r}" + + +# ── Against the real Compose client, when installed ───────────────────────── + + +def _render(tmp_path: Path, variant: str) -> subprocess.CompletedProcess[str]: + """Render a checkout copy of the compose file the way the wrappers invoke it.""" + docker_dir = tmp_path / "docker" + shutil.copytree(REPO_ROOT / "docker", docker_dir) + (tmp_path / "frontend").mkdir(exist_ok=True) + env = {key: value for key, value in os.environ.items() if not key.startswith(("DEER_FLOW_", "COMPOSE_"))} + # Values deploy.sh / docker.sh export before calling Compose; without them + # the bind-mount specs render empty and Compose fails for another reason. + env.update( + DEER_FLOW_ROOT=str(tmp_path), + DEER_FLOW_HOME=str(tmp_path / "home"), + DEER_FLOW_REPO_ROOT=str(tmp_path), + DEER_FLOW_CONFIG_PATH=str(tmp_path / "config.yaml"), + DEER_FLOW_EXTENSIONS_CONFIG_PATH=str(tmp_path / "extensions_config.json"), + BETTER_AUTH_SECRET="test-secret", + DEER_FLOW_INTERNAL_AUTH_TOKEN="test-token", + ) + return subprocess.run( + [DOCKER, "compose", "-p", "deer-flow-env-file-test", "-f", str(docker_dir / COMPOSE_PATHS[variant].name), "config", "--format", "json"], + env=env, + capture_output=True, + text=True, + timeout=120, + check=False, + ) + + +@requires_docker_compose +@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS)) +def test_real_compose_renders_a_checkout_without_env_files(tmp_path, variant: str): + result = _render(tmp_path, variant) + + assert result.returncode == 0, f"{variant} compose must render without .env files:\n{result.stderr}" + + +@requires_docker_compose +@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS)) +def test_real_compose_still_loads_env_files_that_exist(tmp_path, variant: str): + """Optional must not mean ignored: a present .env still reaches each service.""" + (tmp_path / "frontend").mkdir() + (tmp_path / ".env").write_text("ROOT_ENV_PROBE=from-root-env\n", encoding="utf-8") + (tmp_path / "frontend" / ".env").write_text("FRONTEND_ENV_PROBE=from-frontend-env\n", encoding="utf-8") + + result = _render(tmp_path, variant) + + assert result.returncode == 0, result.stderr + services = json.loads(result.stdout)["services"] + for service_name, path in EXPECTED_ENV_FILES.items(): + key, value = ("FRONTEND_ENV_PROBE", "from-frontend-env") if path.startswith("../frontend/") else ("ROOT_ENV_PROBE", "from-root-env") + assert services[service_name]["environment"].get(key) == value, f"{variant} {service_name} must still load {path}" + + +@requires_docker_compose +@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS)) +def test_real_compose_still_rejects_an_env_file_it_cannot_read(tmp_path, variant: str): + """`required: false` skips only a missing file; a broken one must stay loud.""" + (tmp_path / ".env").mkdir() + + result = _render(tmp_path, variant) + + assert result.returncode != 0 + assert ".env" in result.stderr diff --git a/backend/tests/test_gateway_startup.py b/backend/tests/test_gateway_startup.py index 63109e337d..2e7dfdc5db 100644 --- a/backend/tests/test_gateway_startup.py +++ b/backend/tests/test_gateway_startup.py @@ -8,6 +8,7 @@ import subprocess from pathlib import Path import pytest +from support.compose import DOCKER, requires_docker_compose from support.shell import find_script_bash REPO_ROOT = Path(__file__).resolve().parents[2] @@ -117,6 +118,58 @@ def test_deploy_failure_prints_gateway_diagnostics_and_never_claims_success(tmp_ assert "DeerFlow is running!" not in result.stdout assert "DeerFlow services failed to become ready" in result.stderr assert "supports `docker compose up --wait`" in result.stderr + assert "upgrade to Docker Compose 2.24+" in result.stderr calls = capture.read_text(encoding="utf-8") assert any(call.endswith(" ps") for call in calls.splitlines()) assert " logs --no-color --tail 100 gateway" in calls + + +# Hands only read-only calls to the real client: `up`, `down` and `logs` become +# `config --quiet` on the same global flags, so Compose parses exactly the +# project the entry point would act on without touching a running stack. +# Anything else fails loudly instead of reaching the real daemon. +_READ_ONLY_COMPOSE = """#!/usr/bin/env bash +args=() +for arg in "$@"; do + case "$arg" in + up|down|logs) exec "$REAL_DOCKER" "${args[@]}" config --quiet ;; + config|version) exec "$REAL_DOCKER" "$@" ;; + esac + args+=("$arg") +done +echo "unexpected docker call: $*" >&2 +exit 97 +""" + + +@requires_script_bash +@requires_docker_compose +@pytest.mark.parametrize( + "command", + [("deploy.sh",), ("deploy.sh", "start"), ("deploy.sh", "down"), ("docker.sh", "logs", "--prod")], + ids=["make-up", "deploy-start", "make-down", "make-prod-logs"], +) +def test_production_entry_points_parse_the_compose_project_without_env_files(tmp_path: Path, command: tuple[str, ...]) -> None: + """A fresh clone has no .env or frontend/.env (both gitignored). + + `make up`, `make down` and `make prod-logs` all load docker-compose.yaml, + so each must get past Compose's parse on such a checkout. + """ + worktree, env = _deploy_fixture(tmp_path, docker_script=_READ_ONLY_COMPOSE) + env["REAL_DOCKER"] = DOCKER + env["DEER_FLOW_HOME"] = str(tmp_path / "deer-flow-home") + assert not (worktree / ".env").exists() + assert not (worktree / "frontend" / ".env").exists() + + script, *args = command + result = subprocess.run( + [SCRIPT_BASH, str(worktree / "scripts" / script), *args], + cwd=worktree, + env=env, + check=False, + capture_output=True, + text=True, + ) + + assert result.returncode == 0, result.stderr + assert "env file" not in result.stderr diff --git a/backend/tests/test_support_compose.py b/backend/tests/test_support_compose.py new file mode 100644 index 0000000000..ccddfb6f82 --- /dev/null +++ b/backend/tests/test_support_compose.py @@ -0,0 +1,80 @@ +"""Offline coverage for the Compose client gate used by real rendering tests.""" + +from __future__ import annotations + +import subprocess + +import pytest +from support import compose + +DOCKER = "/fake/docker" + + +@pytest.fixture +def probe(monkeypatch): + monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER) + + def detect(stdout: str, *, returncode: int = 0): + def run(command, **kwargs): + return subprocess.CompletedProcess(command, returncode, stdout=stdout, stderr="") + + monkeypatch.setattr(compose.subprocess, "run", run) + return compose.find_docker_compose() + + return detect + + +@pytest.mark.parametrize("version", ["2.24.0", "v2.24.0", "2.24.0-desktop.1", "2.24.7+vendor.1", "2.25.0", "3.0.0", "5.1.4"]) +def test_supported_compose_client_is_available(version, probe): + assert probe(f"{version}\n") == (DOCKER, "") + + +@pytest.mark.parametrize("version", ["2.23.3", "v2.9.9", "1.29.2"]) +def test_old_compose_client_is_skipped_with_its_version(version, probe): + assert probe(f"{version}\n") == (None, f"Compose 2.24+ required, found {version}") + + +def test_probe_requests_the_short_version(monkeypatch): + commands = [] + monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER) + + def run(command, **kwargs): + commands.append(command) + return subprocess.CompletedProcess(command, 0, stdout="2.24.0\n", stderr="") + + monkeypatch.setattr(compose.subprocess, "run", run) + + assert compose.find_docker_compose() == (DOCKER, "") + assert commands == [[DOCKER, "compose", "version", "--short"]] + + +@pytest.mark.parametrize("output", ["", "garbage", "Docker Compose version v2.24.0", "2"]) +def test_unreadable_compose_version_is_skipped(output, probe): + docker, reason = probe(output) + + assert docker is None + assert "Compose 2.24+ required" in reason + assert "could not determine version" in reason + + +def test_missing_docker_is_skipped_without_probing(monkeypatch): + monkeypatch.setattr(compose.shutil, "which", lambda name: None) + monkeypatch.setattr(compose.subprocess, "run", lambda *args, **kwargs: pytest.fail("missing Docker must not be executed")) + + assert compose.find_docker_compose() == (None, "no docker compose client installed") + + +def test_missing_compose_plugin_is_skipped(probe): + assert probe("", returncode=1) == (None, "no docker compose client installed") + + +@pytest.mark.parametrize("error", [OSError("cannot execute docker"), subprocess.TimeoutExpired([DOCKER, "compose", "version", "--short"], 60)]) +def test_unavailable_compose_probe_is_skipped(monkeypatch, error): + monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER) + + def run(*args, **kwargs): + raise error + + monkeypatch.setattr(compose.subprocess, "run", run) + + assert compose.find_docker_compose() == (None, "no docker compose client installed") diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index cb10792330..eea51fc96b 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -1,6 +1,12 @@ # DeerFlow Production Environment # Usage: make up # +# Requires Docker Compose >= 2.24. ../.env and ../frontend/.env are gitignored, +# so a fresh checkout has neither; the env_file entries below use the long-form +# path/required syntax so Compose loads them when present instead of aborting +# with "env file ... not found". Older clients reject this file with: +# services..env_file.0 must be a string +# # Services: # - nginx: Reverse proxy (port 2026, configurable via PORT env var) # - frontend: Next.js production server @@ -91,7 +97,8 @@ services: - BETTER_AUTH_SECRET=${BETTER_AUTH_SECRET} - DEER_FLOW_INTERNAL_GATEWAY_BASE_URL=http://gateway:8001 env_file: - - ../frontend/.env + - path: ../frontend/.env + required: false networks: - deer-flow restart: unless-stopped @@ -165,7 +172,8 @@ services: - NO_PROXY=${NO_PROXY:-}${NO_PROXY:+,}localhost,127.0.0.1,::1,gateway,frontend,nginx,provisioner,openviking,host.docker.internal - no_proxy=${no_proxy:-}${no_proxy:+,}localhost,127.0.0.1,::1,gateway,frontend,nginx,provisioner,openviking,host.docker.internal env_file: - - ../.env + - path: ../.env + required: false extra_hosts: - "host.docker.internal:host-gateway" depends_on: @@ -212,7 +220,8 @@ services: - NODE_HOST=host.docker.internal - K8S_API_SERVER=https://host.docker.internal:26443 env_file: - - ../.env + - path: ../.env + required: false extra_hosts: - "host.docker.internal:host-gateway" networks: diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 87f64e52d9..18824baf4d 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -29,6 +29,15 @@ synchronized environment with `uv run --no-sync`. Production Compose probes Gateway `/health`, and `deploy.sh` waits for all services before reporting success; failures print Compose status and recent Gateway logs. +Both compose files mark `../.env` and `../frontend/.env` optional +(`path`/`required: false`, Compose 2.24+), so `make up`, `make down` and +`make prod-logs` on a fresh checkout neither abort nor create them; an +unreadable `.env` still fails. Do not seed them from the examples in +`deploy.sh` as `docker.sh start` does: `.env.example` holds placeholder API +keys the production Gateway would receive, and `make config` skips files that +exist. Pinned by `backend/tests/test_compose_default_bind_host.py` and +`backend/tests/test_gateway_startup.py`. + `deploy.sh` never sources the repo-root `.env`; Compose reads it via `--env-file`, and shell exports outrank that file during interpolation (an exported-but-empty variable still wins). So `BETTER_AUTH_SECRET` and diff --git a/scripts/deploy.sh b/scripts/deploy.sh index 44c2ece5a2..ebc67c5d73 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -442,6 +442,7 @@ report_startup_failure() { echo -e "${RED}✗ DeerFlow services failed to become ready.${NC}" >&2 echo ' If Docker Compose reports "unknown flag: --wait", upgrade to a version that' >&2 echo ' supports `docker compose up --wait`.' >&2 + echo ' If it reports "env_file.0 must be a string", upgrade to Docker Compose 2.24+.' >&2 echo " Container status:" >&2 "${COMPOSE_CMD[@]}" ps >&2 || true echo "" >&2 diff --git a/scripts/docker.sh b/scripts/docker.sh index c0352e1ce6..ff669b8c49 100755 --- a/scripts/docker.sh +++ b/scripts/docker.sh @@ -28,7 +28,7 @@ _refresh_compose_cmd() { } _refresh_compose_cmd -# docker-compose-dev.yaml marks its env_file entries optional with the long-form +# Both compose files mark their env_file entries optional with the long-form # `- path: ... / required: false` syntax, understood by Compose v2.24.0 and up. # Older clients abort while parsing the file, before any preflight below can run. COMPOSE_MIN_VERSION="2.24.0"