fix(deploy): let make up start on a checkout without .env files (#6370)

* fix(deploy): let make up start on a checkout without .env files

.env and frontend/.env are gitignored and make up does not create them,
but docker/docker-compose.yaml listed both as short-form env_file entries,
which Compose treats as required. On a fresh clone make up, make down and
make prod-logs all stopped with "env file .../.env not found" (exit 14)
before building or touching anything.

Mark the three entries optional with the long-form path/required syntax
the development compose file already uses. Compose loads them when they
exist and skips them otherwise; an unreadable or non-file .env still
fails. This needs Docker Compose 2.24+, the floor the README already
documents, and deploy.sh's startup-failure hint now names it for older
clients.

deploy.sh deliberately does not seed the files from the examples the way
docker.sh start does: .env.example carries placeholder API keys the
production Gateway would receive, and make config skips files that exist.

* docs(changelog): reference #6370

* test(deploy): gate Compose clients and cover make up

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

CHANGELOG.md

@@ -623,6 +623,12 @@ This release closes that milestone with **439 merged pull requests**.
 
 ### Fixed
 
+- **deploy:** `make up`, `make down` and `make prod-logs` no longer stop on a
+  fresh checkout with `env file .../.env not found`. `.env` and `frontend/.env` are gitignored and
+  `make up` does not create them, but the production compose file required
+  both. Its `env_file` entries are now optional, as in the development compose
+  file: Compose loads them when present and skips them otherwise. This needs
+  Docker Compose 2.24 or newer, the floor the README already documents. ([#6370])
 - **memory:** DeerMem relevance ranking no longer counts punctuation as query
   terms when jieba is installed (`memory-zh` extra). `tokenize()` dropped only
   whitespace from `jieba.cut`, which emits `，`, `。`, `,` and `!` as standalone
@@ -8869,4 +8875,5 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
 [#6351]: https://github.com/bytedance/deer-flow/pull/6351
 [#6354]: https://github.com/bytedance/deer-flow/pull/6354
 [#6365]: https://github.com/bytedance/deer-flow/pull/6365
+[#6370]: https://github.com/bytedance/deer-flow/pull/6370
 [#6388]: https://github.com/bytedance/deer-flow/pull/6388

CHANGELOG_zh.md

@@ -558,6 +558,11 @@
 
 ### 修复
 
+- **部署：** 在全新检出的仓库上，`make up`、`make down` 与 `make prod-logs` 
+  不再因 `env file .../.env not found` 而中止。`.env` 与
+  `frontend/.env` 已被 gitignore，`make up` 也不会创建它们，但生产 compose 文件此前要求两者都存在。
+  现在其 `env_file` 条目与开发 compose 文件一样为可选：文件存在时加载，不存在时跳过。此写法需要
+  Docker Compose 2.24 或更高版本，即 README 已注明的最低版本。([#6370])
 - **记忆：** 安装 jieba（`memory-zh` 扩展）时，DeerMem 相关度排序不再把标点计为查询词。
   `tokenize()` 只过滤了 `jieba.cut` 输出中的空白，而 jieba 会把 `，`、`。`、`,`、`!`
   切成独立 token，因此同样含逗号的查询与无关事实也会得到非零分，近似去重的相似度被抬高，
@@ -7324,4 +7329,5 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
 [#6350]: https://github.com/bytedance/deer-flow/pull/6350
 [#6351]: https://github.com/bytedance/deer-flow/pull/6351
 [#6354]: https://github.com/bytedance/deer-flow/pull/6354
+[#6370]: https://github.com/bytedance/deer-flow/pull/6370
 [#6388]: https://github.com/bytedance/deer-flow/pull/6388

CONTRIBUTING.md

@@ -13,8 +13,8 @@ Docker provides a consistent, isolated environment with all dependencies pre-con
 #### Prerequisites
 
 - Docker Desktop or Docker Engine
-- Docker Compose **v2.24 or newer** (check with `docker compose version`). The dev
-  Compose file marks its `env_file` entries optional using the long-form
+- Docker Compose **v2.24 or newer** (check with `docker compose version`). Both
+  Compose files mark their `env_file` entries optional using the long-form
   `path`/`required` syntax; older clients reject it with
   `services.gateway.env_file.0 must be a string`. `make docker-start` verifies the
   version and tells you to upgrade — direct `docker compose` callers get that raw

README.md

@@ -393,7 +393,7 @@ Use the table below as a practical starting point when choosing how to run DeerF
 
 Requires Docker Desktop / Docker Engine and **Docker Compose v2.24+**
 (`docker compose version`). Older Compose clients cannot parse the optional
-`env_file` syntax in `docker/docker-compose-dev.yaml`.
+`env_file` syntax in `docker/docker-compose.yaml` and `docker/docker-compose-dev.yaml`.
 
 **Development** (hot-reload, source mounts):
 

README_pt.md

@@ -310,7 +310,7 @@ Use a tabela abaixo como ponto de partida prático para escolher como rodar o De
 
 #### Opção 1: Docker (recomendado)
 
-Requer Docker Desktop / Docker Engine e **Docker Compose v2.24+** (`docker compose version`). Clientes Compose mais antigos não conseguem interpretar a sintaxe opcional de `env_file` em `docker/docker-compose-dev.yaml`.
+Requer Docker Desktop / Docker Engine e **Docker Compose v2.24+** (`docker compose version`). Clientes Compose mais antigos não conseguem interpretar a sintaxe opcional de `env_file` em `docker/docker-compose.yaml` e `docker/docker-compose-dev.yaml`.
 
 **Desenvolvimento** (hot-reload, mounts do código-fonte):
 

README_zh.md

@@ -255,7 +255,7 @@ DeerFlow 新近集成了 BytePlus 自研的智能搜索与抓取工具集——[
 
 需要 Docker Desktop / Docker Engine，以及 **Docker Compose v2.24+**
 （`docker compose version`）。更旧的 Compose 客户端无法解析
-`docker/docker-compose-dev.yaml` 里的可选 `env_file` 语法。
+`docker/docker-compose.yaml` 与 `docker/docker-compose-dev.yaml` 里的可选 `env_file` 语法。
 
 **开发模式**（支持热更新，挂载源码）：
 

backend/tests/AGENTS.md

@@ -28,6 +28,14 @@ The local sandbox's UTF-8 subprocess guard inspects each text-mode call with
 `ast`, checking both `encoding` and `errors`; module-wide literal counts can
 hide unpinned calls behind unrelated settings.
 
+## Real Compose tests
+
+`support/compose.py` probes `docker compose version --short` and requires Compose
+2.24+ for optional `env_file` syntax. Skip missing, old or unreadable clients with
+an actionable reason; cover version detection offline in `test_support_compose.py`.
+Real rendering and production entry-point tests use only read-only Compose calls.
+Never start or stop a stack from these tests.
+
 ## Claude provider tests
 
 `test_claude_provider_prompt_caching.py` exercises real Anthropic SDK serialization

backend/tests/support/compose.py

@@ -0,0 +1,41 @@
+"""Locate a Docker CLI whose Compose plugin can render the repo's compose files.
+
+Tests that hand a compose file to the real client prove what Compose itself
+accepts, which a YAML-shape assertion cannot. They skip without a compatible client.
+"""
+
+from __future__ import annotations
+
+import re
+import shutil
+import subprocess
+
+import pytest
+
+MIN_COMPOSE_VERSION = (2, 24)
+
+
+def find_docker_compose() -> tuple[str | None, str]:
+    """Return a compatible ``docker`` executable and an empty or actionable skip reason."""
+    docker = shutil.which("docker")
+    if docker is None:
+        return None, "no docker compose client installed"
+    try:
+        probe = subprocess.run([docker, "compose", "version", "--short"], capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=60, check=False)
+    except (OSError, subprocess.TimeoutExpired):
+        return None, "no docker compose client installed"
+    if probe.returncode != 0:
+        return None, "no docker compose client installed"
+
+    version = probe.stdout.strip()
+    requirement = f"Compose {MIN_COMPOSE_VERSION[0]}.{MIN_COMPOSE_VERSION[1]}+ required"
+    match = re.fullmatch(r"v?(\d+)\.(\d+)(?:\.\d+)?(?:[-+][a-zA-Z0-9.-]+)?", version)
+    if match is None:
+        return None, f"{requirement}, could not determine version from {version!r}"
+    if tuple(int(part) for part in match.groups()) < MIN_COMPOSE_VERSION:
+        return None, f"{requirement}, found {version}"
+    return docker, ""
+
+
+DOCKER, _skip_reason = find_docker_compose()
+requires_docker_compose = pytest.mark.skipif(DOCKER is None, reason=_skip_reason)

backend/tests/test_compose_default_bind_host.py

@@ -17,10 +17,15 @@ intentionally expose the stack behind their own TLS/auth front door.
 
 from __future__ import annotations
 
+import json
+import os
+import shutil
+import subprocess
 from pathlib import Path
 
 import pytest
 import yaml
+from support.compose import DOCKER, requires_docker_compose
 
 REPO_ROOT = Path(__file__).resolve().parents[2]
 COMPOSE_PATHS = {
@@ -120,14 +125,89 @@ def _bind_address(mapping: str) -> str | None:
     return segments[0] if len(segments) >= 3 else None
 
 
-def test_dev_compose_env_files_are_optional():
-    """Missing .env files must not fail `docker compose -f docker/docker-compose-dev.yaml`."""
-    compose = yaml.safe_load(COMPOSE_PATHS["dev"].read_text(encoding="utf-8"))
-    expected = {
-        "provisioner": "../.env",
-        "frontend": "../frontend/.env",
-        "gateway": "../.env",
-    }
-    for service_name, path in expected.items():
+EXPECTED_ENV_FILES = {
+    "provisioner": "../.env",
+    "frontend": "../frontend/.env",
+    "gateway": "../.env",
+}
+
+
+@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS))
+def test_compose_env_files_are_optional(variant: str):
+    """Missing .env files must not fail either compose file.
+
+    Both are gitignored, so a fresh checkout has neither. ``make up`` seeds
+    config.yaml but not these, and the short ``- ../.env`` form makes Compose
+    abort with ``env file ... not found`` before building anything.
+    """
+    compose = yaml.safe_load(COMPOSE_PATHS[variant].read_text(encoding="utf-8"))
+    for service_name, path in EXPECTED_ENV_FILES.items():
         entries = compose["services"][service_name]["env_file"]
-        assert entries == [{"path": path, "required": False}], f"{service_name} env_file must be optional; got: {entries!r}"
+        assert entries == [{"path": path, "required": False}], f"{variant} {service_name} env_file must be optional; got: {entries!r}"
+
+
+# ── Against the real Compose client, when installed ─────────────────────────
+
+
+def _render(tmp_path: Path, variant: str) -> subprocess.CompletedProcess[str]:
+    """Render a checkout copy of the compose file the way the wrappers invoke it."""
+    docker_dir = tmp_path / "docker"
+    shutil.copytree(REPO_ROOT / "docker", docker_dir)
+    (tmp_path / "frontend").mkdir(exist_ok=True)
+    env = {key: value for key, value in os.environ.items() if not key.startswith(("DEER_FLOW_", "COMPOSE_"))}
+    # Values deploy.sh / docker.sh export before calling Compose; without them
+    # the bind-mount specs render empty and Compose fails for another reason.
+    env.update(
+        DEER_FLOW_ROOT=str(tmp_path),
+        DEER_FLOW_HOME=str(tmp_path / "home"),
+        DEER_FLOW_REPO_ROOT=str(tmp_path),
+        DEER_FLOW_CONFIG_PATH=str(tmp_path / "config.yaml"),
+        DEER_FLOW_EXTENSIONS_CONFIG_PATH=str(tmp_path / "extensions_config.json"),
+        BETTER_AUTH_SECRET="test-secret",
+        DEER_FLOW_INTERNAL_AUTH_TOKEN="test-token",
+    )
+    return subprocess.run(
+        [DOCKER, "compose", "-p", "deer-flow-env-file-test", "-f", str(docker_dir / COMPOSE_PATHS[variant].name), "config", "--format", "json"],
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=120,
+        check=False,
+    )
+
+
+@requires_docker_compose
+@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS))
+def test_real_compose_renders_a_checkout_without_env_files(tmp_path, variant: str):
+    result = _render(tmp_path, variant)
+
+    assert result.returncode == 0, f"{variant} compose must render without .env files:\n{result.stderr}"
+
+
+@requires_docker_compose
+@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS))
+def test_real_compose_still_loads_env_files_that_exist(tmp_path, variant: str):
+    """Optional must not mean ignored: a present .env still reaches each service."""
+    (tmp_path / "frontend").mkdir()
+    (tmp_path / ".env").write_text("ROOT_ENV_PROBE=from-root-env\n", encoding="utf-8")
+    (tmp_path / "frontend" / ".env").write_text("FRONTEND_ENV_PROBE=from-frontend-env\n", encoding="utf-8")
+
+    result = _render(tmp_path, variant)
+
+    assert result.returncode == 0, result.stderr
+    services = json.loads(result.stdout)["services"]
+    for service_name, path in EXPECTED_ENV_FILES.items():
+        key, value = ("FRONTEND_ENV_PROBE", "from-frontend-env") if path.startswith("../frontend/") else ("ROOT_ENV_PROBE", "from-root-env")
+        assert services[service_name]["environment"].get(key) == value, f"{variant} {service_name} must still load {path}"
+
+
+@requires_docker_compose
+@pytest.mark.parametrize("variant", sorted(COMPOSE_PATHS))
+def test_real_compose_still_rejects_an_env_file_it_cannot_read(tmp_path, variant: str):
+    """`required: false` skips only a missing file; a broken one must stay loud."""
+    (tmp_path / ".env").mkdir()
+
+    result = _render(tmp_path, variant)
+
+    assert result.returncode != 0
+    assert ".env" in result.stderr

backend/tests/test_gateway_startup.py

@@ -8,6 +8,7 @@ import subprocess
 from pathlib import Path
 
 import pytest
+from support.compose import DOCKER, requires_docker_compose
 from support.shell import find_script_bash
 
 REPO_ROOT = Path(__file__).resolve().parents[2]
@@ -117,6 +118,58 @@ def test_deploy_failure_prints_gateway_diagnostics_and_never_claims_success(tmp_
     assert "DeerFlow is running!" not in result.stdout
     assert "DeerFlow services failed to become ready" in result.stderr
     assert "supports `docker compose up --wait`" in result.stderr
+    assert "upgrade to Docker Compose 2.24+" in result.stderr
     calls = capture.read_text(encoding="utf-8")
     assert any(call.endswith(" ps") for call in calls.splitlines())
     assert " logs --no-color --tail 100 gateway" in calls
+
+
+# Hands only read-only calls to the real client: `up`, `down` and `logs` become
+# `config --quiet` on the same global flags, so Compose parses exactly the
+# project the entry point would act on without touching a running stack.
+# Anything else fails loudly instead of reaching the real daemon.
+_READ_ONLY_COMPOSE = """#!/usr/bin/env bash
+args=()
+for arg in "$@"; do
+  case "$arg" in
+    up|down|logs) exec "$REAL_DOCKER" "${args[@]}" config --quiet ;;
+    config|version) exec "$REAL_DOCKER" "$@" ;;
+  esac
+  args+=("$arg")
+done
+echo "unexpected docker call: $*" >&2
+exit 97
+"""
+
+
+@requires_script_bash
+@requires_docker_compose
+@pytest.mark.parametrize(
+    "command",
+    [("deploy.sh",), ("deploy.sh", "start"), ("deploy.sh", "down"), ("docker.sh", "logs", "--prod")],
+    ids=["make-up", "deploy-start", "make-down", "make-prod-logs"],
+)
+def test_production_entry_points_parse_the_compose_project_without_env_files(tmp_path: Path, command: tuple[str, ...]) -> None:
+    """A fresh clone has no .env or frontend/.env (both gitignored).
+
+    `make up`, `make down` and `make prod-logs` all load docker-compose.yaml,
+    so each must get past Compose's parse on such a checkout.
+    """
+    worktree, env = _deploy_fixture(tmp_path, docker_script=_READ_ONLY_COMPOSE)
+    env["REAL_DOCKER"] = DOCKER
+    env["DEER_FLOW_HOME"] = str(tmp_path / "deer-flow-home")
+    assert not (worktree / ".env").exists()
+    assert not (worktree / "frontend" / ".env").exists()
+
+    script, *args = command
+    result = subprocess.run(
+        [SCRIPT_BASH, str(worktree / "scripts" / script), *args],
+        cwd=worktree,
+        env=env,
+        check=False,
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "env file" not in result.stderr

backend/tests/test_support_compose.py

@@ -0,0 +1,80 @@
+"""Offline coverage for the Compose client gate used by real rendering tests."""
+
+from __future__ import annotations
+
+import subprocess
+
+import pytest
+from support import compose
+
+DOCKER = "/fake/docker"
+
+
+@pytest.fixture
+def probe(monkeypatch):
+    monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER)
+
+    def detect(stdout: str, *, returncode: int = 0):
+        def run(command, **kwargs):
+            return subprocess.CompletedProcess(command, returncode, stdout=stdout, stderr="")
+
+        monkeypatch.setattr(compose.subprocess, "run", run)
+        return compose.find_docker_compose()
+
+    return detect
+
+
+@pytest.mark.parametrize("version", ["2.24.0", "v2.24.0", "2.24.0-desktop.1", "2.24.7+vendor.1", "2.25.0", "3.0.0", "5.1.4"])
+def test_supported_compose_client_is_available(version, probe):
+    assert probe(f"{version}\n") == (DOCKER, "")
+
+
+@pytest.mark.parametrize("version", ["2.23.3", "v2.9.9", "1.29.2"])
+def test_old_compose_client_is_skipped_with_its_version(version, probe):
+    assert probe(f"{version}\n") == (None, f"Compose 2.24+ required, found {version}")
+
+
+def test_probe_requests_the_short_version(monkeypatch):
+    commands = []
+    monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER)
+
+    def run(command, **kwargs):
+        commands.append(command)
+        return subprocess.CompletedProcess(command, 0, stdout="2.24.0\n", stderr="")
+
+    monkeypatch.setattr(compose.subprocess, "run", run)
+
+    assert compose.find_docker_compose() == (DOCKER, "")
+    assert commands == [[DOCKER, "compose", "version", "--short"]]
+
+
+@pytest.mark.parametrize("output", ["", "garbage", "Docker Compose version v2.24.0", "2"])
+def test_unreadable_compose_version_is_skipped(output, probe):
+    docker, reason = probe(output)
+
+    assert docker is None
+    assert "Compose 2.24+ required" in reason
+    assert "could not determine version" in reason
+
+
+def test_missing_docker_is_skipped_without_probing(monkeypatch):
+    monkeypatch.setattr(compose.shutil, "which", lambda name: None)
+    monkeypatch.setattr(compose.subprocess, "run", lambda *args, **kwargs: pytest.fail("missing Docker must not be executed"))
+
+    assert compose.find_docker_compose() == (None, "no docker compose client installed")
+
+
+def test_missing_compose_plugin_is_skipped(probe):
+    assert probe("", returncode=1) == (None, "no docker compose client installed")
+
+
+@pytest.mark.parametrize("error", [OSError("cannot execute docker"), subprocess.TimeoutExpired([DOCKER, "compose", "version", "--short"], 60)])
+def test_unavailable_compose_probe_is_skipped(monkeypatch, error):
+    monkeypatch.setattr(compose.shutil, "which", lambda name: DOCKER)
+
+    def run(*args, **kwargs):
+        raise error
+
+    monkeypatch.setattr(compose.subprocess, "run", run)
+
+    assert compose.find_docker_compose() == (None, "no docker compose client installed")

docker/docker-compose.yaml

@@ -1,6 +1,12 @@
 # DeerFlow Production Environment
 # Usage: make up
 #
+# Requires Docker Compose >= 2.24. ../.env and ../frontend/.env are gitignored,
+# so a fresh checkout has neither; the env_file entries below use the long-form
+# path/required syntax so Compose loads them when present instead of aborting
+# with "env file ... not found". Older clients reject this file with:
+#   services.<name>.env_file.0 must be a string
+#
 # Services:
 #   - nginx:       Reverse proxy (port 2026, configurable via PORT env var)
 #   - frontend:    Next.js production server
@@ -91,7 +97,8 @@ services:
       - BETTER_AUTH_SECRET=${BETTER_AUTH_SECRET}
       - DEER_FLOW_INTERNAL_GATEWAY_BASE_URL=http://gateway:8001
     env_file:
-      - ../frontend/.env
+      - path: ../frontend/.env
+        required: false
     networks:
       - deer-flow
     restart: unless-stopped
@@ -165,7 +172,8 @@ services:
       - NO_PROXY=${NO_PROXY:-}${NO_PROXY:+,}localhost,127.0.0.1,::1,gateway,frontend,nginx,provisioner,openviking,host.docker.internal
       - no_proxy=${no_proxy:-}${no_proxy:+,}localhost,127.0.0.1,::1,gateway,frontend,nginx,provisioner,openviking,host.docker.internal
     env_file:
-      - ../.env
+      - path: ../.env
+        required: false
     extra_hosts:
       - "host.docker.internal:host-gateway"
     depends_on:
@@ -212,7 +220,8 @@ services:
       - NODE_HOST=host.docker.internal
       - K8S_API_SERVER=https://host.docker.internal:26443
     env_file:
-      - ../.env
+      - path: ../.env
+        required: false
     extra_hosts:
       - "host.docker.internal:host-gateway"
     networks:

scripts/AGENTS.md

@@ -29,6 +29,15 @@ synchronized environment with `uv run --no-sync`. Production Compose probes
 Gateway `/health`, and `deploy.sh` waits for all services before reporting
 success; failures print Compose status and recent Gateway logs.
 
+Both compose files mark `../.env` and `../frontend/.env` optional
+(`path`/`required: false`, Compose 2.24+), so `make up`, `make down` and
+`make prod-logs` on a fresh checkout neither abort nor create them; an
+unreadable `.env` still fails. Do not seed them from the examples in
+`deploy.sh` as `docker.sh start` does: `.env.example` holds placeholder API
+keys the production Gateway would receive, and `make config` skips files that
+exist. Pinned by `backend/tests/test_compose_default_bind_host.py` and
+`backend/tests/test_gateway_startup.py`.
+
 `deploy.sh` never sources the repo-root `.env`; Compose reads it via
 `--env-file`, and shell exports outrank that file during interpolation (an
 exported-but-empty variable still wins). So `BETTER_AUTH_SECRET` and

scripts/deploy.sh

@@ -442,6 +442,7 @@ report_startup_failure() {
     echo -e "${RED}✗ DeerFlow services failed to become ready.${NC}" >&2
     echo '  If Docker Compose reports "unknown flag: --wait", upgrade to a version that' >&2
     echo '  supports `docker compose up --wait`.' >&2
+    echo '  If it reports "env_file.0 must be a string", upgrade to Docker Compose 2.24+.' >&2
     echo "  Container status:" >&2
     "${COMPOSE_CMD[@]}" ps >&2 || true
     echo "" >&2

scripts/docker.sh

@@ -28,7 +28,7 @@ _refresh_compose_cmd() {
 }
 _refresh_compose_cmd
 
-# docker-compose-dev.yaml marks its env_file entries optional with the long-form
+# Both compose files mark their env_file entries optional with the long-form
 # `- path: ... / required: false` syntax, understood by Compose v2.24.0 and up.
 # Older clients abort while parsing the file, before any preflight below can run.
 COMPOSE_MIN_VERSION="2.24.0"