From 2f833669bd9458c60da5ca14d6eb6a0439348b0e Mon Sep 17 00:00:00 2001 From: Hyeonsang Cho Date: Tue, 6 Oct 2026 16:32:46 +0900 Subject: [PATCH] fix(make): name what make clean deletes and refuse under a live gateway (#6351) * fix(make): name what make clean deletes and refuse under a live gateway `make help` described `make clean` as cleaning up "temporary files", but it deletes backend/.deer-flow: the local database, users, threads, uploads, memory, and secrets. Both Docker stacks mount that directory into the deer-flow-gateway container, and `make stop` only stops local services, so the data could be deleted under a running Gateway. Say what is deleted in the help text and before deleting, and refuse while the deer-flow-gateway container is running. The check runs before `make stop`, which would otherwise stop the running stack's sandboxes first; a missing or unreachable Docker passes. * docs(changelog): reference #6351 for the make clean guard * fix(make): share make clean's data list and pin stop before the delete The help line listed "database, threads, uploads, secrets" while the deletion notice also named users and memory. Define the list once in RUNTIME_DATA_CONTENTS and use it in both. Running sandbox containers left behind by `make down` are already stopped by `make stop` (serve.sh --stop reaps deer-flow-sandbox*), which runs after the guard and before the delete. Note that in the guard script, and pin the ordering: the stub stop now records that the data still exists. --------- Co-authored-by: Willem Jiang --- .../templates/report.docker.template.md | 2 +- CHANGELOG.md | 10 ++ CHANGELOG_zh.md | 6 + Makefile | 13 +- backend/docs/APPLE_CONTAINER.md | 2 +- backend/tests/test_make_clean_guard.py | 132 ++++++++++++++++++ scripts/AGENTS.md | 8 ++ scripts/check-data-not-in-use.sh | 38 +++++ 8 files changed, 205 insertions(+), 6 deletions(-) create mode 100644 backend/tests/test_make_clean_guard.py create mode 100755 scripts/check-data-not-in-use.sh diff --git a/.agent/skills/smoke-test/templates/report.docker.template.md b/.agent/skills/smoke-test/templates/report.docker.template.md index 7313ed105c..5bebe1ea7c 100644 --- a/.agent/skills/smoke-test/templates/report.docker.template.md +++ b/.agent/skills/smoke-test/templates/report.docker.template.md @@ -162,7 +162,7 @@ Commit Message: {{git_commit_message}} 1. [ ] Review references/troubleshooting.md for common solutions 2. [ ] Check Docker logs: `make docker-logs` 3. [ ] Verify configuration file format and content -4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start` +4. [ ] If needed, fully reset the environment: `make docker-stop && make clean && make config && make docker-init && make docker-start` --- diff --git a/CHANGELOG.md b/CHANGELOG.md index cb3f66e871..4917003b1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -623,6 +623,15 @@ This release closes that milestone with **439 merged pull requests**. ### Fixed +- **make:** `make clean` now says what it deletes and refuses to run under a live + Docker Gateway. `make help` described it as cleaning up "temporary files", but + it deletes `backend/.deer-flow`: the local database, users, threads, uploads, + memory, and secrets. Both Docker stacks mount that directory into the + `deer-flow-gateway` container, which `make stop` leaves running, so the data + could be deleted under a live Gateway. The help text and a notice before + deletion now name the directory, and `make clean` stops with a hint to run + `make down` or `make docker-stop` while that container is running, before it + stops anything. ([#6351]) - **tui:** A new prompt no longer races an interrupted run in the same conversation. After `Ctrl+C` the run's worker keeps going until its current step returns, so a tool call already in progress (a long shell command, for @@ -8818,3 +8827,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#6344]: https://github.com/bytedance/deer-flow/pull/6344 [#6347]: https://github.com/bytedance/deer-flow/pull/6347 [#6350]: https://github.com/bytedance/deer-flow/pull/6350 +[#6351]: https://github.com/bytedance/deer-flow/pull/6351 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index 467f2d8c8e..4053062c45 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -558,6 +558,11 @@ ### 修复 +- **make：** `make clean` 现在会说明它删除的内容，并拒绝在运行中的 Docker Gateway 下执行。`make help` + 此前称其清理"临时文件"，但它实际删除 `backend/.deer-flow`：本地数据库、用户、线程、上传、记忆和密钥。 + 两套 Docker 栈都把该目录挂载进 `deer-flow-gateway` 容器，而 `make stop` 不会停止它，因此数据可能在 + Gateway 运行时被删除。现在帮助文本和删除前的提示会写明该目录；当该容器仍在运行时，`make clean` + 会在停止任何服务之前退出，并提示先执行 `make down` 或 `make docker-stop`。([#6351]) - **TUI：** 新的提示不再与同一会话中被中断的运行发生竞争。按下 `Ctrl+C` 后，该运行的工作线程会 一直执行到当前步骤返回，因此正在进行的工具调用（例如耗时较长的 shell 命令）仍会完成并写入检查点， 而下一条提示已在同一线程上运行；最后写入检查点的运行会成为会话历史，新的一轮可能因此悄然消失。 @@ -7303,3 +7308,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子 [#6343]: https://github.com/bytedance/deer-flow/pull/6343 [#6344]: https://github.com/bytedance/deer-flow/pull/6344 [#6350]: https://github.com/bytedance/deer-flow/pull/6350 +[#6351]: https://github.com/bytedance/deer-flow/pull/6351 diff --git a/Makefile b/Makefile index 72ee584e78..9ccaea34eb 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,8 @@ else endif FRONTEND_PNPM = $(PYTHON) ../scripts/pnpm.py +# What `make clean` deletes; shared by its help line and deletion notice. +RUNTIME_DATA_CONTENTS = database, users, threads, uploads, memory, secrets help: @echo "DeerFlow Development Commands:" @@ -46,7 +48,7 @@ help: @echo " make start-daemon - Start prod services in background (daemon mode)" @echo " make nginx - Start nginx alone in the foreground (local dev config)" @echo " make stop - Stop all running services" - @echo " make clean - Clean up processes and temporary files" + @echo " make clean - Stop local services and DELETE local runtime data (backend/.deer-flow: $(RUNTIME_DATA_CONTENTS)) and logs" @echo "" @echo "Docker Production Commands:" @echo " make up - Build and start production Docker services (localhost:2026)" @@ -172,9 +174,12 @@ nginx: stop: @$(RUN_SHELL_SCRIPT) ./scripts/serve.sh --stop -# Clean up -clean: stop - @echo "Cleaning up..." +# Clean up: deletes local runtime data, not just temporary files. The guard +# runs before stop, which would otherwise stop a running stack's sandboxes. +clean: + @$(RUN_SHELL_SCRIPT) ./scripts/check-data-not-in-use.sh + @$(MAKE) --no-print-directory stop + @echo "Deleting local runtime data in backend/.deer-flow ($(RUNTIME_DATA_CONTENTS)) and logs/*.log..." @-rm -rf backend/.deer-flow 2>/dev/null || true @-rm -rf logs/*.log 2>/dev/null || true @echo "✓ Cleanup complete" diff --git a/backend/docs/APPLE_CONTAINER.md b/backend/docs/APPLE_CONTAINER.md index a58bee19f8..e25c9ed66d 100644 --- a/backend/docs/APPLE_CONTAINER.md +++ b/backend/docs/APPLE_CONTAINER.md @@ -170,7 +170,7 @@ The project includes a unified cleanup script that handles both runtimes: All cleanup commands in `Makefile` automatically handle both runtimes: ```bash make stop # Stops all services and cleans up containers -make clean # Full cleanup including logs +make clean # Also deletes local runtime data (backend/.deer-flow) and logs ``` ## Testing diff --git a/backend/tests/test_make_clean_guard.py b/backend/tests/test_make_clean_guard.py new file mode 100644 index 0000000000..79ef756b83 --- /dev/null +++ b/backend/tests/test_make_clean_guard.py @@ -0,0 +1,132 @@ +"""`make clean` says what it deletes and refuses while the Gateway container runs. + +The target deletes ``backend/.deer-flow`` -- the database, users, threads, +uploads, and secrets -- but ``make help`` described it as cleaning up +"temporary files". Both Docker stacks mount that directory into the +``deer-flow-gateway`` container, and ``make stop`` only stops local services, +so ``make clean`` used to delete the data under a running Gateway. + +These tests run the real ``make clean`` from the real Makefile and guard script +in a scratch checkout. ``scripts/serve.sh`` is a stub that records the stop +call, so nothing touches the host's processes or containers, and ``docker`` is +a fake on a PATH limited to the tools the recipe needs. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + +pytestmark = pytest.mark.skipif( + os.name == "nt" or shutil.which("make") is None or shutil.which("bash") is None, + reason="runs the POSIX make recipe", +) + +_TOOLS = ("make", "bash", "sh", "rm", "grep", "tr") + + +def _checkout(tmp_path: Path) -> Path: + root = tmp_path / "checkout" + (root / "scripts").mkdir(parents=True) + shutil.copy2(REPO_ROOT / "Makefile", root / "Makefile") + shutil.copy2(REPO_ROOT / "scripts" / "check-data-not-in-use.sh", root / "scripts" / "check-data-not-in-use.sh") + # Records each stop call and whether the data still existed at that point: + # the real serve.sh --stop also stops deer-flow-sandbox* containers, which + # bind-mount thread directories under backend/.deer-flow. + (root / "scripts" / "serve.sh").write_text( + '#!/usr/bin/env bash\nif [ -e backend/.deer-flow ]; then state=data-present; else state=data-gone; fi\necho "$* $state" >> stop-calls\n', + encoding="utf-8", + ) + data = root / "backend" / ".deer-flow" / "data" + data.mkdir(parents=True) + (data / "deerflow.db").write_text("db", encoding="utf-8") + (root / "logs").mkdir() + (root / "logs" / "gateway.log").write_text("log", encoding="utf-8") + return root + + +def _bin(tmp_path: Path, docker: str | None) -> Path: + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + for tool in _TOOLS: + found = shutil.which(tool) + assert found, tool + (bin_dir / tool).symlink_to(found) + if docker is not None: + fake = bin_dir / "docker" + fake.write_text(f"#!/bin/sh\n{docker}\n", encoding="utf-8") + fake.chmod(0o755) + return bin_dir + + +def _make(root: Path, bin_dir: Path, target: str = "clean") -> subprocess.CompletedProcess[str]: + env = {"PATH": str(bin_dir), "HOME": str(root)} + return subprocess.run(["make", "-s", target], cwd=root, env=env, capture_output=True, text=True, timeout=60) + + +def test_help_says_clean_deletes_runtime_data(tmp_path: Path) -> None: + root = _checkout(tmp_path) + + help_line = next(line for line in _make(root, _bin(tmp_path, None), "help").stdout.splitlines() if "make clean" in line) + + assert "temporary files" not in help_line + assert "DELETE" in help_line and "backend/.deer-flow" in help_line + + +@pytest.mark.parametrize( + "names", + ["deer-flow-gateway", "deer-flow-redis\ndeer-flow-gateway", "deer-flow-gateway\r"], + ids=["gateway", "whole-stack", "crlf"], +) +def test_clean_refuses_while_the_gateway_container_runs(tmp_path: Path, names: str) -> None: + root = _checkout(tmp_path) + listing = names.replace("\n", "\\n").replace("\r", "\\r") + + result = _make(root, _bin(tmp_path, f"printf '{listing}\\n'")) + + assert result.returncode != 0 + assert "deer-flow-gateway container is running" in result.stderr + assert (root / "backend" / ".deer-flow" / "data" / "deerflow.db").exists() + assert (root / "logs" / "gateway.log").exists() + # The guard runs before stop, which would stop the stack's sandboxes. + assert not (root / "stop-calls").exists() + + +@pytest.mark.parametrize( + "docker", + [None, "exit 1", "printf 'deer-flow-gateway-old\\nmy-deer-flow-gateway\\ndeer-flow-sandbox-1\\n'"], + ids=["no-docker", "daemon-down", "other-containers"], +) +def test_clean_deletes_runtime_data_when_no_gateway_container_runs(tmp_path: Path, docker: str | None) -> None: + root = _checkout(tmp_path) + + result = _make(root, _bin(tmp_path, docker)) + + assert result.returncode == 0, result.stderr + assert "Deleting local runtime data in backend/.deer-flow" in result.stdout + assert not (root / "backend" / ".deer-flow").exists() + assert not (root / "logs" / "gateway.log").exists() + # stop (and with it the sandbox-container cleanup) runs before the deletion. + assert (root / "stop-calls").read_text(encoding="utf-8").split() == ["--stop", "data-present"] + + +def _listed_contents(line: str, prefix: str) -> list[str]: + return line.split(prefix, 1)[1].split(")", 1)[0].split(", ") + + +def test_help_and_deletion_notice_list_the_same_data(tmp_path: Path) -> None: + root = _checkout(tmp_path) + bin_dir = _bin(tmp_path, None) + + help_line = next(line for line in _make(root, bin_dir, "help").stdout.splitlines() if "make clean" in line) + notice = next(line for line in _make(root, bin_dir).stdout.splitlines() if line.startswith("Deleting local runtime data")) + + listed = _listed_contents(help_line, "backend/.deer-flow: ") + assert listed == _listed_contents(notice, "backend/.deer-flow (") + assert {"database", "users", "threads", "uploads", "memory", "secrets"} <= set(listed) diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 57446b26b9..87f64e52d9 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -90,6 +90,14 @@ likewise prefix the target with `bash`. This keeps documented `make` commands working when a source archive, `core.fileMode=false`, or a non-POSIX filesystem does not preserve executable bits. +`make clean` deletes `backend/.deer-flow` (database, users, threads, uploads, +secrets), which both compose stacks mount into `deer-flow-gateway`. Its recipe +runs `check-data-not-in-use.sh` before `make stop` (which would stop a live +stack's sandboxes) and refuses while that container runs; an absent or +unreachable Docker passes. `make stop` must still run before the delete: it +stops `deer-flow-sandbox*` containers, whose thread mounts live in that tree. +`RUNTIME_DATA_CONTENTS` feeds both the help line and the deletion notice. + Host-side pnpm calls must go through `scripts/pnpm.py`. With native Windows Python (`os.name == "nt"`), it checks `pnpm.cmd` before the generic `pnpm` lookup, which uses `PATH`/`PATHEXT` and may select an `.exe` or `.bat` in the diff --git a/scripts/check-data-not-in-use.sh b/scripts/check-data-not-in-use.sh new file mode 100755 index 0000000000..294f210bb3 --- /dev/null +++ b/scripts/check-data-not-in-use.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# +# check-data-not-in-use.sh - Refuse while a DeerFlow Gateway container is running +# +# `make clean` deletes backend/.deer-flow, the runtime data directory (database, +# users, threads, uploads, secrets). Both Docker stacks mount it into the +# Gateway container: docker-compose.yaml mounts ${DEER_FLOW_HOME} (default +# backend/.deer-flow) and docker-compose-dev.yaml mounts ../backend/. `make stop` +# only stops local services, so deleting the directory under a running Gateway +# would pull its open database out from under it. +# +# Exits 0 when Docker is missing or unreachable: no container can then be +# holding this checkout's data. +# +# Sandbox containers are not checked here: AIO/DooD sandboxes bind-mount thread +# directories under the same tree, but `make clean` runs `make stop` after this +# check and before deleting, and `serve.sh --stop` stops every +# deer-flow-sandbox* container (as `make docker-stop` does). Checking them here +# would refuse the ordinary local AIO case that stop already cleans up. + +set -e + +GATEWAY_CONTAINER="deer-flow-gateway" + +if ! command -v docker >/dev/null 2>&1; then + exit 0 +fi + +# Match the exact name: `docker ps --filter name=` is a substring match. Docker +# on Windows may end lines with CRLF. +if ! names=$(docker ps --format '{{.Names}}' 2>/dev/null); then + exit 0 +fi +if printf '%s\n' "$names" | tr -d '\r' | grep -qx "$GATEWAY_CONTAINER"; then + echo "Refusing to delete DeerFlow runtime data: the $GATEWAY_CONTAINER container is running, and both Docker stacks mount backend/.deer-flow into it." >&2 + echo "Stop the Docker stack first (make down, or make docker-stop for the dev stack), then run make clean again." >&2 + exit 1 +fi