diff --git a/.agent/skills/smoke-test/templates/report.docker.template.md b/.agent/skills/smoke-test/templates/report.docker.template.md index 7313ed105c..5bebe1ea7c 100644 --- a/.agent/skills/smoke-test/templates/report.docker.template.md +++ b/.agent/skills/smoke-test/templates/report.docker.template.md @@ -162,7 +162,7 @@ Commit Message: {{git_commit_message}} 1. [ ] Review references/troubleshooting.md for common solutions 2. [ ] Check Docker logs: `make docker-logs` 3. [ ] Verify configuration file format and content -4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start` +4. [ ] If needed, fully reset the environment: `make docker-stop && make clean && make config && make docker-init && make docker-start` --- diff --git a/CHANGELOG.md b/CHANGELOG.md index cb3f66e871..4917003b1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -623,6 +623,15 @@ This release closes that milestone with **439 merged pull requests**. ### Fixed +- **make:** `make clean` now says what it deletes and refuses to run under a live + Docker Gateway. `make help` described it as cleaning up "temporary files", but + it deletes `backend/.deer-flow`: the local database, users, threads, uploads, + memory, and secrets. Both Docker stacks mount that directory into the + `deer-flow-gateway` container, which `make stop` leaves running, so the data + could be deleted under a live Gateway. The help text and a notice before + deletion now name the directory, and `make clean` stops with a hint to run + `make down` or `make docker-stop` while that container is running, before it + stops anything. ([#6351]) - **tui:** A new prompt no longer races an interrupted run in the same conversation. After `Ctrl+C` the run's worker keeps going until its current step returns, so a tool call already in progress (a long shell command, for @@ -8818,3 +8827,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#6344]: https://github.com/bytedance/deer-flow/pull/6344 [#6347]: https://github.com/bytedance/deer-flow/pull/6347 [#6350]: https://github.com/bytedance/deer-flow/pull/6350 +[#6351]: https://github.com/bytedance/deer-flow/pull/6351 diff --git a/CHANGELOG_zh.md b/CHANGELOG_zh.md index 467f2d8c8e..4053062c45 100644 --- a/CHANGELOG_zh.md +++ b/CHANGELOG_zh.md @@ -558,6 +558,11 @@ ### 修复 +- **make：** `make clean` 现在会说明它删除的内容，并拒绝在运行中的 Docker Gateway 下执行。`make help` + 此前称其清理"临时文件"，但它实际删除 `backend/.deer-flow`：本地数据库、用户、线程、上传、记忆和密钥。 + 两套 Docker 栈都把该目录挂载进 `deer-flow-gateway` 容器，而 `make stop` 不会停止它，因此数据可能在 + Gateway 运行时被删除。现在帮助文本和删除前的提示会写明该目录；当该容器仍在运行时，`make clean` + 会在停止任何服务之前退出，并提示先执行 `make down` 或 `make docker-stop`。([#6351]) - **TUI：** 新的提示不再与同一会话中被中断的运行发生竞争。按下 `Ctrl+C` 后，该运行的工作线程会 一直执行到当前步骤返回，因此正在进行的工具调用（例如耗时较长的 shell 命令）仍会完成并写入检查点， 而下一条提示已在同一线程上运行；最后写入检查点的运行会成为会话历史，新的一轮可能因此悄然消失。 @@ -7303,3 +7308,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子 [#6343]: https://github.com/bytedance/deer-flow/pull/6343 [#6344]: https://github.com/bytedance/deer-flow/pull/6344 [#6350]: https://github.com/bytedance/deer-flow/pull/6350 +[#6351]: https://github.com/bytedance/deer-flow/pull/6351 diff --git a/Makefile b/Makefile index 72ee584e78..9ccaea34eb 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,8 @@ else endif FRONTEND_PNPM = $(PYTHON) ../scripts/pnpm.py +# What `make clean` deletes; shared by its help line and deletion notice. +RUNTIME_DATA_CONTENTS = database, users, threads, uploads, memory, secrets help: @echo "DeerFlow Development Commands:" @@ -46,7 +48,7 @@ help: @echo " make start-daemon - Start prod services in background (daemon mode)" @echo " make nginx - Start nginx alone in the foreground (local dev config)" @echo " make stop - Stop all running services" - @echo " make clean - Clean up processes and temporary files" + @echo " make clean - Stop local services and DELETE local runtime data (backend/.deer-flow: $(RUNTIME_DATA_CONTENTS)) and logs" @echo "" @echo "Docker Production Commands:" @echo " make up - Build and start production Docker services (localhost:2026)" @@ -172,9 +174,12 @@ nginx: stop: @$(RUN_SHELL_SCRIPT) ./scripts/serve.sh --stop -# Clean up -clean: stop - @echo "Cleaning up..." +# Clean up: deletes local runtime data, not just temporary files. The guard +# runs before stop, which would otherwise stop a running stack's sandboxes. +clean: + @$(RUN_SHELL_SCRIPT) ./scripts/check-data-not-in-use.sh + @$(MAKE) --no-print-directory stop + @echo "Deleting local runtime data in backend/.deer-flow ($(RUNTIME_DATA_CONTENTS)) and logs/*.log..." @-rm -rf backend/.deer-flow 2>/dev/null || true @-rm -rf logs/*.log 2>/dev/null || true @echo "✓ Cleanup complete" diff --git a/backend/docs/APPLE_CONTAINER.md b/backend/docs/APPLE_CONTAINER.md index a58bee19f8..e25c9ed66d 100644 --- a/backend/docs/APPLE_CONTAINER.md +++ b/backend/docs/APPLE_CONTAINER.md @@ -170,7 +170,7 @@ The project includes a unified cleanup script that handles both runtimes: All cleanup commands in `Makefile` automatically handle both runtimes: ```bash make stop # Stops all services and cleans up containers -make clean # Full cleanup including logs +make clean # Also deletes local runtime data (backend/.deer-flow) and logs ``` ## Testing diff --git a/backend/tests/test_make_clean_guard.py b/backend/tests/test_make_clean_guard.py new file mode 100644 index 0000000000..79ef756b83 --- /dev/null +++ b/backend/tests/test_make_clean_guard.py @@ -0,0 +1,132 @@ +"""`make clean` says what it deletes and refuses while the Gateway container runs. + +The target deletes ``backend/.deer-flow`` -- the database, users, threads, +uploads, and secrets -- but ``make help`` described it as cleaning up +"temporary files". Both Docker stacks mount that directory into the +``deer-flow-gateway`` container, and ``make stop`` only stops local services, +so ``make clean`` used to delete the data under a running Gateway. + +These tests run the real ``make clean`` from the real Makefile and guard script +in a scratch checkout. ``scripts/serve.sh`` is a stub that records the stop +call, so nothing touches the host's processes or containers, and ``docker`` is +a fake on a PATH limited to the tools the recipe needs. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + +pytestmark = pytest.mark.skipif( + os.name == "nt" or shutil.which("make") is None or shutil.which("bash") is None, + reason="runs the POSIX make recipe", +) + +_TOOLS = ("make", "bash", "sh", "rm", "grep", "tr") + + +def _checkout(tmp_path: Path) -> Path: + root = tmp_path / "checkout" + (root / "scripts").mkdir(parents=True) + shutil.copy2(REPO_ROOT / "Makefile", root / "Makefile") + shutil.copy2(REPO_ROOT / "scripts" / "check-data-not-in-use.sh", root / "scripts" / "check-data-not-in-use.sh") + # Records each stop call and whether the data still existed at that point: + # the real serve.sh --stop also stops deer-flow-sandbox* containers, which + # bind-mount thread directories under backend/.deer-flow. + (root / "scripts" / "serve.sh").write_text( + '#!/usr/bin/env bash\nif [ -e backend/.deer-flow ]; then state=data-present; else state=data-gone; fi\necho "$* $state" >> stop-calls\n', + encoding="utf-8", + ) + data = root / "backend" / ".deer-flow" / "data" + data.mkdir(parents=True) + (data / "deerflow.db").write_text("db", encoding="utf-8") + (root / "logs").mkdir() + (root / "logs" / "gateway.log").write_text("log", encoding="utf-8") + return root + + +def _bin(tmp_path: Path, docker: str | None) -> Path: + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + for tool in _TOOLS: + found = shutil.which(tool) + assert found, tool + (bin_dir / tool).symlink_to(found) + if docker is not None: + fake = bin_dir / "docker" + fake.write_text(f"#!/bin/sh\n{docker}\n", encoding="utf-8") + fake.chmod(0o755) + return bin_dir + + +def _make(root: Path, bin_dir: Path, target: str = "clean") -> subprocess.CompletedProcess[str]: + env = {"PATH": str(bin_dir), "HOME": str(root)} + return subprocess.run(["make", "-s", target], cwd=root, env=env, capture_output=True, text=True, timeout=60) + + +def test_help_says_clean_deletes_runtime_data(tmp_path: Path) -> None: + root = _checkout(tmp_path) + + help_line = next(line for line in _make(root, _bin(tmp_path, None), "help").stdout.splitlines() if "make clean" in line) + + assert "temporary files" not in help_line + assert "DELETE" in help_line and "backend/.deer-flow" in help_line + + +@pytest.mark.parametrize( + "names", + ["deer-flow-gateway", "deer-flow-redis\ndeer-flow-gateway", "deer-flow-gateway\r"], + ids=["gateway", "whole-stack", "crlf"], +) +def test_clean_refuses_while_the_gateway_container_runs(tmp_path: Path, names: str) -> None: + root = _checkout(tmp_path) + listing = names.replace("\n", "\\n").replace("\r", "\\r") + + result = _make(root, _bin(tmp_path, f"printf '{listing}\\n'")) + + assert result.returncode != 0 + assert "deer-flow-gateway container is running" in result.stderr + assert (root / "backend" / ".deer-flow" / "data" / "deerflow.db").exists() + assert (root / "logs" / "gateway.log").exists() + # The guard runs before stop, which would stop the stack's sandboxes. + assert not (root / "stop-calls").exists() + + +@pytest.mark.parametrize( + "docker", + [None, "exit 1", "printf 'deer-flow-gateway-old\\nmy-deer-flow-gateway\\ndeer-flow-sandbox-1\\n'"], + ids=["no-docker", "daemon-down", "other-containers"], +) +def test_clean_deletes_runtime_data_when_no_gateway_container_runs(tmp_path: Path, docker: str | None) -> None: + root = _checkout(tmp_path) + + result = _make(root, _bin(tmp_path, docker)) + + assert result.returncode == 0, result.stderr + assert "Deleting local runtime data in backend/.deer-flow" in result.stdout + assert not (root / "backend" / ".deer-flow").exists() + assert not (root / "logs" / "gateway.log").exists() + # stop (and with it the sandbox-container cleanup) runs before the deletion. + assert (root / "stop-calls").read_text(encoding="utf-8").split() == ["--stop", "data-present"] + + +def _listed_contents(line: str, prefix: str) -> list[str]: + return line.split(prefix, 1)[1].split(")", 1)[0].split(", ") + + +def test_help_and_deletion_notice_list_the_same_data(tmp_path: Path) -> None: + root = _checkout(tmp_path) + bin_dir = _bin(tmp_path, None) + + help_line = next(line for line in _make(root, bin_dir, "help").stdout.splitlines() if "make clean" in line) + notice = next(line for line in _make(root, bin_dir).stdout.splitlines() if line.startswith("Deleting local runtime data")) + + listed = _listed_contents(help_line, "backend/.deer-flow: ") + assert listed == _listed_contents(notice, "backend/.deer-flow (") + assert {"database", "users", "threads", "uploads", "memory", "secrets"} <= set(listed) diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 57446b26b9..87f64e52d9 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -90,6 +90,14 @@ likewise prefix the target with `bash`. This keeps documented `make` commands working when a source archive, `core.fileMode=false`, or a non-POSIX filesystem does not preserve executable bits. +`make clean` deletes `backend/.deer-flow` (database, users, threads, uploads, +secrets), which both compose stacks mount into `deer-flow-gateway`. Its recipe +runs `check-data-not-in-use.sh` before `make stop` (which would stop a live +stack's sandboxes) and refuses while that container runs; an absent or +unreachable Docker passes. `make stop` must still run before the delete: it +stops `deer-flow-sandbox*` containers, whose thread mounts live in that tree. +`RUNTIME_DATA_CONTENTS` feeds both the help line and the deletion notice. + Host-side pnpm calls must go through `scripts/pnpm.py`. With native Windows Python (`os.name == "nt"`), it checks `pnpm.cmd` before the generic `pnpm` lookup, which uses `PATH`/`PATHEXT` and may select an `.exe` or `.bat` in the diff --git a/scripts/check-data-not-in-use.sh b/scripts/check-data-not-in-use.sh new file mode 100755 index 0000000000..294f210bb3 --- /dev/null +++ b/scripts/check-data-not-in-use.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# +# check-data-not-in-use.sh - Refuse while a DeerFlow Gateway container is running +# +# `make clean` deletes backend/.deer-flow, the runtime data directory (database, +# users, threads, uploads, secrets). Both Docker stacks mount it into the +# Gateway container: docker-compose.yaml mounts ${DEER_FLOW_HOME} (default +# backend/.deer-flow) and docker-compose-dev.yaml mounts ../backend/. `make stop` +# only stops local services, so deleting the directory under a running Gateway +# would pull its open database out from under it. +# +# Exits 0 when Docker is missing or unreachable: no container can then be +# holding this checkout's data. +# +# Sandbox containers are not checked here: AIO/DooD sandboxes bind-mount thread +# directories under the same tree, but `make clean` runs `make stop` after this +# check and before deleting, and `serve.sh --stop` stops every +# deer-flow-sandbox* container (as `make docker-stop` does). Checking them here +# would refuse the ordinary local AIO case that stop already cleans up. + +set -e + +GATEWAY_CONTAINER="deer-flow-gateway" + +if ! command -v docker >/dev/null 2>&1; then + exit 0 +fi + +# Match the exact name: `docker ps --filter name=` is a substring match. Docker +# on Windows may end lines with CRLF. +if ! names=$(docker ps --format '{{.Names}}' 2>/dev/null); then + exit 0 +fi +if printf '%s\n' "$names" | tr -d '\r' | grep -qx "$GATEWAY_CONTAINER"; then + echo "Refusing to delete DeerFlow runtime data: the $GATEWAY_CONTAINER container is running, and both Docker stacks mount backend/.deer-flow into it." >&2 + echo "Stop the Docker stack first (make down, or make docker-stop for the dev stack), then run make clean again." >&2 + exit 1 +fi