diff --git a/README.md b/README.md index 91867a6b58..4f99ef5e30 100644 --- a/README.md +++ b/README.md @@ -1857,6 +1857,16 @@ Outline titles are limited to 200 characters and fallback previews to 2,000 characters per file, with truncation markers. Full uploaded files remain available for targeted reads. +Converted upload outlines and previews require a matching source version, including +modification timestamps. A detected source-version change invalidates its previous +conversion, including equal-length edits. Older ownership records without source timestamps +are also rejected; re-upload the source with `uploads.auto_convert_documents: true` +to restore conversion-backed outlines. Files remain available, and unvalidated +Markdown conversions appear as standalone files in the agent's historical listing. +On Windows, `st_ctime_ns` may represent creation time; a same-size rewrite that +restores the original `mtime` can evade validation. Timestamp checks are +conservative metadata validation, not a guarantee of content equality. + Image bytes loaded for a vision-model call are transient: DeerFlow removes the hidden base64 message after the model consumes it so later checkpoints do not keep duplicating that payload. After each run, DeerFlow records a workspace change summary for the run-owned `workspace` and `outputs` directories. The Web UI shows a compact "files changed" badge on the assistant turn; opening it reveals created, modified, and deleted files with text diffs when safe to display. Uploads are excluded because they are user inputs, not agent-generated changes, and stdio MCP temporary/debug files under the DeerFlow-owned `.mcp/` namespace are excluded because they are process-internal state (like `.git/` and `node_modules/`, any directory named `.mcp` is excluded at any depth). Large, binary, or sensitive-looking files are shown as metadata only. diff --git a/README_zh.md b/README_zh.md index 8315c55cf6..ae65cb70f0 100644 --- a/README_zh.md +++ b/README_zh.md @@ -772,6 +772,8 @@ lead agent 只会在委派具有明确净收益时动态拉起 sub-agents,例 ### Sandbox 与文件系统 +上传文档的转换大纲和预览会校验原文件版本,包括修改时间戳。检测到原文件版本变化后,即使大小不变,也不会再使用旧转换结果。缺少原文件时间戳的旧归属记录同样会被拒绝;可在启用 `uploads.auto_convert_documents: true` 后重新上传原文件以恢复转换大纲。文件都会保留,未通过校验的转换 Markdown 会在 Agent 的历史文件列表中作为独立文件显示。Windows 上的 `st_ctime_ns` 可能表示创建时间,因此等长度覆盖后若恢复原 `mtime`,校验可能无法识别变化。时间戳校验是保守的元数据校验,不能保证文件内容完全一致。 + DeerFlow 不只是“会说它能做”,它是真的有一台自己的“电脑”。 每个任务都运行在隔离的 Docker 容器里,里面有完整的文件系统,包括 skills、workspace、uploads、outputs。agent 可以读写和编辑文件,可以执行 bash 命令和代码,也可以查看图片。整个过程都在 sandbox 内完成,可审计、会隔离,不会在不同 session 之间互相污染。 diff --git a/backend/AGENTS.md b/backend/AGENTS.md index bec03d7ae9..a894bb04b8 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -338,7 +338,7 @@ Outlines use ATX syntax (1–6 hashes, space/tab separator, ≤3 leading spaces) - Rejects directories before copying to keep uploads all-or-nothing - One conversion worker per request when called from an active event loop - Files stored in thread-isolated directories under the resolving user's bucket (`users/{user_id}/threads/{thread_id}/user-data/uploads`). For IM channels the owner is threaded explicitly via the `user_id=` kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from `get_effective_user_id()` -- Thread ownership: `upload-companions/`. +- Per-thread `upload-companions/`: source mtime/ctime. - Duplicate filenames within one request get `_N` suffixes to prevent overwrites. - Gateway HTTP uploads stage `.upload-*.part` files, hidden from upload listings, agent context, and sandbox listings/searches. After size validation, publication is atomic; staged-name cleanup logs errors and leaves leftovers for startup sweep. - Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with `SandboxProvider.acquire_async()` and offload `read_bytes()` plus `sandbox.update_file()` together. diff --git a/backend/docs/FILE_UPLOAD.md b/backend/docs/FILE_UPLOAD.md index 942635afac..ebdd1b0090 100644 --- a/backend/docs/FILE_UPLOAD.md +++ b/backend/docs/FILE_UPLOAD.md @@ -118,7 +118,9 @@ DELETE /api/threads/{thread_id}/uploads/{filename} 转换后的 Markdown 文件会保存在同一目录下,通常使用原文件的主干名加 `.md`;若该名称已被占用,则追加 `_N` 后缀。上传响应中的 `markdown_file` 给出实际名称。 -新转换会在沙箱挂载目录之外保存原文件与转换文件的归属记录。Agent 的历史文件列表只隐藏归属已验证的转换文件;文档大纲也只读取这份记录指定的 Markdown。旧版本生成的转换文件没有归属记录,升级后会作为独立 Markdown 显示,原文件也不会再从同名文件推断大纲。因为旧文件无法与用户自行上传的同名 Markdown 安全地区分,系统不会自动补建记录;需要大纲时,可在启用自动转换后重新上传原文件。 +新转换会在沙箱挂载目录之外保存原文件与转换文件的归属记录。Agent 的历史文件列表只隐藏归属已验证的转换文件;文档大纲也只读取这份记录指定的 Markdown。原文件校验包含修改时间戳,即使修改前后大小相同,也会拒绝使用旧转换结果。旧版本没有归属记录或记录缺少原文件时间戳的转换文件,升级后会作为独立 Markdown 显示,原文件也不会再从这些文件读取大纲。系统不会自动补建记录或填入当前时间戳,以免将过期转换结果认定为有效;需要大纲时,可在启用 `uploads.auto_convert_documents: true` 后重新上传原文件。原文件和转换文件都会保留。 + +时间戳校验是保守的元数据校验,不能保证文件内容完全一致。Windows 上的 `st_ctime_ns` 可能表示创建时间,因此等长度覆盖后若恢复原 `mtime`,校验可能无法识别变化。 默认情况下,自动转换是关闭的,以避免在网关主机上对不受信任的 Office/PDF 上传执行解析。只有在受信任部署中明确接受此风险时,才应将 `uploads.auto_convert_documents` 设置为 `true`。 diff --git a/backend/packages/harness/deerflow/uploads/companions.py b/backend/packages/harness/deerflow/uploads/companions.py index 50f02f99a9..325c3cc977 100644 --- a/backend/packages/harness/deerflow/uploads/companions.py +++ b/backend/packages/harness/deerflow/uploads/companions.py @@ -8,10 +8,13 @@ from __future__ import annotations import hashlib import json +import logging import os import tempfile from pathlib import Path +logger = logging.getLogger(__name__) + def _records_dir(uploads_dir: Path) -> Path: # Production layout is /user-data/uploads. Tests and embedded @@ -41,7 +44,7 @@ def register_companion(original: Path, markdown: Path) -> None: { "original": original.name, "markdown": markdown.name, - "source_identity": [source_stat.st_dev, source_stat.st_ino, source_stat.st_size], + "source_identity": [source_stat.st_dev, source_stat.st_ino, source_stat.st_size, source_stat.st_mtime_ns, source_stat.st_ctime_ns], "markdown_identity": [markdown_stat.st_dev, markdown_stat.st_ino, markdown_stat.st_size, markdown_stat.st_ctime_ns], }, output, @@ -87,13 +90,21 @@ def resolve_companion(original: Path) -> Path | None: if data["original"] != original.name or not isinstance(name, str) or Path(name).name != name or not name.endswith(".md"): return None source_stat = original.stat(follow_symlinks=False) - if data.get("source_identity") != [source_stat.st_dev, source_stat.st_ino, source_stat.st_size]: + # Legacy three-field records cannot establish whether an equal-size + # source edit happened. Reject them rather than backfilling timestamps. + if data.get("source_identity") != [source_stat.st_dev, source_stat.st_ino, source_stat.st_size, source_stat.st_mtime_ns, source_stat.st_ctime_ns]: + source_identity = data.get("source_identity") + if isinstance(source_identity, list) and len(source_identity) == 3: + logger.debug("Rejected upload companion for %r: legacy source identity lacks version timestamps", original.name) + else: + logger.debug("Rejected upload companion for %r: source identity or version timestamps do not match", original.name) return None markdown = original.parent / name if not markdown.is_file() or markdown.is_symlink(): return None markdown_stat = markdown.stat(follow_symlinks=False) if data.get("markdown_identity") != [markdown_stat.st_dev, markdown_stat.st_ino, markdown_stat.st_size, markdown_stat.st_ctime_ns]: + logger.debug("Rejected upload companion for %r: Markdown identity or version timestamp does not match", original.name) return None return markdown except (OSError, ValueError, KeyError, TypeError): diff --git a/backend/tests/test_list_uploaded_files_tool.py b/backend/tests/test_list_uploaded_files_tool.py index 8c0a4d4572..125a1a1bd2 100644 --- a/backend/tests/test_list_uploaded_files_tool.py +++ b/backend/tests/test_list_uploaded_files_tool.py @@ -286,6 +286,24 @@ class TestListUploadedFiles: assert "outline" not in files["report.pdf"] assert files["report.md"]["outline"] == [{"title": "User replacement with different size", "line": 1}] + def test_same_size_source_edit_exposes_companion_and_drops_stale_outline(self, tmp_path): + uploads_dir = _uploads_dir(tmp_path) + original = uploads_dir / "report.pdf" + companion = uploads_dir / "report.md" + original.write_bytes(b"%PDF ORIGINAL A") + companion.write_text("# Original A\n", encoding="utf-8") + register_companion(original, companion) + before = original.stat() + original.write_bytes(b"%PDF REPLACED B") + os.utime(original, ns=(before.st_atime_ns, before.st_mtime_ns + 2_000_000_000)) + + result = _list_uploaded_files_impl(include_outline=True, runtime=_runtime(), _paths=_paths(tmp_path)) + + files = {f["filename"]: f for f in result["files"]} + assert set(files) == {"report.pdf", "report.md"} + assert "outline" not in files["report.pdf"] + assert files["report.md"]["outline"] == [{"title": "Original A", "line": 1}] + def test_cross_turn_state_clear_does_not_exclude_historical_file(self, tmp_path): """Two-turn regression: file uploaded in turn 1 must appear in turn 2. diff --git a/backend/tests/test_upload_companions.py b/backend/tests/test_upload_companions.py new file mode 100644 index 0000000000..837eb9112a --- /dev/null +++ b/backend/tests/test_upload_companions.py @@ -0,0 +1,100 @@ +"""Source-version validation for converted upload ownership records.""" + +import json +import os +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from deerflow.sandbox.local.local_sandbox import LocalSandbox, PathMapping +from deerflow.uploads.companions import companion_names, register_companion, resolve_companion +from deerflow.utils.file_outline import extract_outline_for_file + + +@pytest.fixture +def converted_upload(tmp_path): + uploads = tmp_path / "thread" / "user-data" / "uploads" + uploads.mkdir(parents=True) + original = uploads / "report.pdf" + markdown = uploads / "report.md" + original.write_bytes(b"%PDF ORIGINAL A") + markdown.write_text("# Original A\n", encoding="utf-8") + register_companion(original, markdown) + return original, markdown + + +def test_unchanged_source_keeps_companion_after_read(converted_upload): + original, markdown = converted_upload + assert original.read_bytes() == b"%PDF ORIGINAL A" + assert resolve_companion(original) == markdown + assert companion_names(original.parent) == {markdown.name} + assert extract_outline_for_file(original)[0] == [{"title": "Original A", "line": 1}] + + +def test_same_size_sandbox_rewrite_rejects_stale_outline(converted_upload): + original, markdown = converted_upload + before = original.stat() + sandbox = LocalSandbox("repro", [PathMapping("/mnt/user-data", str(original.parent.parent))]) + sandbox.write_file("/mnt/user-data/uploads/report.pdf", "%PDF REPLACED B") + # Force a distinct modification time even on coarse-resolution filesystems. + os.utime(original, ns=(before.st_atime_ns, before.st_mtime_ns + 2_000_000_000)) + after = original.stat() + assert (before.st_dev, before.st_ino, before.st_size) == (after.st_dev, after.st_ino, after.st_size) + + assert resolve_companion(original) is None + assert companion_names(original.parent) == set() + assert extract_outline_for_file(original) == ([], []) + assert markdown.read_text(encoding="utf-8") == "# Original A\n" + + +@pytest.mark.parametrize("timestamp", ["st_mtime_ns", "st_ctime_ns"]) +def test_each_source_timestamp_invalidates_companion(converted_upload, monkeypatch, timestamp, caplog): + original, _ = converted_upload + real_stat = Path.stat + before = original.stat() + + def changed_stat(path, *args, **kwargs): + result = real_stat(path, *args, **kwargs) + if path == original: + fields = {name: getattr(before, name) for name in ("st_mode", "st_dev", "st_ino", "st_size", "st_mtime_ns", "st_ctime_ns")} + fields[timestamp] += 2_000_000_000 + return SimpleNamespace(**fields) + return result + + # Model mtime-only updates (e.g. Windows creation-time ctime) and ctime-only + # updates (e.g. POSIX writes followed by restoring the old mtime). + monkeypatch.setattr(Path, "stat", changed_stat) + with caplog.at_level("DEBUG", logger="deerflow.uploads.companions"): + assert resolve_companion(original) is None + assert "source identity or version timestamps do not match" in caplog.text + + +def test_legacy_source_record_requires_reregistration(converted_upload, caplog): + original, markdown = converted_upload + record = next((original.parent.parent.parent / "upload-companions").glob("*.json")) + data = json.loads(record.read_text(encoding="utf-8")) + data["source_identity"] = data["source_identity"][:3] + record.write_text(json.dumps(data), encoding="utf-8") + legacy = record.read_bytes() + + with caplog.at_level("DEBUG", logger="deerflow.uploads.companions"): + assert resolve_companion(original) is None + assert "legacy source identity lacks version timestamps" in caplog.text + assert extract_outline_for_file(original) == ([], []) + assert record.read_bytes() == legacy # Do not bless stale content on read. + + register_companion(original, markdown) + assert resolve_companion(original) == markdown + + +def test_new_conversion_restores_outline_after_source_edit(converted_upload): + original, markdown = converted_upload + before = original.stat() + original.write_bytes(b"%PDF REPLACED B") + os.utime(original, ns=(before.st_atime_ns, before.st_mtime_ns + 2_000_000_000)) + markdown.write_text("# Replaced B\n", encoding="utf-8") + register_companion(original, markdown) + + assert resolve_companion(original) == markdown + assert extract_outline_for_file(original)[0] == [{"title": "Replaced B", "line": 1}]