diff --git a/README.md b/README.md index 958cd4e53..14df997f4 100644 --- a/README.md +++ b/README.md @@ -961,6 +961,8 @@ DeerFlow doesn't just *talk* about doing things. It has its own computer. Each task gets its own execution environment with a full filesystem view — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It can view images and, when configured safely, execute shell commands. +Uploads from the Web UI, embedded client, and IM channels share one collision-safe storage rule. A completed payload is published only if its candidate name does not exist; concurrent `report.pdf` uploads become `report.pdf`, `report_1.pdf`, `report_2.pdf`, and so on without replacing one another. Optional document conversions are system-owned assets under `/mnt/user-data/.upload-conversions/.md`. They are returned through the upload response but omitted from the primary upload listing. Deleting a primary removes only its exact generated asset and never infers that a user-uploaded sibling such as `uploads/report.md` is disposable. + The built-in `grep` tool searches either one text file or all matching text files below a directory, so an agent can search an uploaded document directly without first broadening the request to the entire uploads directory. Image bytes loaded for a vision-model call are transient: DeerFlow removes the hidden base64 message after the model consumes it so later checkpoints do not keep duplicating that payload. @@ -990,9 +992,10 @@ This is the difference between a chatbot with tool access and an agent with an a ``` # Paths inside the sandbox container /mnt/user-data/ -├── uploads/ ← your files -├── workspace/ ← agents' working directory -└── outputs/ ← final deliverables +├── uploads/ ← your primary files +├── .upload-conversions/ ← generated Markdown (hidden from upload listings) +├── workspace/ ← agents' working directory +└── outputs/ ← final deliverables ``` ### Agentic Browser Control diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 566baca45..8f9b4ffc7 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -1460,8 +1460,9 @@ Multi-file upload with automatic document conversion: - Rejects directory inputs before copying so uploads stay all-or-nothing - Reuses one conversion worker per request when called from an active event loop - Files stored in thread-isolated directories under the resolving user's bucket (`users/{user_id}/threads/{thread_id}/user-data/uploads`). For IM channels the owner is threaded explicitly via the `user_id=` kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from `get_effective_user_id()` -- Duplicate filenames in a single upload request are auto-renamed with `_N` suffixes so later files do not truncate earlier files -- Gateway HTTP uploads stage bytes as `.upload-*.part` files and atomically replace the destination only after size validation. These staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind. +- Every ingress stages a complete payload and atomically publishes it without replacing an existing entry. Collisions across requests, processes, HTTP, embedded, and IM adapters use `name.ext`, `name_1.ext`, `name_2.ext`; storage that cannot provide atomic no-replace publication fails explicitly. +- Gateway HTTP uploads use same-directory `.upload-*.part` staging files. Staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind. +- Generated Markdown is owned by `user-data/.upload-conversions/.md` and is omitted from primary upload listings. Deletion removes only the selected primary and that exact generated asset; it never guesses or deletes a legacy/user-owned `uploads/.md` sibling. - Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with `SandboxProvider.acquire_async()` and offload `read_bytes()` plus `sandbox.update_file()` together. - Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate `sandbox.thread_data_mounts: true`; omission preserves backend auto-detection. - Agent receives uploaded file list via `UploadsMiddleware` diff --git a/backend/docs/API.md b/backend/docs/API.md index f6948d27c..d02b65fd6 100644 --- a/backend/docs/API.md +++ b/backend/docs/API.md @@ -620,10 +620,10 @@ Content-Type: multipart/form-data "path": ".deer-flow/threads/abc123/user-data/uploads/document.pdf", "virtual_path": "/mnt/user-data/uploads/document.pdf", "artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/uploads/document.pdf", - "markdown_file": "document.md", - "markdown_path": ".deer-flow/threads/abc123/user-data/uploads/document.md", - "markdown_virtual_path": "/mnt/user-data/uploads/document.md", - "markdown_artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/uploads/document.md" + "markdown_file": "document.pdf.md", + "markdown_path": ".deer-flow/threads/abc123/user-data/.upload-conversions/document.pdf.md", + "markdown_virtual_path": "/mnt/user-data/.upload-conversions/document.pdf.md", + "markdown_artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/document.pdf.md" } ], "message": "Successfully uploaded 1 file(s)" @@ -636,6 +636,8 @@ Content-Type: multipart/form-data - Excel (`.xls`, `.xlsx`) - Word (`.doc`, `.docx`) +All upload entry points publish complete payloads without replacing an existing name. Concurrent collisions are returned as `document.pdf`, `document_1.pdf`, `document_2.pdf`, and so on. Generated Markdown is stored outside the primary namespace and is not returned by the list endpoint. Deleting `document.pdf` also deletes only `.upload-conversions/document.pdf.md`; an independent `uploads/document.md` is preserved. + #### List Uploaded Files ```http diff --git a/backend/docs/FILE_UPLOAD.md b/backend/docs/FILE_UPLOAD.md index 778b57210..1e62b80c7 100644 --- a/backend/docs/FILE_UPLOAD.md +++ b/backend/docs/FILE_UPLOAD.md @@ -9,6 +9,8 @@ DeerFlow 后端提供了完整的文件上传功能，支持多文件上传， - ✅ 支持多文件同时上传 - ✅ 可选地转换文档为 Markdown（PDF、PPT、Excel、Word） - ✅ 文件存储在线程隔离的目录中 +- ✅ 跨请求、跨进程的同名文件不会互相覆盖 +- ✅ 生成的 Markdown 与用户上传命名空间隔离 - ✅ Agent 自动感知当前消息中附带的文件 - ✅ 支持文件列表查询和删除 @@ -35,10 +37,10 @@ POST /api/threads/{thread_id}/uploads "path": ".deer-flow/threads/{thread_id}/user-data/uploads/document.pdf", "virtual_path": "/mnt/user-data/uploads/document.pdf", "artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.pdf", - "markdown_file": "document.md", - "markdown_path": ".deer-flow/threads/{thread_id}/user-data/uploads/document.md", - "markdown_virtual_path": "/mnt/user-data/uploads/document.md", - "markdown_artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.md" + "markdown_file": "document.pdf.md", + "markdown_path": ".deer-flow/threads/{thread_id}/user-data/.upload-conversions/document.pdf.md", + "markdown_virtual_path": "/mnt/user-data/.upload-conversions/document.pdf.md", + "markdown_artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/.upload-conversions/document.pdf.md" } ], "message": "Successfully uploaded 1 file(s)" @@ -50,6 +52,8 @@ POST /api/threads/{thread_id}/uploads - `virtual_path`: Agent 在沙箱中使用的虚拟路径 - `artifact_url`: 前端通过 HTTP 访问文件的 URL +所有上传入口都先完整写入同目录暂存文件，再以“不替换已有条目”的原子操作发布。同名碰撞依次命名为 `document.pdf`、`document_1.pdf`、`document_2.pdf`；响应中的 `filename` 和各路径字段始终使用实际发布名。 + ### 2. 查询上传限制 ``` GET /api/threads/{thread_id}/uploads/limits @@ -89,6 +93,8 @@ GET /api/threads/{thread_id}/uploads/list } ``` +列表只包含 `uploads/` 下的用户主文件；系统生成的 `.upload-conversions/` 资产不会出现在该接口中。 + ### 4. 删除文件 ``` DELETE /api/threads/{thread_id}/uploads/{filename} @@ -102,6 +108,8 @@ DELETE /api/threads/{thread_id}/uploads/{filename} } ``` +删除 `document.pdf` 时，只会额外删除它精确拥有的 `.upload-conversions/document.pdf.md`。系统不会推断或删除 `uploads/document.md`；该文件可能是用户独立上传的内容。 + ## 支持的文档格式 以下格式在显式启用 `uploads.auto_convert_documents: true` 时会自动转换为 Markdown： @@ -110,7 +118,15 @@ DELETE /api/threads/{thread_id}/uploads/{filename} - Excel (`.xls`, `.xlsx`) - Word (`.doc`, `.docx`) -转换后的 Markdown 文件会保存在同一目录下，文件名为原文件名 + `.md` 扩展名。 +转换后的 Markdown 文件保存在系统拥有的 `.upload-conversions/` 目录中，文件名包含完整的实际主文件名。例如： + +```text +Primary: /mnt/user-data/uploads/report.pdf +Generated: /mnt/user-data/.upload-conversions/report.pdf.md +Collision: report.pdf, report_1.pdf, report_2.pdf +Deletion: 删除 report.pdf 时只删除 .upload-conversions/report.pdf.md； + /mnt/user-data/uploads/report.md 永远不会被推断为生成文件或自动删除。 +``` 默认情况下，自动转换是关闭的，以避免在网关主机上对不受信任的 Office/PDF 上传执行解析。只有在受信任部署中明确接受此风险时，才应将 `uploads.auto_convert_documents` 设置为 `true`。 @@ -149,13 +165,14 @@ Agent 在沙箱中运行，使用虚拟路径访问文件。Agent 可以直接 read_file(path="/mnt/user-data/uploads/document.pdf") # 读取转换后的 Markdown（推荐） -read_file(path="/mnt/user-data/uploads/document.md") +read_file(path="/mnt/user-data/.upload-conversions/document.pdf.md") ``` **路径映射关系：** - Agent 使用：`/mnt/user-data/uploads/document.pdf`（虚拟路径） - 实际存储：`backend/.deer-flow/threads/{thread_id}/user-data/uploads/document.pdf` - 前端访问：`/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.pdf`（HTTP URL） +- 转换结果：`/mnt/user-data/.upload-conversions/document.pdf.md`（以上传响应的 `markdown_virtual_path` 为准，不要自行推导） 上传流程采用“线程目录优先”策略： - 先写入 `backend/.deer-flow/threads/{thread_id}/user-data/uploads/` 作为权威存储 @@ -222,12 +239,13 @@ print(response.json()) backend/.deer-flow/threads/ └── {thread_id}/ └── user-data/ - └── uploads/ - ├── document.pdf # 原始文件 - ├── document.md # 转换后的 Markdown - ├── presentation.pptx - ├── presentation.md - └── ... + ├── uploads/ + │ ├── document.pdf # 用户主文件 + │ ├── document.md # 用户独立上传，绝不按名称推断归属 + │ └── presentation.pptx + └── .upload-conversions/ + ├── document.pdf.md # document.pdf 的生成结果 + └── presentation.pptx.md # presentation.pptx 的生成结果 ``` ## 限制 @@ -243,7 +261,8 @@ backend/.deer-flow/threads/ 1. **Upload Router** (`app/gateway/routers/uploads.py`) - 处理文件上传、列表、删除请求 - - 使用 markitdown 转换文档 + - 流式写入暂存文件，并通过共享上传管理器原子发布 + - 使用 markitdown 转换文档；生成文件发布到系统拥有的隔离目录 2. **Uploads Middleware** (`packages/harness/deerflow/agents/middlewares/uploads_middleware.py`) - 读取当前消息的 `additional_kwargs.files` diff --git a/backend/docs/PATH_EXAMPLES.md b/backend/docs/PATH_EXAMPLES.md index f9d2b9f16..a019c22a3 100644 --- a/backend/docs/PATH_EXAMPLES.md +++ b/backend/docs/PATH_EXAMPLES.md @@ -102,18 +102,18 @@ async function uploadAndProcess(threadId: string, file: File) { // path: ".deer-flow/threads/abc123/user-data/uploads/report.pdf", // virtual_path: "/mnt/user-data/uploads/report.pdf", // artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/uploads/report.pdf", - // markdown_file: "report.md", - // markdown_path: ".deer-flow/threads/abc123/user-data/uploads/report.md", - // markdown_virtual_path: "/mnt/user-data/uploads/report.md", - // markdown_artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/uploads/report.md" + // markdown_file: "report.pdf.md", + // markdown_path: ".deer-flow/threads/abc123/user-data/.upload-conversions/report.pdf.md", + // markdown_virtual_path: "/mnt/user-data/.upload-conversions/report.pdf.md", + // markdown_artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/report.pdf.md" // } // 2. 发送消息给 Agent await sendMessage(threadId, "请分析刚上传的 PDF 文件"); - // Agent 会自动看到文件列表，包含： + // Agent 的当前上传上下文包含主文件： // - report.pdf (虚拟路径: /mnt/user-data/uploads/report.pdf) - // - report.md (虚拟路径: /mnt/user-data/uploads/report.md) + // 转换结果必须使用上传响应返回的 markdown_virtual_path，不要推导同目录文件名。 // 3. 前端可以直接访问转换后的 Markdown const mdResponse = await fetch(fileInfo.markdown_artifact_url); @@ -135,6 +135,8 @@ async function uploadAndProcess(threadId: string, file: File) { | 服务器后端代码直接访问 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` | | Agent 工具调用 | `virtual_path` | `/mnt/user-data/uploads/file.pdf` | | 前端下载/预览 | `artifact_url` | `/api/threads/abc123/artifacts/mnt/user-data/uploads/file.pdf` | +| Agent 读取生成 Markdown | `markdown_virtual_path` | `/mnt/user-data/.upload-conversions/file.pdf.md` | +| 前端读取生成 Markdown | `markdown_artifact_url` | `/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/file.pdf.md` | | 备份脚本 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` | | 日志记录 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` | @@ -172,10 +174,8 @@ async function listUploadedFiles(threadId) { console.log(`下载: ${file.artifact_url}?download=true`); console.log(`预览: ${file.artifact_url}`); - // 如果是文档，还有 Markdown 版本 - if (file.markdown_artifact_url) { - console.log(`Markdown: ${file.markdown_artifact_url}`); - } + // 列表接口只返回主文件。markdown_* 字段仅在本次上传响应中返回， + // 调用方如需保留转换链接，应保存该响应元数据。 }); return data.files; @@ -204,7 +204,6 @@ interface UploadedFile { artifact_url: string; extension: string; modified: number; - markdown_artifact_url?: string; } function FileUploadList({ threadId }: { threadId: string }) { @@ -254,9 +253,6 @@ function FileUploadList({ threadId }: { threadId: string }) { {file.filename} 预览 下载 - {file.markdown_artifact_url && ( - Markdown - )} ))} @@ -285,5 +281,8 @@ function FileUploadList({ threadId }: { threadId: string }) { 4. **Markdown 转换** - 转换成功时，会返回额外的 `markdown_*` 字段 + - 生成文件位于 `.upload-conversions/<完整主文件名>.md`，不会出现在主文件列表中 + - 同名主文件按 `file.pdf`、`file_1.pdf`、`file_2.pdf` 原子发布，不会覆盖 + - 删除主文件只删除其精确生成资产，不会删除用户上传的 `uploads/file.md` - 建议优先使用 Markdown 版本（更易处理） - 原始文件始终保留 diff --git a/backend/docs/rfc-extract-shared-modules.md b/backend/docs/rfc-extract-shared-modules.md index 8c931a7d5..abfcb997c 100644 --- a/backend/docs/rfc-extract-shared-modules.md +++ b/backend/docs/rfc-extract-shared-modules.md @@ -30,6 +30,8 @@ Gateway (`app/gateway/routers/skills.py`, `uploads.py`) and Client (`deerflow/cl **The same traversal check is written twice** — any security fix must be applied to both locations. +The final shared design below supersedes this historical baseline: both adapters now use the same atomic no-replace publisher and owned conversion layout. + ## 2. Design Principles ### Dependency Direction @@ -91,11 +93,15 @@ class SkillAlreadyExistsError(ValueError) get_uploads_dir(thread_id: str) -> Path # Pure path, no side effects ensure_uploads_dir(thread_id: str) -> Path # Creates directory (for write paths) -# Filename safety +# Filename safety and atomic primary publication normalize_filename(filename: str) -> str # Path.name extraction + rejects ".." / "." / backslash / >255 bytes -deduplicate_filename(name: str, seen: set) -> str - # _N suffix increment for dedup, mutates seen in place +create_upload_staging_file(base_dir: Path) -> StagedUpload +publish_staged_upload(staged, preferred_filename) -> Path +publish_upload_bytes(base_dir, preferred_filename, data) -> Path +publish_upload_copy(base_dir, preferred_filename, source_path) -> Path + # Complete same-directory staging + atomic hard-link no-replace publication. + # Collisions retry name.ext, name_1.ext, name_2.ext across requests/processes. # Path safety validate_path_traversal(path: Path, base: Path) -> None @@ -107,7 +113,7 @@ list_files_in_dir(directory: Path) -> dict # follow_symlinks=False to prevent metadata leakage # Non-existent directory returns empty list delete_file_safe(base_dir: Path, filename: str) -> dict - # Validates traversal first, then unlinks + # Deletes the primary and only its exact owned conversion # URL helpers upload_artifact_url(thread_id, filename) -> str # Percent-encoded for HTTP safety @@ -115,6 +121,8 @@ upload_virtual_path(filename) -> str # Sandbox-internal path enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies sizes ``` +`deerflow.uploads.layout` owns primary/conversion physical paths, sandbox virtual paths, and artifact URLs. Generated Markdown is published through `deerflow.uploads.conversion` at `user-data/.upload-conversions/.md`; it is outside the primary listing namespace and never inferred from an `uploads/.md` sibling. + ## 4. Changes ### 4.1 Gateway Slimming @@ -127,8 +135,9 @@ enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies size **`app/gateway/routers/uploads.py`**: - Remove inline `get_uploads_dir` (replaced by `ensure_uploads_dir`/`get_uploads_dir`) - `upload_files` uses `normalize_filename()` instead of inline safety checks +- Streamed bytes are completed in shared staging and `publish_staged_upload()` returns the actual collision-safe name - `list_uploaded_files` uses `list_files_in_dir()` + enrichment -- `delete_uploaded_file` uses `delete_file_safe()` + companion markdown cleanup +- `delete_uploaded_file` uses `delete_file_safe()` for exact owned-conversion cleanup ### 4.2 Client Slimming @@ -136,7 +145,7 @@ enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies size - Remove `_get_uploads_dir` static method - Remove ~50 lines of inline zip handling in `install_skill` - `install_skill` delegates to `install_skill_from_archive()` -- `upload_files` uses `deduplicate_filename()` + `ensure_uploads_dir()` +- `upload_files` uses `publish_upload_copy()` + `ensure_uploads_dir()` - `list_uploads` uses `get_uploads_dir()` + `list_files_in_dir()` - `delete_upload` uses `get_uploads_dir()` + `delete_file_safe()` - `update_mcp_config` / `update_skill` now reset `_agent_config_key = None` @@ -164,6 +173,8 @@ Read paths no longer have `mkdir` side effects — non-existent directories retu | Listing symlink leak | `follow_symlinks=True` (default) | `follow_symlinks=False` | | 409 status routing | `"already exists" in str(e)` | `SkillAlreadyExistsError` type match | | Artifact URL encoding | Raw filename in URL | `urllib.parse.quote()` | +| Concurrent same-name writes | Scan/claim then replace | Atomic no-replace publication with `_N` retry | +| Generated Markdown ownership | Guessed `uploads/.md` sibling | Exact `.upload-conversions/.md` asset | ## 6. Alternatives Considered diff --git a/backend/packages/harness/deerflow/agents/lead_agent/prompt.py b/backend/packages/harness/deerflow/agents/lead_agent/prompt.py index 235f6461f..26cbe6357 100644 --- a/backend/packages/harness/deerflow/agents/lead_agent/prompt.py +++ b/backend/packages/harness/deerflow/agents/lead_agent/prompt.py @@ -597,7 +597,8 @@ You: "Deploying to staging..." [proceed] **File Management:** - Newly uploaded files in this run are listed in the `` section before your first response - Use `read_file` tool to read uploaded files using their paths from the list -- For PDF, PPT, Excel, and Word files, converted Markdown versions (*.md) are available alongside originals +- For PDF, PPT, Excel, and Word files, a generated Markdown version may be available under `/mnt/user-data/.upload-conversions/.md` +- Use the exact generated path supplied in upload metadata or outlines and never infer an `uploads/.md` sibling - Files uploaded in previous turns are NOT automatically listed. Use `list_uploaded_files` to discover them on demand — it returns filenames, sizes, and optionally document outlines - All temporary work happens in `/mnt/user-data/workspace` - Treat `/mnt/user-data/workspace` as your default current working directory for coding and file-editing tasks diff --git a/backend/tests/test_lead_agent_prompt.py b/backend/tests/test_lead_agent_prompt.py index 5ad449d57..4dc63843f 100644 --- a/backend/tests/test_lead_agent_prompt.py +++ b/backend/tests/test_lead_agent_prompt.py @@ -103,6 +103,8 @@ def test_apply_prompt_template_includes_relative_path_guidance(monkeypatch): assert "Treat `/mnt/user-data/workspace` as your default current working directory" in prompt assert "`hello.txt`, `../uploads/data.csv`, and `../outputs/report.md`" in prompt + assert "`/mnt/user-data/.upload-conversions/.md`" in prompt + assert "never infer an `uploads/.md` sibling" in prompt def test_apply_prompt_template_includes_memory_tool_guidance_only_in_tool_mode(monkeypatch):