docs: define collision-safe upload layout

This commit is contained in:
hetaoBackend 2026-08-06 10:26:53 +08:00
parent d1f41a9d57
commit 8a3cb391f6
8 changed files with 81 additions and 43 deletions

View File

@ -961,6 +961,8 @@ DeerFlow doesn't just *talk* about doing things. It has its own computer.
Each task gets its own execution environment with a full filesystem view — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It can view images and, when configured safely, execute shell commands.
Uploads from the Web UI, embedded client, and IM channels share one collision-safe storage rule. A completed payload is published only if its candidate name does not exist; concurrent `report.pdf` uploads become `report.pdf`, `report_1.pdf`, `report_2.pdf`, and so on without replacing one another. Optional document conversions are system-owned assets under `/mnt/user-data/.upload-conversions/<actual-upload-name>.md`. They are returned through the upload response but omitted from the primary upload listing. Deleting a primary removes only its exact generated asset and never infers that a user-uploaded sibling such as `uploads/report.md` is disposable.
The built-in `grep` tool searches either one text file or all matching text files below a directory, so an agent can search an uploaded document directly without first broadening the request to the entire uploads directory.
Image bytes loaded for a vision-model call are transient: DeerFlow removes the hidden base64 message after the model consumes it so later checkpoints do not keep duplicating that payload.
@ -990,9 +992,10 @@ This is the difference between a chatbot with tool access and an agent with an a
```
# Paths inside the sandbox container
/mnt/user-data/
├── uploads/ ← your files
├── workspace/ ← agents' working directory
└── outputs/ ← final deliverables
├── uploads/ ← your primary files
├── .upload-conversions/ ← generated Markdown (hidden from upload listings)
├── workspace/ ← agents' working directory
└── outputs/ ← final deliverables
```
### Agentic Browser Control

View File

@ -1460,8 +1460,9 @@ Multi-file upload with automatic document conversion:
- Rejects directory inputs before copying so uploads stay all-or-nothing
- Reuses one conversion worker per request when called from an active event loop
- Files stored in thread-isolated directories under the resolving user's bucket (`users/{user_id}/threads/{thread_id}/user-data/uploads`). For IM channels the owner is threaded explicitly via the `user_id=` kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from `get_effective_user_id()`
- Duplicate filenames in a single upload request are auto-renamed with `_N` suffixes so later files do not truncate earlier files
- Gateway HTTP uploads stage bytes as `.upload-*.part` files and atomically replace the destination only after size validation. These staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind.
- Every ingress stages a complete payload and atomically publishes it without replacing an existing entry. Collisions across requests, processes, HTTP, embedded, and IM adapters use `name.ext`, `name_1.ext`, `name_2.ext`; storage that cannot provide atomic no-replace publication fails explicitly.
- Gateway HTTP uploads use same-directory `.upload-*.part` staging files. Staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind.
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion removes only the selected primary and that exact generated asset; it never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling.
- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with `SandboxProvider.acquire_async()` and offload `read_bytes()` plus `sandbox.update_file()` together.
- Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate `sandbox.thread_data_mounts: true`; omission preserves backend auto-detection.
- Agent receives uploaded file list via `UploadsMiddleware`

View File

@ -620,10 +620,10 @@ Content-Type: multipart/form-data
"path": ".deer-flow/threads/abc123/user-data/uploads/document.pdf",
"virtual_path": "/mnt/user-data/uploads/document.pdf",
"artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/uploads/document.pdf",
"markdown_file": "document.md",
"markdown_path": ".deer-flow/threads/abc123/user-data/uploads/document.md",
"markdown_virtual_path": "/mnt/user-data/uploads/document.md",
"markdown_artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/uploads/document.md"
"markdown_file": "document.pdf.md",
"markdown_path": ".deer-flow/threads/abc123/user-data/.upload-conversions/document.pdf.md",
"markdown_virtual_path": "/mnt/user-data/.upload-conversions/document.pdf.md",
"markdown_artifact_url": "/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/document.pdf.md"
}
],
"message": "Successfully uploaded 1 file(s)"
@ -636,6 +636,8 @@ Content-Type: multipart/form-data
- Excel (`.xls`, `.xlsx`)
- Word (`.doc`, `.docx`)
All upload entry points publish complete payloads without replacing an existing name. Concurrent collisions are returned as `document.pdf`, `document_1.pdf`, `document_2.pdf`, and so on. Generated Markdown is stored outside the primary namespace and is not returned by the list endpoint. Deleting `document.pdf` also deletes only `.upload-conversions/document.pdf.md`; an independent `uploads/document.md` is preserved.
#### List Uploaded Files
```http

View File

@ -9,6 +9,8 @@ DeerFlow 后端提供了完整的文件上传功能,支持多文件上传,
- ✅ 支持多文件同时上传
- ✅ 可选地转换文档为 MarkdownPDF、PPT、Excel、Word
- ✅ 文件存储在线程隔离的目录中
- ✅ 跨请求、跨进程的同名文件不会互相覆盖
- ✅ 生成的 Markdown 与用户上传命名空间隔离
- ✅ Agent 自动感知当前消息中附带的文件
- ✅ 支持文件列表查询和删除
@ -35,10 +37,10 @@ POST /api/threads/{thread_id}/uploads
"path": ".deer-flow/threads/{thread_id}/user-data/uploads/document.pdf",
"virtual_path": "/mnt/user-data/uploads/document.pdf",
"artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.pdf",
"markdown_file": "document.md",
"markdown_path": ".deer-flow/threads/{thread_id}/user-data/uploads/document.md",
"markdown_virtual_path": "/mnt/user-data/uploads/document.md",
"markdown_artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.md"
"markdown_file": "document.pdf.md",
"markdown_path": ".deer-flow/threads/{thread_id}/user-data/.upload-conversions/document.pdf.md",
"markdown_virtual_path": "/mnt/user-data/.upload-conversions/document.pdf.md",
"markdown_artifact_url": "/api/threads/{thread_id}/artifacts/mnt/user-data/.upload-conversions/document.pdf.md"
}
],
"message": "Successfully uploaded 1 file(s)"
@ -50,6 +52,8 @@ POST /api/threads/{thread_id}/uploads
- `virtual_path`: Agent 在沙箱中使用的虚拟路径
- `artifact_url`: 前端通过 HTTP 访问文件的 URL
所有上传入口都先完整写入同目录暂存文件,再以“不替换已有条目”的原子操作发布。同名碰撞依次命名为 `document.pdf``document_1.pdf``document_2.pdf`;响应中的 `filename` 和各路径字段始终使用实际发布名。
### 2. 查询上传限制
```
GET /api/threads/{thread_id}/uploads/limits
@ -89,6 +93,8 @@ GET /api/threads/{thread_id}/uploads/list
}
```
列表只包含 `uploads/` 下的用户主文件;系统生成的 `.upload-conversions/` 资产不会出现在该接口中。
### 4. 删除文件
```
DELETE /api/threads/{thread_id}/uploads/{filename}
@ -102,6 +108,8 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
}
```
删除 `document.pdf` 时,只会额外删除它精确拥有的 `.upload-conversions/document.pdf.md`。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。
## 支持的文档格式
以下格式在显式启用 `uploads.auto_convert_documents: true` 时会自动转换为 Markdown
@ -110,7 +118,15 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
- Excel (`.xls`, `.xlsx`)
- Word (`.doc`, `.docx`)
转换后的 Markdown 文件会保存在同一目录下,文件名为原文件名 + `.md` 扩展名。
转换后的 Markdown 文件保存在系统拥有的 `.upload-conversions/` 目录中,文件名包含完整的实际主文件名。例如:
```text
Primary: /mnt/user-data/uploads/report.pdf
Generated: /mnt/user-data/.upload-conversions/report.pdf.md
Collision: report.pdf, report_1.pdf, report_2.pdf
Deletion: 删除 report.pdf 时只删除 .upload-conversions/report.pdf.md
/mnt/user-data/uploads/report.md 永远不会被推断为生成文件或自动删除。
```
默认情况下,自动转换是关闭的,以避免在网关主机上对不受信任的 Office/PDF 上传执行解析。只有在受信任部署中明确接受此风险时,才应将 `uploads.auto_convert_documents` 设置为 `true`
@ -149,13 +165,14 @@ Agent 在沙箱中运行使用虚拟路径访问文件。Agent 可以直接
read_file(path="/mnt/user-data/uploads/document.pdf")
# 读取转换后的 Markdown推荐
read_file(path="/mnt/user-data/uploads/document.md")
read_file(path="/mnt/user-data/.upload-conversions/document.pdf.md")
```
**路径映射关系:**
- Agent 使用:`/mnt/user-data/uploads/document.pdf`(虚拟路径)
- 实际存储:`backend/.deer-flow/threads/{thread_id}/user-data/uploads/document.pdf`
- 前端访问:`/api/threads/{thread_id}/artifacts/mnt/user-data/uploads/document.pdf`HTTP URL
- 转换结果:`/mnt/user-data/.upload-conversions/document.pdf.md`(以上传响应的 `markdown_virtual_path` 为准,不要自行推导)
上传流程采用“线程目录优先”策略:
- 先写入 `backend/.deer-flow/threads/{thread_id}/user-data/uploads/` 作为权威存储
@ -222,12 +239,13 @@ print(response.json())
backend/.deer-flow/threads/
└── {thread_id}/
└── user-data/
└── uploads/
├── document.pdf # 原始文件
├── document.md # 转换后的 Markdown
├── presentation.pptx
├── presentation.md
└── ...
├── uploads/
│ ├── document.pdf # 用户主文件
│ ├── document.md # 用户独立上传,绝不按名称推断归属
│ └── presentation.pptx
└── .upload-conversions/
├── document.pdf.md # document.pdf 的生成结果
└── presentation.pptx.md # presentation.pptx 的生成结果
```
## 限制
@ -243,7 +261,8 @@ backend/.deer-flow/threads/
1. **Upload Router** (`app/gateway/routers/uploads.py`)
- 处理文件上传、列表、删除请求
- 使用 markitdown 转换文档
- 流式写入暂存文件,并通过共享上传管理器原子发布
- 使用 markitdown 转换文档;生成文件发布到系统拥有的隔离目录
2. **Uploads Middleware** (`packages/harness/deerflow/agents/middlewares/uploads_middleware.py`)
- 读取当前消息的 `additional_kwargs.files`

View File

@ -102,18 +102,18 @@ async function uploadAndProcess(threadId: string, file: File) {
// path: ".deer-flow/threads/abc123/user-data/uploads/report.pdf",
// virtual_path: "/mnt/user-data/uploads/report.pdf",
// artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/uploads/report.pdf",
// markdown_file: "report.md",
// markdown_path: ".deer-flow/threads/abc123/user-data/uploads/report.md",
// markdown_virtual_path: "/mnt/user-data/uploads/report.md",
// markdown_artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/uploads/report.md"
// markdown_file: "report.pdf.md",
// markdown_path: ".deer-flow/threads/abc123/user-data/.upload-conversions/report.pdf.md",
// markdown_virtual_path: "/mnt/user-data/.upload-conversions/report.pdf.md",
// markdown_artifact_url: "/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/report.pdf.md"
// }
// 2. 发送消息给 Agent
await sendMessage(threadId, "请分析刚上传的 PDF 文件");
// Agent 会自动看到文件列表,包含
// Agent 的当前上传上下文包含主文件
// - report.pdf (虚拟路径: /mnt/user-data/uploads/report.pdf)
// - report.md (虚拟路径: /mnt/user-data/uploads/report.md)
// 转换结果必须使用上传响应返回的 markdown_virtual_path不要推导同目录文件名。
// 3. 前端可以直接访问转换后的 Markdown
const mdResponse = await fetch(fileInfo.markdown_artifact_url);
@ -135,6 +135,8 @@ async function uploadAndProcess(threadId: string, file: File) {
| 服务器后端代码直接访问 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` |
| Agent 工具调用 | `virtual_path` | `/mnt/user-data/uploads/file.pdf` |
| 前端下载/预览 | `artifact_url` | `/api/threads/abc123/artifacts/mnt/user-data/uploads/file.pdf` |
| Agent 读取生成 Markdown | `markdown_virtual_path` | `/mnt/user-data/.upload-conversions/file.pdf.md` |
| 前端读取生成 Markdown | `markdown_artifact_url` | `/api/threads/abc123/artifacts/mnt/user-data/.upload-conversions/file.pdf.md` |
| 备份脚本 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` |
| 日志记录 | `path` | `.deer-flow/threads/abc123/user-data/uploads/file.pdf` |
@ -172,10 +174,8 @@ async function listUploadedFiles(threadId) {
console.log(`下载: ${file.artifact_url}?download=true`);
console.log(`预览: ${file.artifact_url}`);
// 如果是文档,还有 Markdown 版本
if (file.markdown_artifact_url) {
console.log(`Markdown: ${file.markdown_artifact_url}`);
}
// 列表接口只返回主文件。markdown_* 字段仅在本次上传响应中返回,
// 调用方如需保留转换链接,应保存该响应元数据。
});
return data.files;
@ -204,7 +204,6 @@ interface UploadedFile {
artifact_url: string;
extension: string;
modified: number;
markdown_artifact_url?: string;
}
function FileUploadList({ threadId }: { threadId: string }) {
@ -254,9 +253,6 @@ function FileUploadList({ threadId }: { threadId: string }) {
<span>{file.filename}</span>
<a href={file.artifact_url} target="_blank">预览</a>
<a href={`${file.artifact_url}?download=true`}>下载</a>
{file.markdown_artifact_url && (
<a href={file.markdown_artifact_url} target="_blank">Markdown</a>
)}
<button onClick={() => handleDelete(file.filename)}>删除</button>
</li>
))}
@ -285,5 +281,8 @@ function FileUploadList({ threadId }: { threadId: string }) {
4. **Markdown 转换**
- 转换成功时,会返回额外的 `markdown_*` 字段
- 生成文件位于 `.upload-conversions/<完整主文件名>.md`,不会出现在主文件列表中
- 同名主文件按 `file.pdf``file_1.pdf``file_2.pdf` 原子发布,不会覆盖
- 删除主文件只删除其精确生成资产,不会删除用户上传的 `uploads/file.md`
- 建议优先使用 Markdown 版本(更易处理)
- 原始文件始终保留

View File

@ -30,6 +30,8 @@ Gateway (`app/gateway/routers/skills.py`, `uploads.py`) and Client (`deerflow/cl
**The same traversal check is written twice** — any security fix must be applied to both locations.
The final shared design below supersedes this historical baseline: both adapters now use the same atomic no-replace publisher and owned conversion layout.
## 2. Design Principles
### Dependency Direction
@ -91,11 +93,15 @@ class SkillAlreadyExistsError(ValueError)
get_uploads_dir(thread_id: str) -> Path # Pure path, no side effects
ensure_uploads_dir(thread_id: str) -> Path # Creates directory (for write paths)
# Filename safety
# Filename safety and atomic primary publication
normalize_filename(filename: str) -> str
# Path.name extraction + rejects ".." / "." / backslash / >255 bytes
deduplicate_filename(name: str, seen: set) -> str
# _N suffix increment for dedup, mutates seen in place
create_upload_staging_file(base_dir: Path) -> StagedUpload
publish_staged_upload(staged, preferred_filename) -> Path
publish_upload_bytes(base_dir, preferred_filename, data) -> Path
publish_upload_copy(base_dir, preferred_filename, source_path) -> Path
# Complete same-directory staging + atomic hard-link no-replace publication.
# Collisions retry name.ext, name_1.ext, name_2.ext across requests/processes.
# Path safety
validate_path_traversal(path: Path, base: Path) -> None
@ -107,7 +113,7 @@ list_files_in_dir(directory: Path) -> dict
# follow_symlinks=False to prevent metadata leakage
# Non-existent directory returns empty list
delete_file_safe(base_dir: Path, filename: str) -> dict
# Validates traversal first, then unlinks
# Deletes the primary and only its exact owned conversion
# URL helpers
upload_artifact_url(thread_id, filename) -> str # Percent-encoded for HTTP safety
@ -115,6 +121,8 @@ upload_virtual_path(filename) -> str # Sandbox-internal path
enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies sizes
```
`deerflow.uploads.layout` owns primary/conversion physical paths, sandbox virtual paths, and artifact URLs. Generated Markdown is published through `deerflow.uploads.conversion` at `user-data/.upload-conversions/<actual-primary-filename>.md`; it is outside the primary listing namespace and never inferred from an `uploads/<stem>.md` sibling.
## 4. Changes
### 4.1 Gateway Slimming
@ -127,8 +135,9 @@ enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies size
**`app/gateway/routers/uploads.py`**:
- Remove inline `get_uploads_dir` (replaced by `ensure_uploads_dir`/`get_uploads_dir`)
- `upload_files` uses `normalize_filename()` instead of inline safety checks
- Streamed bytes are completed in shared staging and `publish_staged_upload()` returns the actual collision-safe name
- `list_uploaded_files` uses `list_files_in_dir()` + enrichment
- `delete_uploaded_file` uses `delete_file_safe()` + companion markdown cleanup
- `delete_uploaded_file` uses `delete_file_safe()` for exact owned-conversion cleanup
### 4.2 Client Slimming
@ -136,7 +145,7 @@ enrich_file_listing(result, thread_id) -> dict # Adds URLs, stringifies size
- Remove `_get_uploads_dir` static method
- Remove ~50 lines of inline zip handling in `install_skill`
- `install_skill` delegates to `install_skill_from_archive()`
- `upload_files` uses `deduplicate_filename()` + `ensure_uploads_dir()`
- `upload_files` uses `publish_upload_copy()` + `ensure_uploads_dir()`
- `list_uploads` uses `get_uploads_dir()` + `list_files_in_dir()`
- `delete_upload` uses `get_uploads_dir()` + `delete_file_safe()`
- `update_mcp_config` / `update_skill` now reset `_agent_config_key = None`
@ -164,6 +173,8 @@ Read paths no longer have `mkdir` side effects — non-existent directories retu
| Listing symlink leak | `follow_symlinks=True` (default) | `follow_symlinks=False` |
| 409 status routing | `"already exists" in str(e)` | `SkillAlreadyExistsError` type match |
| Artifact URL encoding | Raw filename in URL | `urllib.parse.quote()` |
| Concurrent same-name writes | Scan/claim then replace | Atomic no-replace publication with `_N` retry |
| Generated Markdown ownership | Guessed `uploads/<stem>.md` sibling | Exact `.upload-conversions/<full-primary-name>.md` asset |
## 6. Alternatives Considered

View File

@ -597,7 +597,8 @@ You: "Deploying to staging..." [proceed]
**File Management:**
- Newly uploaded files in this run are listed in the `<current_uploads>` section before your first response
- Use `read_file` tool to read uploaded files using their paths from the list
- For PDF, PPT, Excel, and Word files, converted Markdown versions (*.md) are available alongside originals
- For PDF, PPT, Excel, and Word files, a generated Markdown version may be available under `/mnt/user-data/.upload-conversions/<full-upload-filename>.md`
- Use the exact generated path supplied in upload metadata or outlines and never infer an `uploads/<stem>.md` sibling
- Files uploaded in previous turns are NOT automatically listed. Use `list_uploaded_files` to discover them on demand it returns filenames, sizes, and optionally document outlines
- All temporary work happens in `/mnt/user-data/workspace`
- Treat `/mnt/user-data/workspace` as your default current working directory for coding and file-editing tasks

View File

@ -103,6 +103,8 @@ def test_apply_prompt_template_includes_relative_path_guidance(monkeypatch):
assert "Treat `/mnt/user-data/workspace` as your default current working directory" in prompt
assert "`hello.txt`, `../uploads/data.csv`, and `../outputs/report.md`" in prompt
assert "`/mnt/user-data/.upload-conversions/<full-upload-filename>.md`" in prompt
assert "never infer an `uploads/<stem>.md` sibling" in prompt
def test_apply_prompt_template_includes_memory_tool_guidance_only_in_tool_mode(monkeypatch):