mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-17 01:56:18 +00:00
docs: record audited upload adapters
This commit is contained in:
parent
d170c7e088
commit
1eed820213
@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
**Goal:** Make every DeerFlow upload ingress preserve same-name files under concurrent writes while isolating generated Markdown so conversion and deletion never overwrite or remove user uploads.
|
**Goal:** Make every DeerFlow upload ingress preserve same-name files under concurrent writes while isolating generated Markdown so conversion and deletion never overwrite or remove user uploads.
|
||||||
|
|
||||||
**Architecture:** A shared upload manager stages complete payloads and publishes them with an atomic hard-link no-replace primitive, returning the actual `_N` filename. A focused layout module maps each primary upload to a system-owned `.upload-conversions/<full-primary-name>.md` asset, and a conversion wrapper publishes that asset atomically. Gateway, embedded client, generic IM, and DingTalk become transport adapters over these shared primitives.
|
**Architecture:** A shared upload manager stages complete payloads and publishes them with an atomic hard-link no-replace primitive, returning the actual `_N` filename. A focused layout module maps each primary upload to a system-owned `.upload-conversions/<full-primary-name>.md` asset, and a conversion wrapper publishes that asset atomically. Gateway, embedded client, generic IM, Feishu, DingTalk, and WeChat download staging become transport adapters over these shared primitives.
|
||||||
|
|
||||||
**Tech Stack:** Python 3.12+, pathlib/os/tempfile/shutil, asyncio, FastAPI/Pydantic, pytest/pytest-asyncio, unittest.mock, Ruff.
|
**Tech Stack:** Python 3.12+, pathlib/os/tempfile/shutil, asyncio, FastAPI/Pydantic, pytest/pytest-asyncio, unittest.mock, Ruff.
|
||||||
|
|
||||||
@ -1135,11 +1135,11 @@ In DingTalk `_persist`, keep owner-scoped directory creation but replace the loc
|
|||||||
paths.ensure_thread_dirs(thread_id, user_id=effective_user_id)
|
paths.ensure_thread_dirs(thread_id, user_id=effective_user_id)
|
||||||
uploads_dir = paths.sandbox_uploads_dir(
|
uploads_dir = paths.sandbox_uploads_dir(
|
||||||
thread_id, user_id=effective_user_id
|
thread_id, user_id=effective_user_id
|
||||||
).resolve()
|
)
|
||||||
return publish_upload_bytes(uploads_dir, safe_filename, content)
|
return publish_upload_bytes(uploads_dir, safe_filename, content)
|
||||||
```
|
```
|
||||||
|
|
||||||
Remove `_file_write_lock` because the shared primitive supplies cross-thread and cross-process correctness. Keep the other DingTalk locks. Build the returned path through `upload_virtual_path(resolved_target.name)`.
|
Do not resolve the upload directory before publication: the shared publisher must inspect and reject a planted directory symlink itself. Remove `_file_write_lock` because the shared primitive supplies cross-thread and cross-process correctness. Keep the other DingTalk locks. Build the returned path through `upload_virtual_path(resolved_target.name)`.
|
||||||
|
|
||||||
- [ ] **Step 5: Refresh the channel blocking-I/O regression descriptions**
|
- [ ] **Step 5: Refresh the channel blocking-I/O regression descriptions**
|
||||||
|
|
||||||
|
|||||||
@ -1,8 +1,8 @@
|
|||||||
# Upload Collision Safety Design
|
# Upload Collision Safety Design
|
||||||
|
|
||||||
**Issue:** #3750
|
**Issue:** #3750
|
||||||
**Status:** Approved in conversation; awaiting review of this written specification
|
**Status:** Approved
|
||||||
**Scope:** Gateway uploads, embedded client uploads, inbound IM attachments, DingTalk attachments, generated Markdown conversions, outline lookup, and upload deletion
|
**Scope:** Gateway uploads, embedded client uploads, inbound IM attachments, generated Markdown conversions, outline lookup, and upload deletion
|
||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
@ -100,7 +100,8 @@ request-local `seen_filenames` are removed from adapters rather than retained as
|
|||||||
naming mechanism.
|
naming mechanism.
|
||||||
|
|
||||||
The publisher returns the actual filename chosen. Gateway, embedded client, generic IM,
|
The publisher returns the actual filename chosen. Gateway, embedded client, generic IM,
|
||||||
and DingTalk adapters all use it instead of implementing their own scan-then-write flow.
|
Feishu, DingTalk, and WeChat download staging all use it instead of implementing their
|
||||||
|
own scan-then-write flow.
|
||||||
|
|
||||||
## Conversion Publication and Ownership
|
## Conversion Publication and Ownership
|
||||||
|
|
||||||
@ -152,10 +153,17 @@ layout as the Gateway. Repeated calls cannot overwrite an earlier upload.
|
|||||||
Downloaded attachment bytes are staged and published by the shared publisher. Correctness
|
Downloaded attachment bytes are staged and published by the shared publisher. Correctness
|
||||||
does not depend on a pre-download directory scan, so parallel messages are safe.
|
does not depend on a pre-download directory scan, so parallel messages are safe.
|
||||||
|
|
||||||
### DingTalk
|
### Feishu and DingTalk
|
||||||
|
|
||||||
DingTalk uses the same publisher. Its in-process lock may remain for provider-specific
|
Both direct-to-thread adapters use the same publisher and preserve the unresolved upload
|
||||||
coordination, but it is not relied on for cross-process or cross-adapter safety.
|
directory path so the publisher can reject a planted directory symlink. No process-local
|
||||||
|
lock or pre-publication directory scan is relied on for correctness.
|
||||||
|
|
||||||
|
### WeChat
|
||||||
|
|
||||||
|
WeChat first downloads and decrypts inbound media into its channel state directory before
|
||||||
|
the generic IM ingestion step. This temporary materialization also uses the publisher, so
|
||||||
|
parallel messages cannot overwrite one another before the thread upload copy occurs.
|
||||||
|
|
||||||
### Sandbox synchronization
|
### Sandbox synchronization
|
||||||
|
|
||||||
@ -190,7 +198,7 @@ Tests are written before implementation and cover:
|
|||||||
deterministic unique names.
|
deterministic unique names.
|
||||||
2. Existing files and symlinks are never followed or replaced.
|
2. Existing files and symlinks are never followed or replaced.
|
||||||
3. Gateway uploads collide safely across separate and concurrent requests.
|
3. Gateway uploads collide safely across separate and concurrent requests.
|
||||||
4. Embedded, generic IM, and DingTalk ingress paths follow the same naming behavior.
|
4. Embedded, generic IM, Feishu, DingTalk, and WeChat ingress paths follow the same naming behavior.
|
||||||
5. Same-stem files with different extensions receive distinct conversion paths.
|
5. Same-stem files with different extensions receive distinct conversion paths.
|
||||||
6. A user-uploaded `report.md` is neither overwritten by converting `report.pdf` nor
|
6. A user-uploaded `report.md` is neither overwritten by converting `report.pdf` nor
|
||||||
removed when `report.pdf` is deleted.
|
removed when `report.pdf` is deleted.
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user