* fix(uploads): stop Windows replacement uploads from being skipped by the multi-link guard
NTFS never decrements st_nlink when one of a file's names is removed, so
every upload published through os.link(staging, dest) + unlink keeps
reading st_nlink == 2 on Windows. The non-exclusive replacement guards
then reject that destination forever, and DeerFlowClient.upload_files
turns the UnsafeUploadPathError into a log line - a replacement upload
is silently skipped (Closes#6820).
- Add _replacement_destination_is_hardlinked: POSIX-authoritative
st_nlink > 1, skipped on Windows, used only on the not exclusive
replacement path. The exclusive (first-claim) path keeps the strict
count check, as does the POSIX O_NOFOLLOW branch.
- Decide "published alias" in cleanup_stale_upload_staging_files by
inode identity against siblings (st_dev/st_ino via os.lstat) instead
of the link count; DirEntry.stat reports zeros for both on Windows,
matching the existing _staging_entry_stat note.
- Guard the POSIX-only st_nlink assertions in the affected tests and
mark the three inbound-attachment mode-bit tests POSIX-only, mirroring
the existing skipif in the same class.
* fix(uploads): restore the exclusive post-open link guard and harden the staging sweep
Address the three review findings on 5353715:
- Restore the strict post-open st_nlink != 1 check for exclusive=True
on the no-O_NOFOLLOW (Windows fallback) branch, which the previous
commit dropped there against what the PR description claimed. Only
the non-exclusive replacement path defers to the Windows-aware guard.
- Never match a degenerate st_ino == 0 in the sibling-inode sweep: on
FAT/exFAT or some SMB volumes every entry would look like a published
alias and in-flight uploads would be deleted at any age. Such volumes
fall back to the min_age guard only.
- Build one {name: stat} map per uploads directory before the entry
loop so the sweep scans each directory once instead of rescanning it
for every staging file (O(N) instead of O(N^2) lstat calls).
Tests: 407 passed / 27 skipped across test_uploads_manager.py,
test_uploads_router.py, test_upload_copy_publication.py,
test_upload_windows_staging_aliases.py, test_channel_file_attachments.py
(Windows 11, Python 3.13 venv). The three new regression tests fail on
5353715 and pass with this change.
* style(uploads): apply Ruff formatting
* fix(uploads): report staging directory scan failures
---------
Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): convert the captured upload source
* test(uploads): cover ACL sources and scope capture failures
* fix(uploads): preserve concurrent uploads from one captured source
Combine the embedded client's source capture with no-replace publication for
first-time filenames and Markdown companions. Hold one private capture across
collision retries, retain original descriptor metadata, and keep replacement
behavior for names present when the call starts.
Port concurrent-upload and staging-publication regressions from PR #6687.
Extend source-edit/removal and real XLSX cases to both publication paths, and
cover edits or removal during two filename collisions in ordinary and event-loop
calls. Document capture, retry, publication, and cleanup contracts.
Validation: related suites 469 passed, 24 skipped; full offline suite 28,788
passed, 488 skipped; blocking-I/O suite 245 passed, 1 skipped; Ruff, guidance,
and diff checks passed. Nine race/collision cases failed on the #6632 baseline.
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* docs: keep agent guidance within CI budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* fix(uploads): skip files that vanish mid-scan in list_files_in_dir
The listing iterates scandir entries and stats each survivor. A
concurrent upload delete or the staging sweep can remove a file in
between, and the resulting FileNotFoundError escapes as a 500 on the
listing endpoint. is_file() does not cover this: on POSIX it answers
from the scandir d_type without touching the filesystem again.
Catch OSError around the stat and skip the entry on ENOENT/ENOTDIR/
ELOOP, the same tolerance the chmod path just above already applies to
replacement races. Operational errors such as EACCES still propagate.
Regression tests pin both directions: a vanished entry is skipped while
the rest of the listing survives, and a permission error still raises.
Fixes#6097
* fix(uploads): log vanished entries and fix comment per review
Address willem-bd's review on #6533: the skip now emits a debug line so
"my file disappeared from the list" reports are diagnosable without
changing the response contract, and the race comment names the actual
concurrent removers (the DELETE endpoint, possibly from another
replica, and sandbox processes) instead of the staging sweep, which
cannot remove a listed non-staging file.
The directory-level race between the is_dir() guard and os.scandir()
stays out of scope; it is covered by the still-open #6096 and its PR.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(uploads): tolerate directory races during listing
* docs(uploads): scope listing guidance to avoid inherited size limits
* fix(uploads): log directory races during listing
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): keep in-flight staging files out of the startup sweep
`cleanup_stale_upload_staging_files` removed every `.upload-*.part` it
found at Gateway startup. That assumed a single Gateway: nothing can be
uploading while it boots, so every staging file is a crash leftover.
With several replicas sharing one home volume (Helm RWX), a replica
starting during a rolling update deleted the staging file of an upload
another replica was still writing; that upload then failed at its
`os.link` commit and lost its bytes.
The sweep now skips staging files younger than `UPLOAD_STAGING_MIN_AGE`
(24 hours, the same guard as the project-document `.staging` orphan
sweep in `projects/trash.py`). Chunk writes refresh the mtime, so an
upload in flight stays younger than the guard; an unreadable mtime keeps
the file. The startup call site is unchanged and uses the default.
Tests: the existing sweep tests backdate their orphans; new cases pin
young-kept / old-removed, the configurable `min_age`, unreadable and
vanished mtimes, and a router test that runs the sweep while an upload
is blocked mid-write and asserts the upload still commits.
* docs(changelog): link #6445
* fix(uploads): reclaim published staging aliases at any age
Review follow-up (#6445): a Gateway that dies after the commit's
`os.link` but before the staged-name removal leaves the final file and
its `.upload-*.part` alias on one inode. With only the age guard, the
next startup kept that alias for 24 hours, and because the sweep runs
only at startup, `copy_upload_file_no_symlink` (embedded
`DeerFlowClient.upload_files`) kept rejecting the destination for its
multiple links.
The sweep now stats each staging name with `os.lstat` (a `DirEntry`
stat reports `st_nlink == 0` on Windows) and removes a name with
`st_nlink > 1` at any age: a staged part only ever gains its second
link through its own commit, so such a name is a published alias, never
an upload in flight. Lone parts keep the 24 hour mtime guard. The
destination hardlink check is unchanged.
Tests cover the alias next to a young lone part, the replacement upload
succeeding afterwards, and the crash reproduced through the real upload
path with the staged-name removal patched out.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): reject filenames reserved for temporary staging
Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace.
* fix(uploads): report reserved names before HTTP batch ingestion
Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation.
* fix(uploads): reject Windows-style staging basenames on POSIX
Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation.
Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(paths): reject Windows reserved names on host-visible paths
Uploads, skill support files, and sandbox path checks now share one helper that rejects CON, PRN, AUX, NUL, COM1-COM9, and LPT1-LPT9 (including names like con.txt) and components that end in a dot or space. The check is not limited to Windows so a Linux host cannot store a name that breaks when the same tree is used on Windows.
* fix(paths): validate quoted local bash paths as whole words
The absolute-path scan stopped at whitespace, so a quoted upload name such as report. final.txt or CON notes.txt was rejected as a truncated segment. Local bash checks now use the shlex argument for traversal and Windows-incompatible segments.
* fix(paths): do not block reads of existing host names
* style(paths): format with ruff 0.15.12
* fix(channels): fix Telegram inbound file download and sandbox readability
- Shut down the download Bot on the Telegram loop (Bot.shutdown(), which
closes its HTTPX clients) before the loop stops, instead of the
non-existent Bot.session.close().
- Grant group/other read on channel-downloaded uploads so the non-root
AIO/Docker sandbox process can read the root-written 0o600 file.
- Apply the sandbox permission change with os.fchmod on a descriptor
opened with O_NOFOLLOW (validated as a regular file via fstat), bound
to the validated upload inode, so a symlink swapped in after lstat
cannot redirect the chmod to a target outside the uploads directory.
The open also uses O_NONBLOCK so a sandbox-swapped FIFO cannot block the
read-only open before the regular-file check (matching the existing
open_upload_file_no_symlink convention).
Centralized in a shared apply_upload_sandbox_permits helper reused by
the channel inbound path and the HTTP upload readable/writable helpers.
- Surface the download failure cause chain in logs with the Bot API URL
masked: the configured token is redacted and both URL forms are
collapsed, covering the file download URL (/file/bot<token>/...) and
the method URLs (/bot<token>/getMe, /bot<token>/getFile).
- Migrate the existing receive_file tests onto the download Bot and add
coverage for _get_download_bot (loop-bound creation + caching, cleanup
on init failure), download-bot routing over the application bot,
real-Bot shutdown closing both HTTPX clients, receive_file
timeout-containment, masked cause-chain logging (file and method URL
forms), and inbound-file sandbox perms (including the swap-after-lstat
symlink regression).
* fix(uploads): surface sandbox permission failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): stop deleting a converted companion we cannot prove we wrote
Conversion names a document's Markdown companion after the document's
stem and falls back to a _N suffix when that name is already taken, so
the .md beside a document may belong to another document sharing the
stem, or to the user. Delete removed it anyway: uploading a.docx and
a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's
companion while orphaning a.pdf's own.
Delete now removes only the file it was asked to remove. The companion
stays listed and can be deleted on its own. Orphans are the cost of not
guessing; issue #5672 covers giving companions a provable owner, which
is what a safe cleanup needs, along with the two related readers that
still guess (the outline injected for a document and the agent's file
listing).
convertible_extensions loses its last use and is dropped from the
signature and both call sites. The gateway router keeps importing
CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in
__all__, where that module documents its re-exports.
* docs(changelog): note that delete keeps the converted markdown (#5673)
* fix(client): stop embedded uploads from writing through symlinks
DeerFlowClient.upload_files copied each file with shutil.copy2 and let
convert_file_to_markdown write the companion straight into the uploads
directory. Local and AIO sandboxes can write to that directory, so a
symlink planted at an upload name or at the companion's name was
followed: the upload's bytes and the converted Markdown landed in
whatever host file the link pointed to, and the call reported success.
The Gateway refuses symlinked destinations and the IM channels write
through write_upload_file_no_symlink; the embedded client never adopted
either.
Uploads now go through copy_upload_file_no_symlink, a new helper next
to write_upload_file_no_symlink. It keeps copy2's content, permission
bits and timestamps, so files stay readable to Docker sandboxes, but
applies them to the descriptor opened with O_NOFOLLOW and opens the
source first so a missing source cannot truncate an existing upload.
As in the Gateway, a file with an unsafe destination is skipped and
listed in skipped_files, success turns false, and the message says how
many were skipped. The companion is converted inside a private temporary
directory and then written with write_upload_file_no_symlink; one whose
name is unsafe is left out like a failed conversion, and the original
upload is kept.
* docs(changelog): note embedded upload symlink fix (#5578)
* fix(client): keep copy2's same-file guard and companion permissions
Review follow-up. Two regressions in the previous commit.
copy_upload_file_no_symlink opened the destination before comparing it
with the source, and that open truncates. Passing a file that already
sits in the thread's uploads directory therefore copied an emptied file
over itself: the upload reported success with size 0 and the original
bytes were gone, where copy2 raised SameFileError and left the file
alone. The destination is now compared with the source through
os.path.samestat before anything is opened, so identity — including a
hardlink or another spelling of the same path — raises SameFileError as
before.
The Markdown companion was published with write_upload_file_no_symlink,
which creates a new file as 0600 and ignores the converted file's mode.
Under umask 022 the companion became 0600 while its own document stayed
0644, so a bind-mounted sandbox running as another uid could read the
upload but not the Markdown the response advertises. It now goes
through the same copy helper as the upload, which preserves the
converter's permission bits.
* fix(uploads): delete the requested upload, not a symlink's target
delete_file_safe resolved the requested path before unlinking it. The
uploads directory is writable from local and AIO sandboxes, so a
symlink planted under an upload name was followed: deleting alias.pdf
removed the victim.pdf it pointed to, and the companion cleanup then
removed victim.md, while the link itself survived and the call reported
"Deleted alias.pdf". A link resolving outside the directory was already
refused by the traversal check, so the damage stayed inside the
thread's uploads.
The function now checks and unlinks the requested entry itself and
treats a symlink as not found, the same way list_files_in_dir already
hides it. unlink() never follows the final component, so a file swapped
for a link between the check and the unlink removes only the link.
Tests cover the helper, the Gateway DELETE route, and
DeerFlowClient.delete_upload.
* docs(changelog): note upload delete symlink fix (#5547)
normalize_filename accepts names up to 255 UTF-8 bytes, but
claim_unique_filename appended _N to the stem without re-checking the
budget. A duplicate at maximum length therefore produced a 257-byte
name, and the write path (open_upload_file_no_symlink ->
normalize_filename) rejected it with ValueError. In the Gateway upload
route that error falls into the generic handler: the whole request
fails with a 500 and files already written in the same batch are rolled
back — including unrelated ones. The same helper backs the Feishu and
DingTalk channel downloads and client-side attachment staging.
Truncate the stem on a UTF-8 code-point boundary when appending the
dedupe tag would exceed 255 bytes, so the result always round-trips
through normalize_filename. Names short enough to fit keep the exact
dedupe shape they had before.
Tests: red on main, green here —
- unit: max-length dedupe stays within the limit and round-trips;
repeated collisions stay unique; multibyte stems truncate on a
code-point boundary; short names keep the historical _N shape
- router: a batch with a max-length duplicate now succeeds and keeps
every file instead of failing with a 500
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
* Fix Windows backend test compatibility
* Preserve ACP path style on Windows
* Fix installer import ordering
* Address review comments for Windows fixes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: extract shared skill installer and upload manager to harness
Move duplicated business logic from Gateway routers and Client into
shared harness modules, eliminating code duplication.
New shared modules:
- deerflow.skills.installer: 6 functions (zip security, extraction, install)
- deerflow.uploads.manager: 7 functions (normalize, deduplicate, validate,
list, delete, get_uploads_dir, ensure_uploads_dir)
Key improvements:
- SkillAlreadyExistsError replaces stringly-typed 409 status routing
- normalize_filename rejects backslash-containing filenames
- Read paths (list/delete) no longer mkdir via get_uploads_dir
- Write paths use ensure_uploads_dir for explicit directory creation
- list_files_in_dir does stat inside scandir context (no re-stat)
- install_skill_from_archive uses single is_file() check (one syscall)
- Fix agent config key not reset on update_mcp_config/update_skill
Tests: 42 new (22 installer + 20 upload manager) + client hardening
* refactor: centralize upload URL construction and clean up installer
- Extract upload_virtual_path(), upload_artifact_url(), enrich_file_listing()
into shared manager.py, eliminating 6 duplicated URL constructions across
Gateway router and Client
- Derive all upload URLs from VIRTUAL_PATH_PREFIX constant instead of
hardcoded "mnt/user-data/uploads" strings
- Eliminate TOCTOU pre-checks and double file read in installer — single
ZipFile() open with exception handling replaces is_file() + is_zipfile()
+ ZipFile() sequence
- Add missing re-exports: ensure_uploads_dir in uploads/__init__.py,
SkillAlreadyExistsError in skills/__init__.py
- Remove redundant .lower() on already-lowercase CONVERTIBLE_EXTENSIONS
- Hoist sandbox_uploads_dir(thread_id) before loop in uploads router
* fix: add input validation for thread_id and filename length
- Reject thread_id containing unsafe filesystem characters (only allow
alphanumeric, hyphens, underscores, dots) — prevents 500 on inputs
like <script> or shell metacharacters
- Reject filenames longer than 255 bytes (OS limit) in normalize_filename
- Gateway upload router maps ValueError to 400 for invalid thread_id
* fix: address PR review — symlink safety, input validation coverage, error ordering
- list_files_in_dir: use follow_symlinks=False to prevent symlink metadata
leakage; check is_dir() instead of exists() for non-directory paths
- install_skill_from_archive: restore is_file() pre-check before extension
validation so error messages match the documented exception contract
- validate_thread_id: move from ensure_uploads_dir to get_uploads_dir so
all entry points (upload/list/delete) are protected
- delete_uploaded_file: catch ValueError from thread_id validation (was 500)
- requires_llm marker: also skip when OPENAI_API_KEY is unset
- e2e fixture: update TitleMiddleware exclusion comment (kept filtering —
middleware triggers extra LLM calls that add non-determinism to tests)
* chore: revert uv.lock to main — no dependency changes in this PR
* fix: use monkeypatch for global config in e2e fixture to prevent test pollution
The e2e_env fixture was calling set_title_config() and
set_summarization_config() directly, which mutated global singletons
without automatic cleanup. When pytest ran test_client_e2e.py before
test_title_middleware_core_logic.py, the leaked enabled=False caused
5 title tests to fail in CI.
Switched to monkeypatch.setattr on the module-level private variables
so pytest restores the originals after each test.
* fix: address code review — URL encoding, API consistency, test isolation
- upload_artifact_url: percent-encode filename to handle spaces/#/?
- deduplicate_filename: mutate seen set in place (caller no longer
needs manual .add() — less error-prone API)
- list_files_in_dir: document that size is int, enrich stringifies
- e2e fixture: monkeypatch _app_config instead of set_app_config()
to prevent global singleton pollution (same pattern as title/summarization fix)
- _make_e2e_config: read LLM connection details from env vars so
external contributors can override defaults
- Update tests to match new deduplicate_filename contract
* docs: rewrite RFC in English and add alternatives/breaking changes sections
* fix: address code review feedback on PR #1202
- Rename deduplicate_filename to claim_unique_filename to make
the in-place set mutation explicit in the function name
- Replace PermissionError with PathTraversalError(ValueError) for
path traversal detection — malformed input is 400, not 403
* fix: set _app_config_is_custom in e2e test fixture to prevent config.yaml lookup in CI
---------
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
