mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-20 11:20:35 +00:00
* fix(uploads): stop Windows replacement uploads from being skipped by the multi-link guard NTFS never decrements st_nlink when one of a file's names is removed, so every upload published through os.link(staging, dest) + unlink keeps reading st_nlink == 2 on Windows. The non-exclusive replacement guards then reject that destination forever, and DeerFlowClient.upload_files turns the UnsafeUploadPathError into a log line - a replacement upload is silently skipped (Closes #6820). - Add _replacement_destination_is_hardlinked: POSIX-authoritative st_nlink > 1, skipped on Windows, used only on the not exclusive replacement path. The exclusive (first-claim) path keeps the strict count check, as does the POSIX O_NOFOLLOW branch. - Decide "published alias" in cleanup_stale_upload_staging_files by inode identity against siblings (st_dev/st_ino via os.lstat) instead of the link count; DirEntry.stat reports zeros for both on Windows, matching the existing _staging_entry_stat note. - Guard the POSIX-only st_nlink assertions in the affected tests and mark the three inbound-attachment mode-bit tests POSIX-only, mirroring the existing skipif in the same class. * fix(uploads): restore the exclusive post-open link guard and harden the staging sweep Address the three review findings on 5353715: - Restore the strict post-open st_nlink != 1 check for exclusive=True on the no-O_NOFOLLOW (Windows fallback) branch, which the previous commit dropped there against what the PR description claimed. Only the non-exclusive replacement path defers to the Windows-aware guard. - Never match a degenerate st_ino == 0 in the sibling-inode sweep: on FAT/exFAT or some SMB volumes every entry would look like a published alias and in-flight uploads would be deleted at any age. Such volumes fall back to the min_age guard only. - Build one {name: stat} map per uploads directory before the entry loop so the sweep scans each directory once instead of rescanning it for every staging file (O(N) instead of O(N^2) lstat calls). Tests: 407 passed / 27 skipped across test_uploads_manager.py, test_uploads_router.py, test_upload_copy_publication.py, test_upload_windows_staging_aliases.py, test_channel_file_attachments.py (Windows 11, Python 3.13 venv). The three new regression tests fail on 5353715 and pass with this change. * style(uploads): apply Ruff formatting * fix(uploads): report staging directory scan failures --------- Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
34 lines
2.0 KiB
Markdown
34 lines
2.0 KiB
Markdown
# Uploads
|
|
|
|
`manager.py::list_files_in_dir` returns a best-effort snapshot under concurrent
|
|
cleanup. Entry stat races (`ENOENT`, `ENOTDIR`, `ELOOP`) are skipped with debug
|
|
logging. Directory removal or replacement by a non-directory during scanning
|
|
returns entries already collected and logs the directory path and retained
|
|
entry count at debug level. Permission and other operational errors propagate.
|
|
|
|
Keep both entry and directory race coverage in
|
|
`backend/tests/test_uploads_manager.py`.
|
|
|
|
Staging cleanup is best-effort: warn with the directory and exception on scan
|
|
errors, keep its files and continue elsewhere. A vanished directory stays quiet.
|
|
|
|
Embedded first-time uploads use `publish_upload_copy` for suffix retries from the
|
|
normalized original name, for originals and companions. With `snapshot_path`,
|
|
capture the opened source once before retries and retain its original `fstat`
|
|
metadata; every attempt reads the held private capture. Replacement copies share
|
|
this capture helper. Never copy snapshot permission bits onto the saved upload.
|
|
Close the caller's descriptor after capture and finish conversion workers before
|
|
removing captures. Never reopen the caller/sandbox upload path for conversion.
|
|
Names already present in the call's initial regular-file snapshot retain
|
|
replacement behavior; these replacements are not serialized. Apply source
|
|
timestamps after new-file publication so cleanup cannot sweep an in-flight copy
|
|
of an old source. New final names must expose complete copies.
|
|
|
|
No-replace publication binds pre-link staging and post-link destination checks to
|
|
the open staging descriptor. Use a no-follow hard link where supported; fallback
|
|
identity checks narrow but do not eliminate the link-time race. Reject substituted
|
|
regular files and symlinks, and remove a mismatched publication before reporting
|
|
failure. Cleanup errors are logged without masking success or the original error.
|
|
Keep publication/fallback coverage in `backend/tests/test_upload_copy_publication.py`
|
|
and capture-through-collision cases in `backend/tests/test_client_upload_snapshot.py`.
|