mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-20 19:50:35 +00:00
* fix(uploads): keep in-flight staging files out of the startup sweep `cleanup_stale_upload_staging_files` removed every `.upload-*.part` it found at Gateway startup. That assumed a single Gateway: nothing can be uploading while it boots, so every staging file is a crash leftover. With several replicas sharing one home volume (Helm RWX), a replica starting during a rolling update deleted the staging file of an upload another replica was still writing; that upload then failed at its `os.link` commit and lost its bytes. The sweep now skips staging files younger than `UPLOAD_STAGING_MIN_AGE` (24 hours, the same guard as the project-document `.staging` orphan sweep in `projects/trash.py`). Chunk writes refresh the mtime, so an upload in flight stays younger than the guard; an unreadable mtime keeps the file. The startup call site is unchanged and uses the default. Tests: the existing sweep tests backdate their orphans; new cases pin young-kept / old-removed, the configurable `min_age`, unreadable and vanished mtimes, and a router test that runs the sweep while an upload is blocked mid-write and asserts the upload still commits. * docs(changelog): link #6445 * fix(uploads): reclaim published staging aliases at any age Review follow-up (#6445): a Gateway that dies after the commit's `os.link` but before the staged-name removal leaves the final file and its `.upload-*.part` alias on one inode. With only the age guard, the next startup kept that alias for 24 hours, and because the sweep runs only at startup, `copy_upload_file_no_symlink` (embedded `DeerFlowClient.upload_files`) kept rejecting the destination for its multiple links. The sweep now stats each staging name with `os.lstat` (a `DirEntry` stat reports `st_nlink == 0` on Windows) and removes a name with `st_nlink > 1` at any age: a staged part only ever gains its second link through its own commit, so such a name is a published alias, never an upload in flight. Lone parts keep the 24 hour mtime guard. The destination hardlink check is unchanged. Tests cover the alias next to a young lone part, the replacement upload succeeding afterwards, and the crash reproduced through the real upload path with the staged-name removal patched out. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>