Nan Gao a83b244a8c
fix(uploads): keep in-flight staging files out of the startup sweep (#6445)
* fix(uploads): keep in-flight staging files out of the startup sweep

`cleanup_stale_upload_staging_files` removed every `.upload-*.part` it
found at Gateway startup. That assumed a single Gateway: nothing can be
uploading while it boots, so every staging file is a crash leftover.
With several replicas sharing one home volume (Helm RWX), a replica
starting during a rolling update deleted the staging file of an upload
another replica was still writing; that upload then failed at its
`os.link` commit and lost its bytes.

The sweep now skips staging files younger than `UPLOAD_STAGING_MIN_AGE`
(24 hours, the same guard as the project-document `.staging` orphan
sweep in `projects/trash.py`). Chunk writes refresh the mtime, so an
upload in flight stays younger than the guard; an unreadable mtime keeps
the file. The startup call site is unchanged and uses the default.

Tests: the existing sweep tests backdate their orphans; new cases pin
young-kept / old-removed, the configurable `min_age`, unreadable and
vanished mtimes, and a router test that runs the sweep while an upload
is blocked mid-write and asserts the upload still commits.

* docs(changelog): link #6445

* fix(uploads): reclaim published staging aliases at any age

Review follow-up (#6445): a Gateway that dies after the commit's
`os.link` but before the staged-name removal leaves the final file and
its `.upload-*.part` alias on one inode. With only the age guard, the
next startup kept that alias for 24 hours, and because the sweep runs
only at startup, `copy_upload_file_no_symlink` (embedded
`DeerFlowClient.upload_files`) kept rejecting the destination for its
multiple links.

The sweep now stats each staging name with `os.lstat` (a `DirEntry`
stat reports `st_nlink == 0` on Windows) and removes a name with
`st_nlink > 1` at any age: a staged part only ever gains its second
link through its own commit, so such a name is a published alias, never
an upload in flight. Lone parts keep the 24 hour mtime guard. The
destination hardlink check is unchanged.

Tests cover the alias next to a young lone part, the replacement upload
succeeding afterwards, and the crash reproduced through the real upload
path with the staged-name removal patched out.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 07:45:29 +08:00
..