mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-14 16:08:41 +00:00
fix: harden upload deletion recovery retries
This commit is contained in:
parent
e579c95356
commit
4f5dcd61dc
@ -1464,7 +1464,7 @@ Multi-file upload with automatic document conversion:
|
||||
- Exact-name generation leases use a portable NFC-plus-casefold coordination key with Win32 trailing-dot/space folding, so filesystem aliases cannot bypass an active generation; filenames that Windows cannot represent losslessly are rejected before staging, while exact legacy POSIX basenames remain deletable after upgrade. Legacy deletion has a POSIX-only lease mode for literal backslashes and components made entirely from dots/spaces; it does not weaken new-upload validation. The original filename remains the published name. Publication tries each candidate lease without blocking and treats a busy canonical key as a collision, so same-batch and inverse concurrent batches advance to a UTF-8-bounded `_N` candidate instead of deadlocking while retaining earlier generations; pathological long suffixes fall back to truncating the complete basename. Deletion rejects an inode that moved outside the requested name's lease. Primary deletion and rollback move the selected primary and its exact generated conversion into one inode-and-intent-tagged transaction directory under the protected conversion namespace before side effects; the primary lives in a dedicated child directory so legal filenames such as `.commit`, `.restore`, and `.conversion` cannot collide with transaction controls. User deletion starts with restore-on-crash intent. Immediately before the first possible remote mutation, a persisted commit marker makes crash recovery discard the authoritative host tombstones; a live, fully compensated remote failure clears it while restoring both host entries. A live rollback writes a restore marker before publishing either hard link, so recovery can finish restoring both entries even when the transaction began with discard intent. A partial remote failure is compensated from the staged authoritative bytes before host restoration; if compensation fails, every remote target is driven back toward the idempotent deletion outcome and host deletion commits instead of publishing a split generation. Failed-upload rollback tombstones otherwise use discard recovery, while legacy primary-only transactions retain their recorded restore/discard behavior. Neither transaction kind is treated as an ordinary disposable upload stage. Final lease release is the commit point: cancellation newly arriving during release is delayed and swallowed so a committed upload is returned as success rather than an indeterminate cancelled result. Embedded-client finalization attempts every lease release and conversion-pool shutdown independently, logging cleanup failures rather than changing an already-committed response or stranding later leases.
|
||||
- Filenames containing NUL, `<`, `>`, or reserved model-context boundary markers are rejected before staging so accepted filenames and exact virtual paths remain lossless in model-visible upload context. Legacy files discovered on disk are still neutralized when listed.
|
||||
- Gateway HTTP uploads use same-directory `.upload-*.part` staging files. Each active stage holds a cross-process liveness lock under `.upload-conversions/.locks/stages/`; startup cleanup skips held stages and sweeps only crash-orphaned files, while `.upload-delete-<restore|discard>-<inode>-<nonce>.part/` transaction directories are resolved according to their persisted crash intent. Legacy intent-less deletion transactions retain restore behavior. Cancellation during staging creation drains the worker and aborts the returned stage before propagating. Staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools.
|
||||
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. Tombstone staging first fsyncs the lazily created namespace in `user-data`, then each rename destination before its source directory, and completes the whole destination tree plus both rename sides before remote mutation. Existing commit/restore markers are re-fsynced in the transaction directory before their phase is reused. Committed cleanup fsyncs every tombstone or recovered visible-alias removal before clearing the transaction-root commit marker. When recovery sees an empty nested primary directory, it re-fsyncs that directory before clearing either commit or restore phase intent. Compensated restore first fsyncs the visible primary and conversion peers, then their tombstone removals, before clearing restore intent. Thus recovery cannot resurrect a remotely deleted generation or lose a compensated host generation. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; journal-abort errors cannot skip that host restore. Failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. The v3 journal binds the provider type, backend-issued namespace, immutable sandbox incarnation, and raw sandbox ID. Journal finalization first persists a `.remote-delete.finalizing` guard; if journal-unlink directory fsync and journal restoration both fail, the guard remains a cross-process basename reservation until startup cleanup or a later deletion preparation fsyncs the transaction directory to make the observed journal absence durable. Gateway startup and later embedded deletion preparation may use only the provider's non-creating exact reconnect/confirmed-absence result—never `acquire()`, a replacement sandbox for the thread, or a different provider/backend/incarnation—before ordinary transaction cleanup removes the tombstones; an unknown lookup stays pending. AIO local Docker uses daemon/container IDs, while the remote Provisioner uses Kubernetes Namespace/Pod UIDs and treats Pod presence as authoritative even when its Service is gone. Providers without an explicit restart-safe reconciliation identity fail deletion preparation before host staging or remote mutation. Live journals remain protected by their stage lease, reconciliation takes the same name lease used by publication, and a disappearing unrelated transaction is treated as completed after the scanner proves it does not reserve the candidate. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
|
||||
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. Tombstone staging first fsyncs the lazily created namespace in `user-data`, then each rename destination before its source directory, and completes the whole destination tree plus both rename sides before remote mutation. Existing commit/restore markers are re-fsynced as exact files and then in the transaction directory before their phase is reused. Committed cleanup fsyncs every tombstone or recovered visible-alias removal before clearing the transaction-root commit marker. When recovery sees an empty nested primary directory, it re-fsyncs that directory before clearing either commit or restore phase intent. Compensated restore first fsyncs the visible primary and conversion peers, then their tombstone removals, before clearing restore intent. After live recovery created a hard link for an unexpected inode, startup cleans the hidden alias only when there is exactly one verified same-inode visible peer; a sole mismatched tombstone remains fail-closed against transaction tampering. This makes the verified live-retry state idempotent without overwriting or deleting the visible replacement. Thus recovery cannot resurrect a remotely deleted generation or lose a compensated host generation. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; journal-abort errors cannot skip that host restore. Failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. The v3 journal binds the provider type, backend-issued namespace, immutable sandbox incarnation, and raw sandbox ID. Journal finalization first persists a `.remote-delete.finalizing` guard; if journal-unlink directory fsync and journal restoration both fail, the guard remains a cross-process basename reservation until startup cleanup or a later deletion preparation fsyncs the transaction directory to make the observed journal absence durable. Gateway startup and later embedded deletion preparation may use only the provider's non-creating exact reconnect/confirmed-absence result—never `acquire()`, a replacement sandbox for the thread, or a different provider/backend/incarnation—before ordinary transaction cleanup removes the tombstones; an unknown lookup stays pending. AIO local Docker uses daemon/container IDs, while the remote Provisioner uses Kubernetes Namespace/Pod UIDs and treats Pod presence as authoritative even when its Service is gone. Providers without an explicit restart-safe reconciliation identity fail deletion preparation before host staging or remote mutation. Live journals remain protected by their stage lease, reconciliation takes the same name lease used by publication, and a disappearing unrelated transaction is treated as completed after the scanner proves it does not reserve the candidate. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
|
||||
- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor; only operations that may block waiting for a name lease use the separate lease-wait pool. Work needed by an existing lease holder and non-blocking publication stays on the general pool, so waiters cannot starve conversion, rollback, or release. Cold sandbox-provider construction is also offloaded. Gateway, embedded-client, and IM ingresses share provider-aware publication: mounted providers make the exact host paths sandbox-readable; non-mounted providers acquire the sandbox and synchronize the primary plus generated conversion to their exact virtual paths. Each ingress records attempted remote paths before the write can commit and, on later failure or cancellation, calls the provider-neutral `Sandbox.remove_file()` for those paths before host rollback and lease release; the command fallback requires a per-call unpredictable exact success trailer. Embedded multi-file calls retain every publication and receipt until the whole response is built, then roll back the complete batch on failure. WeChat download publication uses the cancellation-safe async lease adapter, so cancellation drains and rolls back a publication worker that completes late.
|
||||
- Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate `sandbox.thread_data_mounts: true` plus a Provisioner advertising a compatible mount contract; every upload/delete/artifact decision waits for any in-flight stale capability probe in a worker thread and uses one atomic snapshot. The same frozen version selects the deterministic sandbox ID, mount set, cache/reuse candidate, create precondition, and response validation; if the peer changes versions during acquisition, the whole attempt restarts with a new ID. The required version is sent in the create JSON, and the Provisioner rejects a mismatch before any Kubernetes read or write, preventing a rolling upgrade from creating an orphan under the prior version's ID. After acquisition, consumers use the immutable contract recorded on that sandbox instance rather than the provider's mutable global capability. Unavailable or legacy results use explicit synchronization and are retried without a Gateway restart. A confirmed legacy peer may create thread sandboxes only for the `default` no-auth user, because its primary user-data mount cannot isolate equal thread IDs across authenticated users; authenticated rolling deployments upgrade the Provisioner first. The Provisioner validates the request before its idempotent fast path, labels Pods with hashed user/thread identity and contract version, stores the exact identity plus a mount-signature annotation, and compares the live Pod specification before reuse. Every remote acquisition path, including active-cache and warm-pool reuse, deliberately replays the idempotent create request instead of trusting GET discovery or a liveness response. Contracts newer than v2 are accepted as forward-compatible, but the response must echo the exact frozen version, user, and thread; legacy responses may omit the version or return exact integer zero, never strings or booleans. The negotiated mount-contract version namespaces deterministic AIO sandbox IDs so a container created under another version cannot satisfy the acquisition; active reuse also replaces its client when validation returns a changed service URL. Reconciliation may still enumerate/adopt the old ID for orphan cleanup; it is not selected for the new identity.
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
|
||||
@ -110,7 +110,7 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
|
||||
}
|
||||
```
|
||||
|
||||
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider,系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。若旧线程首次创建 tombstone 根目录,会先同步其 `user-data` 父目录;主文件和转换文件进入 tombstone 时再按“目标目录先、源目录后”同步,只有整个目标树与 rename 两侧都已持久化才允许远端副作用开始。已存在的 commit/restore marker 在复用前也会重新同步事务目录,避免上一次 marker 创建只完成文件同步。提交删除时则先同步所有 tombstone 及崩溃恢复期间可见别名的移除,再持久化清除 commit;即使恢复扫描已经看不到主 tombstone,也会重新同步其嵌套目录后才清除 phase marker。回滚恢复使用相反的安全顺序,先同步可见主文件与转换副本,再移除并同步 tombstone,最后清除 restore。这样掉电既不会恢复远端已删除的旧 generation,也不会丢失已经补偿完成的宿主机 generation。远端失败且所有可能的副作用都成功补偿时,请求会返回错误并恢复宿主机文件;即使持久日志的中止清理失败,本地恢复也会在 `finally` 路径完成。如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation,此时请求仍返回错误。持续未确认时,受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。日志同时绑定 provider 类型、后端不可变命名空间、sandbox 不可变实例 ID 和原始 sandbox ID;日志删除前会先持久化跨进程 finalization guard,目录同步失败时即使可见日志无法重建,所有 Gateway 进程也不会提前释放旧 basename。Gateway 重启或后续嵌入式删除准备只能通过 provider 的非创建型接口精确重连该旧实例,或由同一后端明确确认该实例已经不存在;不会获取线程当前的新 sandbox,也不会跨 provider、后端或同 ID 的新实例清理。AIO 的本地 Docker 使用 daemon/container ID,远端 Provisioner 使用 Kubernetes Namespace/Pod UID;Provisioner 以 Pod 为权威,即使 Service 已经缺失也不会把仍存活的 Pod 报成 absent。其他 provider 若不能提供同等的 restart-safe 身份能力,删除准备会在宿主机暂存和远端删除之前 fail closed。无法确认时继续保留日志与名称预留,远端确认删除或旧实例确认不存在后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞;扫描期间已经完成收尾的无关删除事务会被视为已完成,而不是让当前上传失败。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON`、`report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
|
||||
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider,系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。若旧线程首次创建 tombstone 根目录,会先同步其 `user-data` 父目录;主文件和转换文件进入 tombstone 时再按“目标目录先、源目录后”同步,只有整个目标树与 rename 两侧都已持久化才允许远端副作用开始。已存在的 commit/restore marker 在复用前也会按“精确 marker 文件先、事务目录后”重新同步,覆盖上一次创建在任一 fsync 处失败的情况。提交删除时则先同步所有 tombstone 及崩溃恢复期间可见别名的移除,再持久化清除 commit;即使恢复扫描已经看不到主 tombstone,也会重新同步其嵌套目录后才清除 phase marker。回滚恢复使用相反的安全顺序,先同步可见主文件与转换副本,再移除并同步 tombstone,最后清除 restore。若非协作写入恰好替换了所选 inode 且 live recovery 已建立可见 peer,后续恢复只会在确认恰有一个同 inode peer 后移除隐藏别名,既不覆盖也不删除可见替换文件;没有可验证 peer 的 mismatched tombstone 仍 fail closed。这样掉电既不会恢复远端已删除的旧 generation,也不会丢失已经补偿完成的宿主机 generation。远端失败且所有可能的副作用都成功补偿时,请求会返回错误并恢复宿主机文件;即使持久日志的中止清理失败,本地恢复也会在 `finally` 路径完成。如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation,此时请求仍返回错误。持续未确认时,受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。日志同时绑定 provider 类型、后端不可变命名空间、sandbox 不可变实例 ID 和原始 sandbox ID;日志删除前会先持久化跨进程 finalization guard,目录同步失败时即使可见日志无法重建,所有 Gateway 进程也不会提前释放旧 basename。Gateway 重启或后续嵌入式删除准备只能通过 provider 的非创建型接口精确重连该旧实例,或由同一后端明确确认该实例已经不存在;不会获取线程当前的新 sandbox,也不会跨 provider、后端或同 ID 的新实例清理。AIO 的本地 Docker 使用 daemon/container ID,远端 Provisioner 使用 Kubernetes Namespace/Pod UID;Provisioner 以 Pod 为权威,即使 Service 已经缺失也不会把仍存活的 Pod 报成 absent。其他 provider 若不能提供同等的 restart-safe 身份能力,删除准备会在宿主机暂存和远端删除之前 fail closed。无法确认时继续保留日志与名称预留,远端确认删除或旧实例确认不存在后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞;扫描期间已经完成收尾的无关删除事务会被视为已完成,而不是让当前上传失败。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON`、`report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
|
||||
|
||||
如果日志 unlink 已经可见但目录 fsync 失败,系统会先尝试重建日志;即使磁盘错误导致重建也失败,已经落盘的 `.remote-delete.finalizing` 仍会跨进程保留 fail-closed 的文件名占用并阻止后台事务清理。启动清理或下一次删除准备只有在 guard 仍存在时同步事务目录、确认日志缺失已经持久化,才会删除 guard 并安全释放占用;若崩溃后旧日志恢复,则继续按原 generation 协调。
|
||||
|
||||
|
||||
@ -872,9 +872,24 @@ def _create_deletion_phase_marker(marker: Path, *, error_message: str) -> None:
|
||||
marker_stat = os.lstat(marker)
|
||||
if not stat.S_ISREG(marker_stat.st_mode) or marker_stat.st_nlink != 1:
|
||||
raise UnsafeUploadPathError(error_message)
|
||||
# A prior creation may have file-fsynced the marker but failed while
|
||||
# syncing its directory. Re-confirm the visible entry before callers
|
||||
# rely on this phase across a crash boundary.
|
||||
# A prior creation may have failed while syncing either the marker or
|
||||
# its directory. Open the exact regular entry without following links,
|
||||
# revalidate its inode, then persist file before directory so callers
|
||||
# can safely reuse this phase across a crash boundary.
|
||||
try:
|
||||
existing_descriptor = os.open(
|
||||
marker,
|
||||
os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0),
|
||||
)
|
||||
except OSError as exc:
|
||||
raise UnsafeUploadPathError(error_message) from exc
|
||||
try:
|
||||
existing_stat = os.fstat(existing_descriptor)
|
||||
if not stat.S_ISREG(existing_stat.st_mode) or existing_stat.st_nlink != 1 or (existing_stat.st_dev, existing_stat.st_ino) != (marker_stat.st_dev, marker_stat.st_ino):
|
||||
raise UnsafeUploadPathError(error_message)
|
||||
os.fsync(existing_descriptor)
|
||||
finally:
|
||||
os.close(existing_descriptor)
|
||||
_fsync_directory_durably(marker.parent)
|
||||
return
|
||||
try:
|
||||
@ -1267,13 +1282,46 @@ def _recover_stale_deletion_transaction(transaction_dir: Path) -> bool:
|
||||
try:
|
||||
staged_path = Path(entry.path)
|
||||
staged_stat = os.lstat(staged_path)
|
||||
if not stat.S_ISREG(staged_stat.st_mode) or staged_stat.st_nlink not in {1, 2} or staged_stat.st_ino != expected_inode:
|
||||
if not stat.S_ISREG(staged_stat.st_mode) or staged_stat.st_nlink not in {1, 2}:
|
||||
logger.warning("Refusing unsafe upload deletion tombstone: %s", staged_path)
|
||||
return False
|
||||
identity = UploadIdentity(
|
||||
device=staged_stat.st_dev,
|
||||
inode=staged_stat.st_ino,
|
||||
)
|
||||
if staged_stat.st_ino != expected_inode:
|
||||
# A non-cooperating writer may replace the selected inode between
|
||||
# the identity scan and rename. If live recovery already published
|
||||
# exactly one verified visible hard-link peer, only remove this
|
||||
# hidden alias. A sole mismatched tombstone stays fail-closed: it
|
||||
# cannot be distinguished from post-crash transaction tampering.
|
||||
if staged_stat.st_nlink != 2 or conversion_entries or commit_entries or restore_entries:
|
||||
logger.warning("Refusing unsafe upload deletion tombstone: %s", staged_path)
|
||||
return False
|
||||
visible_matches: list[Path] = []
|
||||
with os.scandir(uploads_dir) as upload_entries:
|
||||
for upload_entry in upload_entries:
|
||||
try:
|
||||
upload_stat = upload_entry.stat(follow_symlinks=False)
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
if stat.S_ISREG(upload_stat.st_mode) and (
|
||||
upload_stat.st_dev,
|
||||
upload_stat.st_ino,
|
||||
) == (identity.device, identity.inode):
|
||||
visible_matches.append(Path(upload_entry.path))
|
||||
if len(visible_matches) != 1:
|
||||
logger.warning(
|
||||
"Refusing unexpected-inode upload recovery with %s visible aliases: %s",
|
||||
len(visible_matches),
|
||||
staged_path,
|
||||
)
|
||||
return False
|
||||
_fsync_directory_durably(uploads_dir)
|
||||
staged_path.unlink()
|
||||
_fsync_directory_durably(staged_path.parent)
|
||||
_finish_deletion_transaction(staged_path)
|
||||
return True
|
||||
if not recover_on_crash:
|
||||
visible_matches: list[Path] = []
|
||||
if staged_stat.st_nlink == 2:
|
||||
@ -1362,8 +1410,28 @@ def _recover_stale_deletion_transaction(transaction_dir: Path) -> bool:
|
||||
def _restore_unexpected_staged_entry(staged_path: Path, original_path: Path) -> None:
|
||||
"""Restore an entry moved during an identity race without unlinking it."""
|
||||
staged_stat = os.lstat(staged_path)
|
||||
if not stat.S_ISREG(staged_stat.st_mode) or staged_stat.st_nlink != 1:
|
||||
if not stat.S_ISREG(staged_stat.st_mode) or staged_stat.st_nlink not in {1, 2}:
|
||||
raise UnsafeUploadPathError("Unsafe staged upload entry cannot be restored")
|
||||
if staged_stat.st_nlink == 2:
|
||||
visible_matches: list[Path] = []
|
||||
with os.scandir(original_path.parent) as upload_entries:
|
||||
for upload_entry in upload_entries:
|
||||
try:
|
||||
upload_stat = upload_entry.stat(follow_symlinks=False)
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
if stat.S_ISREG(upload_stat.st_mode) and (
|
||||
upload_stat.st_dev,
|
||||
upload_stat.st_ino,
|
||||
) == (staged_stat.st_dev, staged_stat.st_ino):
|
||||
visible_matches.append(Path(upload_entry.path))
|
||||
if len(visible_matches) != 1:
|
||||
raise UnsafeUploadPathError("Staged upload entry has an ambiguous hard-link peer")
|
||||
_fsync_directory_durably(original_path.parent)
|
||||
staged_path.unlink()
|
||||
_fsync_directory_durably(staged_path.parent)
|
||||
_finish_deletion_transaction(staged_path)
|
||||
return
|
||||
try:
|
||||
os.link(staged_path, original_path, follow_symlinks=False)
|
||||
except FileExistsError:
|
||||
|
||||
@ -1618,6 +1618,139 @@ class TestDeleteFileSafe:
|
||||
|
||||
assert transaction_dir in fsynced_directories
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("marker_function_name", "marker_path_function_name", "recover_on_crash"),
|
||||
[
|
||||
(
|
||||
"_mark_staged_deletion_committed",
|
||||
"_staged_deletion_commit_marker",
|
||||
True,
|
||||
),
|
||||
(
|
||||
"_mark_staged_deletion_restore",
|
||||
"_staged_deletion_restore_marker",
|
||||
False,
|
||||
),
|
||||
],
|
||||
ids=["commit", "restore"],
|
||||
)
|
||||
def test_existing_phase_marker_resyncs_file_before_parent_after_file_fsync_failure(
|
||||
self,
|
||||
tmp_path,
|
||||
marker_function_name,
|
||||
marker_path_function_name,
|
||||
recover_on_crash,
|
||||
):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
uploads = tmp_path / "user-data" / "uploads"
|
||||
uploads.mkdir(parents=True)
|
||||
primary = uploads / "report.pdf"
|
||||
primary.write_bytes(b"primary")
|
||||
identity = UploadIdentity.from_path(primary)
|
||||
staged_path, stage_lease = upload_manager_module._stage_primary_deletion(
|
||||
uploads,
|
||||
primary,
|
||||
identity,
|
||||
recover_on_crash=recover_on_crash,
|
||||
)
|
||||
transaction_dir = staged_path.parent.parent
|
||||
marker_function = getattr(upload_manager_module, marker_function_name)
|
||||
marker_path = getattr(upload_manager_module, marker_path_function_name)(staged_path)
|
||||
real_fsync = upload_manager_module.os.fsync
|
||||
failed = False
|
||||
|
||||
def fail_marker_file_once(descriptor):
|
||||
nonlocal failed
|
||||
descriptor_stat = os.fstat(descriptor)
|
||||
if stat.S_ISREG(descriptor_stat.st_mode) and not failed:
|
||||
failed = True
|
||||
raise OSError("cannot persist phase marker file")
|
||||
return real_fsync(descriptor)
|
||||
|
||||
try:
|
||||
with patch.object(
|
||||
upload_manager_module.os,
|
||||
"fsync",
|
||||
side_effect=fail_marker_file_once,
|
||||
):
|
||||
with pytest.raises(OSError, match="cannot persist phase marker file"):
|
||||
marker_function(staged_path)
|
||||
|
||||
marker_stat = os.lstat(marker_path)
|
||||
transaction_stat = os.lstat(transaction_dir)
|
||||
marker_identity = (marker_stat.st_dev, marker_stat.st_ino)
|
||||
transaction_identity = (transaction_stat.st_dev, transaction_stat.st_ino)
|
||||
events: list[str] = []
|
||||
|
||||
def observe_fsync(descriptor):
|
||||
descriptor_stat = os.fstat(descriptor)
|
||||
descriptor_identity = (descriptor_stat.st_dev, descriptor_stat.st_ino)
|
||||
if stat.S_ISREG(descriptor_stat.st_mode) and descriptor_identity == marker_identity:
|
||||
events.append("file")
|
||||
elif stat.S_ISDIR(descriptor_stat.st_mode) and descriptor_identity == transaction_identity:
|
||||
events.append("directory")
|
||||
return real_fsync(descriptor)
|
||||
|
||||
with patch.object(
|
||||
upload_manager_module.os,
|
||||
"fsync",
|
||||
side_effect=observe_fsync,
|
||||
):
|
||||
marker_function(staged_path)
|
||||
finally:
|
||||
stage_lease.release()
|
||||
|
||||
marker_file_sync = events.index("file")
|
||||
marker_directory_sync = events.index("directory", marker_file_sync)
|
||||
assert marker_file_sync < marker_directory_sync
|
||||
|
||||
def test_existing_phase_marker_refuses_inode_swap_before_file_fsync(self, tmp_path):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
uploads = tmp_path / "user-data" / "uploads"
|
||||
uploads.mkdir(parents=True)
|
||||
primary = uploads / "report.pdf"
|
||||
primary.write_bytes(b"primary")
|
||||
identity = UploadIdentity.from_path(primary)
|
||||
staged_path, stage_lease = upload_manager_module._stage_primary_deletion(
|
||||
uploads,
|
||||
primary,
|
||||
identity,
|
||||
recover_on_crash=True,
|
||||
)
|
||||
marker_path = upload_manager_module._staged_deletion_commit_marker(staged_path)
|
||||
upload_manager_module._mark_staged_deletion_committed(staged_path)
|
||||
original_marker_stat = os.lstat(marker_path)
|
||||
real_open = upload_manager_module.os.open
|
||||
replaced = False
|
||||
|
||||
def replace_marker_before_open(path, flags, *args):
|
||||
nonlocal replaced
|
||||
if Path(path) == marker_path and not (flags & os.O_CREAT) and not replaced:
|
||||
replaced = True
|
||||
marker_path.unlink()
|
||||
marker_path.write_bytes(b"replacement")
|
||||
return real_open(path, flags, *args)
|
||||
|
||||
try:
|
||||
with patch.object(
|
||||
upload_manager_module.os,
|
||||
"open",
|
||||
side_effect=replace_marker_before_open,
|
||||
):
|
||||
with pytest.raises(UnsafeUploadPathError, match="commit marker"):
|
||||
upload_manager_module._mark_staged_deletion_committed(staged_path)
|
||||
finally:
|
||||
stage_lease.release()
|
||||
|
||||
replacement_stat = os.lstat(marker_path)
|
||||
assert replaced
|
||||
assert (replacement_stat.st_dev, replacement_stat.st_ino) != (
|
||||
original_marker_stat.st_dev,
|
||||
original_marker_stat.st_ino,
|
||||
)
|
||||
|
||||
def test_staged_renames_are_durable_before_remote_mutation(self, tmp_path):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
@ -1754,6 +1887,81 @@ class TestDeleteFileSafe:
|
||||
assert conversion.read_text(encoding="utf-8") == "generated"
|
||||
assert not list(conversion.parent.glob(".upload-delete-*.part"))
|
||||
|
||||
def test_unexpected_inode_restore_recovers_after_visible_peer_fsync_failure(self, tmp_path):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
uploads = tmp_path / "user-data" / "uploads"
|
||||
uploads.mkdir(parents=True)
|
||||
primary = uploads / "report.pdf"
|
||||
primary.write_bytes(b"old")
|
||||
stale_identity = UploadIdentity.from_path(primary)
|
||||
primary.unlink()
|
||||
primary.write_bytes(b"replacement")
|
||||
assert not stale_identity.matches(primary)
|
||||
real_fsync_directory = upload_manager_module._fsync_directory_durably
|
||||
failed = False
|
||||
|
||||
def fail_visible_peer_once(directory):
|
||||
nonlocal failed
|
||||
if Path(directory) == uploads and not failed:
|
||||
failed = True
|
||||
raise OSError("cannot persist visible replacement")
|
||||
return real_fsync_directory(directory)
|
||||
|
||||
with patch.object(
|
||||
upload_manager_module,
|
||||
"_fsync_directory_durably",
|
||||
side_effect=fail_visible_peer_once,
|
||||
):
|
||||
with pytest.raises(OSError, match="cannot persist visible replacement"):
|
||||
upload_manager_module._stage_primary_deletion(
|
||||
uploads,
|
||||
primary,
|
||||
stale_identity,
|
||||
recover_on_crash=False,
|
||||
)
|
||||
|
||||
assert primary.read_bytes() == b"replacement"
|
||||
assert os.lstat(primary).st_nlink == 1
|
||||
assert not list(conversion_dir_for_uploads(uploads).glob(".upload-delete-*.part"))
|
||||
|
||||
def test_startup_cleans_verified_visible_peer_for_unexpected_inode_transaction(self, tmp_path):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
uploads = tmp_path / "user-data" / "uploads"
|
||||
uploads.mkdir(parents=True)
|
||||
primary = uploads / "report.pdf"
|
||||
primary.write_bytes(b"old")
|
||||
stale_identity = UploadIdentity.from_path(primary)
|
||||
primary.unlink()
|
||||
primary.write_bytes(b"replacement")
|
||||
assert not stale_identity.matches(primary)
|
||||
|
||||
with patch.object(
|
||||
upload_manager_module,
|
||||
"_restore_unexpected_staged_entry",
|
||||
side_effect=OSError("simulate crash before live recovery"),
|
||||
):
|
||||
with pytest.raises(OSError, match="simulate crash before live recovery"):
|
||||
upload_manager_module._stage_primary_deletion(
|
||||
uploads,
|
||||
primary,
|
||||
stale_identity,
|
||||
recover_on_crash=False,
|
||||
)
|
||||
|
||||
transactions = list(conversion_dir_for_uploads(uploads).glob(".upload-delete-*.part"))
|
||||
assert len(transactions) == 1
|
||||
transaction_dir = transactions[0]
|
||||
staged_path = transaction_dir / upload_manager_module._UPLOAD_DELETION_PRIMARY_DIRNAME / primary.name
|
||||
assert staged_path.read_bytes() == b"replacement"
|
||||
os.link(staged_path, primary)
|
||||
|
||||
assert upload_manager_module._recover_stale_deletion_transaction(transaction_dir)
|
||||
assert primary.read_bytes() == b"replacement"
|
||||
assert os.lstat(primary).st_nlink == 1
|
||||
assert not transaction_dir.exists()
|
||||
|
||||
def test_primary_tombstone_unlink_is_durable_before_commit_clear(self, tmp_path):
|
||||
import deerflow.uploads.manager as upload_manager_module
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user