fix: persist upload deletion recovery roots

This commit is contained in:
hetaoBackend 2026-08-07 05:47:31 +08:00
parent 3898cc0a5c
commit 5d4d75ad9b
4 changed files with 166 additions and 2 deletions

View File

@ -1464,7 +1464,7 @@ Multi-file upload with automatic document conversion:
- Exact-name generation leases use a portable NFC-plus-casefold coordination key with Win32 trailing-dot/space folding, so filesystem aliases cannot bypass an active generation; filenames that Windows cannot represent losslessly are rejected before staging, while exact legacy POSIX basenames remain deletable after upgrade. Legacy deletion has a POSIX-only lease mode for literal backslashes and components made entirely from dots/spaces; it does not weaken new-upload validation. The original filename remains the published name. Publication tries each candidate lease without blocking and treats a busy canonical key as a collision, so same-batch and inverse concurrent batches advance to a UTF-8-bounded `_N` candidate instead of deadlocking while retaining earlier generations; pathological long suffixes fall back to truncating the complete basename. Deletion rejects an inode that moved outside the requested name's lease. Primary deletion and rollback move the selected primary and its exact generated conversion into one inode-and-intent-tagged transaction directory under the protected conversion namespace before side effects; the primary lives in a dedicated child directory so legal filenames such as `.commit`, `.restore`, and `.conversion` cannot collide with transaction controls. User deletion starts with restore-on-crash intent. Immediately before the first possible remote mutation, a persisted commit marker makes crash recovery discard the authoritative host tombstones; a live, fully compensated remote failure clears it while restoring both host entries. A live rollback writes a restore marker before publishing either hard link, so recovery can finish restoring both entries even when the transaction began with discard intent. A partial remote failure is compensated from the staged authoritative bytes before host restoration; if compensation fails, every remote target is driven back toward the idempotent deletion outcome and host deletion commits instead of publishing a split generation. Failed-upload rollback tombstones otherwise use discard recovery, while legacy primary-only transactions retain their recorded restore/discard behavior. Neither transaction kind is treated as an ordinary disposable upload stage. Final lease release is the commit point: cancellation newly arriving during release is delayed and swallowed so a committed upload is returned as success rather than an indeterminate cancelled result. Embedded-client finalization attempts every lease release and conversion-pool shutdown independently, logging cleanup failures rather than changing an already-committed response or stranding later leases.
- Filenames containing NUL, `<`, `>`, or reserved model-context boundary markers are rejected before staging so accepted filenames and exact virtual paths remain lossless in model-visible upload context. Legacy files discovered on disk are still neutralized when listed.
- Gateway HTTP uploads use same-directory `.upload-*.part` staging files. Each active stage holds a cross-process liveness lock under `.upload-conversions/.locks/stages/`; startup cleanup skips held stages and sweeps only crash-orphaned files, while `.upload-delete-<restore|discard>-<inode>-<nonce>.part/` transaction directories are resolved according to their persisted crash intent. Legacy intent-less deletion transactions retain restore behavior. Cancellation during staging creation drains the worker and aborts the returned stage before propagating. Staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools.
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. Tombstone staging fsyncs each rename destination before its source directory and completes both sides before remote mutation. Committed cleanup fsyncs every tombstone or recovered visible-alias removal before clearing the transaction-root commit marker; compensated restore first fsyncs the visible primary and conversion peers, then their tombstone removals, before clearing restore intent. Thus recovery cannot resurrect a remotely deleted generation or lose a compensated host generation. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; journal-abort errors cannot skip that host restore. Failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. The v3 journal binds the provider type, backend-issued namespace, immutable sandbox incarnation, and raw sandbox ID. Journal finalization first persists a `.remote-delete.finalizing` guard; if journal-unlink directory fsync and journal restoration both fail, the guard remains a cross-process basename reservation until startup cleanup or a later deletion preparation fsyncs the transaction directory to make the observed journal absence durable. Gateway startup and later embedded deletion preparation may use only the provider's non-creating exact reconnect/confirmed-absence result—never `acquire()`, a replacement sandbox for the thread, or a different provider/backend/incarnation—before ordinary transaction cleanup removes the tombstones; an unknown lookup stays pending. AIO local Docker uses daemon/container IDs, while the remote Provisioner uses Kubernetes Namespace/Pod UIDs and treats Pod presence as authoritative even when its Service is gone. Providers without an explicit restart-safe reconciliation identity fail deletion preparation before host staging or remote mutation. Live journals remain protected by their stage lease, reconciliation takes the same name lease used by publication, and a disappearing unrelated transaction is treated as completed after the scanner proves it does not reserve the candidate. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. Tombstone staging first fsyncs the lazily created namespace in `user-data`, then each rename destination before its source directory, and completes the whole destination tree plus both rename sides before remote mutation. Committed cleanup fsyncs every tombstone or recovered visible-alias removal before clearing the transaction-root commit marker. When recovery sees an empty nested primary directory, it re-fsyncs that directory before clearing either commit or restore phase intent. Compensated restore first fsyncs the visible primary and conversion peers, then their tombstone removals, before clearing restore intent. Thus recovery cannot resurrect a remotely deleted generation or lose a compensated host generation. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; journal-abort errors cannot skip that host restore. Failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. The v3 journal binds the provider type, backend-issued namespace, immutable sandbox incarnation, and raw sandbox ID. Journal finalization first persists a `.remote-delete.finalizing` guard; if journal-unlink directory fsync and journal restoration both fail, the guard remains a cross-process basename reservation until startup cleanup or a later deletion preparation fsyncs the transaction directory to make the observed journal absence durable. Gateway startup and later embedded deletion preparation may use only the provider's non-creating exact reconnect/confirmed-absence result—never `acquire()`, a replacement sandbox for the thread, or a different provider/backend/incarnation—before ordinary transaction cleanup removes the tombstones; an unknown lookup stays pending. AIO local Docker uses daemon/container IDs, while the remote Provisioner uses Kubernetes Namespace/Pod UIDs and treats Pod presence as authoritative even when its Service is gone. Providers without an explicit restart-safe reconciliation identity fail deletion preparation before host staging or remote mutation. Live journals remain protected by their stage lease, reconciliation takes the same name lease used by publication, and a disappearing unrelated transaction is treated as completed after the scanner proves it does not reserve the candidate. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor; only operations that may block waiting for a name lease use the separate lease-wait pool. Work needed by an existing lease holder and non-blocking publication stays on the general pool, so waiters cannot starve conversion, rollback, or release. Cold sandbox-provider construction is also offloaded. Gateway, embedded-client, and IM ingresses share provider-aware publication: mounted providers make the exact host paths sandbox-readable; non-mounted providers acquire the sandbox and synchronize the primary plus generated conversion to their exact virtual paths. Each ingress records attempted remote paths before the write can commit and, on later failure or cancellation, calls the provider-neutral `Sandbox.remove_file()` for those paths before host rollback and lease release; the command fallback requires a per-call unpredictable exact success trailer. Embedded multi-file calls retain every publication and receipt until the whole response is built, then roll back the complete batch on failure. WeChat download publication uses the cancellation-safe async lease adapter, so cancellation drains and rolls back a publication worker that completes late.
- Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate `sandbox.thread_data_mounts: true` plus a Provisioner advertising a compatible mount contract; every upload/delete/artifact decision waits for any in-flight stale capability probe in a worker thread and uses one atomic snapshot. The same frozen version selects the deterministic sandbox ID, mount set, cache/reuse candidate, create precondition, and response validation; if the peer changes versions during acquisition, the whole attempt restarts with a new ID. The required version is sent in the create JSON, and the Provisioner rejects a mismatch before any Kubernetes read or write, preventing a rolling upgrade from creating an orphan under the prior version's ID. After acquisition, consumers use the immutable contract recorded on that sandbox instance rather than the provider's mutable global capability. Unavailable or legacy results use explicit synchronization and are retried without a Gateway restart. A confirmed legacy peer may create thread sandboxes only for the `default` no-auth user, because its primary user-data mount cannot isolate equal thread IDs across authenticated users; authenticated rolling deployments upgrade the Provisioner first. The Provisioner validates the request before its idempotent fast path, labels Pods with hashed user/thread identity and contract version, stores the exact identity plus a mount-signature annotation, and compares the live Pod specification before reuse. Every remote acquisition path, including active-cache and warm-pool reuse, deliberately replays the idempotent create request instead of trusting GET discovery or a liveness response. Contracts newer than v2 are accepted as forward-compatible, but the response must echo the exact frozen version, user, and thread; legacy responses may omit the version or return exact integer zero, never strings or booleans. The negotiated mount-contract version namespaces deterministic AIO sandbox IDs so a container created under another version cannot satisfy the acquisition; active reuse also replaces its client when validation returns a changed service URL. Reconciliation may still enumerate/adopt the old ID for orphan cleanup; it is not selected for the new identity.
- Agent receives uploaded file list via `UploadsMiddleware`

View File

@ -110,7 +110,7 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
}
```
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。主文件和转换文件进入 tombstone 时按“目标目录先、源目录后”同步,只有 rename 两侧都已持久化才允许远端副作用开始提交删除时则先同步所有 tombstone 及崩溃恢复期间可见别名的移除,再持久化清除 commit回滚恢复使用相反的安全顺序先同步可见主文件与转换副本再移除并同步 tombstone最后清除 restore。这样掉电既不会恢复远端已删除的旧 generation也不会丢失已经补偿完成的宿主机 generation。远端失败且所有可能的副作用都成功补偿时请求会返回错误并恢复宿主机文件即使持久日志的中止清理失败本地恢复也会在 `finally` 路径完成。如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation此时请求仍返回错误。持续未确认时受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。日志同时绑定 provider 类型、后端不可变命名空间、sandbox 不可变实例 ID 和原始 sandbox ID日志删除前会先持久化跨进程 finalization guard目录同步失败时即使可见日志无法重建所有 Gateway 进程也不会提前释放旧 basename。Gateway 重启或后续嵌入式删除准备只能通过 provider 的非创建型接口精确重连该旧实例,或由同一后端明确确认该实例已经不存在;不会获取线程当前的新 sandbox也不会跨 provider、后端或同 ID 的新实例清理。AIO 的本地 Docker 使用 daemon/container ID远端 Provisioner 使用 Kubernetes Namespace/Pod UIDProvisioner 以 Pod 为权威,即使 Service 已经缺失也不会把仍存活的 Pod 报成 absent。其他 provider 若不能提供同等的 restart-safe 身份能力,删除准备会在宿主机暂存和远端删除之前 fail closed。无法确认时继续保留日志与名称预留远端确认删除或旧实例确认不存在后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞;扫描期间已经完成收尾的无关删除事务会被视为已完成,而不是让当前上传失败。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON``report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。若旧线程首次创建 tombstone 根目录,会先同步其 `user-data` 父目录;主文件和转换文件进入 tombstone 时按“目标目录先、源目录后”同步,只有整个目标树与 rename 两侧都已持久化才允许远端副作用开始提交删除时则先同步所有 tombstone 及崩溃恢复期间可见别名的移除,再持久化清除 commit即使恢复扫描已经看不到主 tombstone也会重新同步其嵌套目录后才清除 phase marker。回滚恢复使用相反的安全顺序,先同步可见主文件与转换副本,再移除并同步 tombstone最后清除 restore。这样掉电既不会恢复远端已删除的旧 generation也不会丢失已经补偿完成的宿主机 generation。远端失败且所有可能的副作用都成功补偿时请求会返回错误并恢复宿主机文件即使持久日志的中止清理失败本地恢复也会在 `finally` 路径完成。如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation此时请求仍返回错误。持续未确认时受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。日志同时绑定 provider 类型、后端不可变命名空间、sandbox 不可变实例 ID 和原始 sandbox ID日志删除前会先持久化跨进程 finalization guard目录同步失败时即使可见日志无法重建所有 Gateway 进程也不会提前释放旧 basename。Gateway 重启或后续嵌入式删除准备只能通过 provider 的非创建型接口精确重连该旧实例,或由同一后端明确确认该实例已经不存在;不会获取线程当前的新 sandbox也不会跨 provider、后端或同 ID 的新实例清理。AIO 的本地 Docker 使用 daemon/container ID远端 Provisioner 使用 Kubernetes Namespace/Pod UIDProvisioner 以 Pod 为权威,即使 Service 已经缺失也不会把仍存活的 Pod 报成 absent。其他 provider 若不能提供同等的 restart-safe 身份能力,删除准备会在宿主机暂存和远端删除之前 fail closed。无法确认时继续保留日志与名称预留远端确认删除或旧实例确认不存在后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞;扫描期间已经完成收尾的无关删除事务会被视为已完成,而不是让当前上传失败。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON``report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
如果日志 unlink 已经可见但目录 fsync 失败,系统会先尝试重建日志;即使磁盘错误导致重建也失败,已经落盘的 `.remote-delete.finalizing` 仍会跨进程保留 fail-closed 的文件名占用并阻止后台事务清理。启动清理或下一次删除准备只有在 guard 仍存在时同步事务目录、确认日志缺失已经持久化,才会删除 guard 并安全释放占用;若崩溃后旧日志恢复,则继续按原 generation 协调。

View File

@ -1213,6 +1213,11 @@ def _recover_stale_deletion_transaction(transaction_dir: Path) -> bool:
return False
if not primary_entries:
if recover_on_crash and not commit_entries and not conversion_entries:
if primary_dir is not None:
# A restore attempt may have unlinked the nested tombstone but
# failed while persisting that absence. Confirm it before
# clearing restore intent or removing the transaction tree.
_fsync_directory_durably(primary_dir)
if restore_entries:
_unlink_deletion_control_durably(Path(restore_entries[0].path))
if primary_dir is not None:
@ -1225,6 +1230,11 @@ def _recover_stale_deletion_transaction(transaction_dir: Path) -> bool:
if conversion_entries:
Path(conversion_entries[0].path).unlink()
_fsync_directory_durably(transaction_dir)
if primary_dir is not None:
# A prior discard may have made the nested primary unlink visible
# but failed while fsyncing this directory. Re-confirm that absence
# before making commit absence durable in the transaction root.
_fsync_directory_durably(primary_dir)
_unlink_deletion_control_durably(Path(commit_entries[0].path))
if primary_dir is not None:
primary_dir.rmdir()
@ -1379,6 +1389,10 @@ def _stage_primary_deletion(
) -> tuple[Path, UploadStageLease]:
"""Move one primary generation and its exact conversion into a transaction."""
staging_dir = ensure_conversion_dir(base_dir)
# The conversion namespace may have been created lazily for a legacy
# thread. Persist its entry in user-data before moving the only primary
# into that subtree and durably removing the source name.
_fsync_directory_durably(staging_dir.parent)
intent = _UPLOAD_DELETION_RESTORE_INTENT if recover_on_crash else _UPLOAD_DELETION_DISCARD_INTENT
while True:
transaction_dir = staging_dir / (f"{UPLOAD_DELETION_TRANSACTION_PREFIX}{intent}-{identity.inode:x}-{secrets.token_hex(16)}{UPLOAD_STAGING_SUFFIX}")

View File

@ -17,6 +17,7 @@ import pytest
from deerflow.uploads.async_helpers import release_published_upload_async
from deerflow.uploads.layout import (
artifact_url_for_virtual_path,
conversion_dir_for_uploads,
conversion_path_for_upload,
conversion_virtual_path,
)
@ -1609,6 +1610,47 @@ class TestDeleteFileSafe:
assert primary_destination_sync < primary_source_sync < conversion_rename
assert conversion_destination_sync < conversion_source_sync
def test_staging_root_is_durable_before_primary_source_removal(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module
user_data = tmp_path / "user-data"
uploads = user_data / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"primary")
assert not conversion_dir_for_uploads(uploads).exists()
events: list[tuple[str, str]] = []
real_rename = upload_manager_module.os.rename
real_fsync_directory = upload_manager_module._fsync_directory_durably
def observe_rename(source, destination):
if Path(source) == primary:
events.append(("rename", "primary-source"))
return real_rename(source, destination)
def observe_fsync_directory(directory):
if Path(directory) == user_data:
events.append(("fsync", "user-data"))
return real_fsync_directory(directory)
with (
patch.object(upload_manager_module.os, "rename", side_effect=observe_rename),
patch.object(
upload_manager_module,
"_fsync_directory_durably",
side_effect=observe_fsync_directory,
),
):
delete_file_safe(
uploads,
primary.name,
delete_remote_copy=lambda *_args: None,
)
staging_root_sync = events.index(("fsync", "user-data"))
primary_rename = events.index(("rename", "primary-source"), staging_root_sync)
assert staging_root_sync < primary_rename
def test_staging_directory_fsync_failure_restores_host_before_remote_mutation(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module
@ -1909,6 +1951,114 @@ class TestDeleteFileSafe:
commit_clear = events.index(("clear", "commit"), uploads_sync)
assert visible_unlink < uploads_sync < commit_clear
def test_missing_primary_recovery_resyncs_tombstone_dir_before_commit_clear(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module
uploads = tmp_path / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"primary")
identity = UploadIdentity.from_path(primary)
staged_path, stage_lease = upload_manager_module._stage_primary_deletion(
uploads,
primary,
identity,
recover_on_crash=True,
)
upload_manager_module._mark_staged_deletion_committed(staged_path)
transaction_dir = staged_path.parent.parent
primary_dir = staged_path.parent
# Model a previous discard whose unlink is visible but whose directory
# fsync failed. Recovery must persist this observed absence itself
# before making commit absence durable.
staged_path.unlink()
stage_lease.release()
events: list[tuple[str, str]] = []
real_fsync_directory = upload_manager_module._fsync_directory_durably
real_unlink_control = upload_manager_module._unlink_deletion_control_durably
def observe_fsync_directory(directory):
if Path(directory) == primary_dir:
events.append(("fsync", "primary-tombstone-dir"))
return real_fsync_directory(directory)
def observe_control_unlink(path):
if path.name == upload_manager_module._UPLOAD_DELETION_COMMIT_MARKER:
events.append(("clear", "commit"))
return real_unlink_control(path)
with (
patch.object(
upload_manager_module,
"_fsync_directory_durably",
side_effect=observe_fsync_directory,
),
patch.object(
upload_manager_module,
"_unlink_deletion_control_durably",
side_effect=observe_control_unlink,
),
):
assert upload_manager_module._recover_stale_deletion_transaction(transaction_dir)
primary_dir_sync = events.index(("fsync", "primary-tombstone-dir"))
commit_clear = events.index(("clear", "commit"), primary_dir_sync)
assert primary_dir_sync < commit_clear
def test_missing_primary_recovery_resyncs_tombstone_dir_before_restore_clear(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module
uploads = tmp_path / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"primary")
identity = UploadIdentity.from_path(primary)
staged_path, stage_lease = upload_manager_module._stage_primary_deletion(
uploads,
primary,
identity,
recover_on_crash=False,
)
upload_manager_module._mark_staged_deletion_restore(staged_path)
os.link(staged_path, primary)
transaction_dir = staged_path.parent.parent
primary_dir = staged_path.parent
# Model a restore that published and persisted the visible peer, then
# made the tombstone unlink visible but failed its directory fsync.
staged_path.unlink()
stage_lease.release()
events: list[tuple[str, str]] = []
real_fsync_directory = upload_manager_module._fsync_directory_durably
real_unlink_control = upload_manager_module._unlink_deletion_control_durably
def observe_fsync_directory(directory):
if Path(directory) == primary_dir:
events.append(("fsync", "primary-tombstone-dir"))
return real_fsync_directory(directory)
def observe_control_unlink(path):
if path.name == upload_manager_module._UPLOAD_DELETION_RESTORE_MARKER:
events.append(("clear", "restore"))
return real_unlink_control(path)
with (
patch.object(
upload_manager_module,
"_fsync_directory_durably",
side_effect=observe_fsync_directory,
),
patch.object(
upload_manager_module,
"_unlink_deletion_control_durably",
side_effect=observe_control_unlink,
),
):
assert upload_manager_module._recover_stale_deletion_transaction(transaction_dir)
primary_dir_sync = events.index(("fsync", "primary-tombstone-dir"))
restore_clear = events.index(("clear", "restore"), primary_dir_sync)
assert primary_dir_sync < restore_clear
def test_commit_marker_failure_prevents_remote_side_effect_and_restores_host(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module