fix: make remote deletion recovery provider safe

This commit is contained in:
hetaoBackend 2026-08-07 04:13:55 +08:00
parent 732cd95f83
commit f6ec5d4ce6
14 changed files with 723 additions and 49 deletions

View File

@ -1464,7 +1464,7 @@ Multi-file upload with automatic document conversion:
- Exact-name generation leases use a portable NFC-plus-casefold coordination key with Win32 trailing-dot/space folding, so filesystem aliases cannot bypass an active generation; filenames that Windows cannot represent losslessly are rejected before staging, while exact legacy POSIX basenames remain deletable after upgrade. Legacy deletion has a POSIX-only lease mode for literal backslashes and components made entirely from dots/spaces; it does not weaken new-upload validation. The original filename remains the published name. Publication tries each candidate lease without blocking and treats a busy canonical key as a collision, so same-batch and inverse concurrent batches advance to a UTF-8-bounded `_N` candidate instead of deadlocking while retaining earlier generations; pathological long suffixes fall back to truncating the complete basename. Deletion rejects an inode that moved outside the requested name's lease. Primary deletion and rollback move the selected primary and its exact generated conversion into one inode-and-intent-tagged transaction directory under the protected conversion namespace before side effects; the primary lives in a dedicated child directory so legal filenames such as `.commit`, `.restore`, and `.conversion` cannot collide with transaction controls. User deletion starts with restore-on-crash intent. Immediately before the first possible remote mutation, a persisted commit marker makes crash recovery discard the authoritative host tombstones; a live, fully compensated remote failure clears it while restoring both host entries. A live rollback writes a restore marker before publishing either hard link, so recovery can finish restoring both entries even when the transaction began with discard intent. A partial remote failure is compensated from the staged authoritative bytes before host restoration; if compensation fails, every remote target is driven back toward the idempotent deletion outcome and host deletion commits instead of publishing a split generation. Failed-upload rollback tombstones otherwise use discard recovery, while legacy primary-only transactions retain their recorded restore/discard behavior. Neither transaction kind is treated as an ordinary disposable upload stage. Final lease release is the commit point: cancellation newly arriving during release is delayed and swallowed so a committed upload is returned as success rather than an indeterminate cancelled result. Embedded-client finalization attempts every lease release and conversion-pool shutdown independently, logging cleanup failures rather than changing an already-committed response or stranding later leases.
- Filenames containing NUL, `<`, `>`, or reserved model-context boundary markers are rejected before staging so accepted filenames and exact virtual paths remain lossless in model-visible upload context. Legacy files discovered on disk are still neutralized when listed.
- Gateway HTTP uploads use same-directory `.upload-*.part` staging files. Each active stage holds a cross-process liveness lock under `.upload-conversions/.locks/stages/`; startup cleanup skips held stages and sweeps only crash-orphaned files, while `.upload-delete-<restore|discard>-<inode>-<nonce>.part/` transaction directories are resolved according to their persisted crash intent. Legacy intent-less deletion transactions retain restore behavior. Cancellation during staging creation drains the worker and aborts the returned stage before propagating. Staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools.
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. Gateway startup and later embedded deletion preparation retry only against the journal's exact recorded sandbox ID—never a replacement sandbox acquired for the thread—before ordinary transaction cleanup removes the tombstones. Live journals remain protected by their stage lease, and reconciliation takes the same name lease used by publication. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
- Generated Markdown is owned by `user-data/.upload-conversions/<actual-primary-filename>.md` and is omitted from primary upload listings. Deletion holds the generation lease while it stages both authoritative host paths, writes a durable remote-deletion journal, persists the remote-mutation phase, removes an explicitly synchronized sandbox primary/conversion, and commits the host transaction. A remote failure is reported and restores the host generation only after every possibly removed remote path has been re-published; journal-abort errors cannot skip that host restore. Failed compensation instead retries idempotent removal of every remote target and commits host deletion. Unconfirmed paths keep the journal, commit marker, and exact hidden host tombstones; that retained primary persistently reserves its portable filename coordination key, so a new upload advances to `_N` rather than letting an old path-based retry delete the new remote generation. The journal binds both the provider/backend reconciliation namespace and raw sandbox ID. Gateway startup and later embedded deletion preparation may use only the provider's non-creating exact reconnect/confirmed-absence result—never `acquire()`, a replacement sandbox for the thread, or a different provider/backend namespace—before ordinary transaction cleanup removes the tombstones; an unknown lookup stays pending. Live journals remain protected by their stage lease, reconciliation takes the same name lease used by publication, and a disappearing unrelated transaction is treated as completed after the scanner proves it does not reserve the candidate. Exact legacy POSIX basenames use the existing-file deletion validator during retry. It never guesses or deletes a legacy/user-owned `uploads/<stem>.md` sibling. Outline extraction opens one descriptor, verifies its `fstat` against the current exclusive regular directory entry, and uses that same descriptor for both outline and preview reads.
- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor; only operations that may block waiting for a name lease use the separate lease-wait pool. Work needed by an existing lease holder and non-blocking publication stays on the general pool, so waiters cannot starve conversion, rollback, or release. Cold sandbox-provider construction is also offloaded. Gateway, embedded-client, and IM ingresses share provider-aware publication: mounted providers make the exact host paths sandbox-readable; non-mounted providers acquire the sandbox and synchronize the primary plus generated conversion to their exact virtual paths. Each ingress records attempted remote paths before the write can commit and, on later failure or cancellation, calls the provider-neutral `Sandbox.remove_file()` for those paths before host rollback and lease release; the command fallback requires a per-call unpredictable exact success trailer. Embedded multi-file calls retain every publication and receipt until the whole response is built, then roll back the complete batch on failure. WeChat download publication uses the cancellation-safe async lease adapter, so cancellation drains and rolls back a publication worker that completes late.
- Mounted upload paths skip both sandbox acquisition and per-file synchronization. For AIO remote/provisioner deployments this requires an explicit, accurate `sandbox.thread_data_mounts: true` plus a Provisioner advertising a compatible mount contract; every upload/delete/artifact decision waits for any in-flight stale capability probe in a worker thread and uses one atomic snapshot. The same frozen version selects the deterministic sandbox ID, mount set, cache/reuse candidate, create precondition, and response validation; if the peer changes versions during acquisition, the whole attempt restarts with a new ID. The required version is sent in the create JSON, and the Provisioner rejects a mismatch before any Kubernetes read or write, preventing a rolling upgrade from creating an orphan under the prior version's ID. After acquisition, consumers use the immutable contract recorded on that sandbox instance rather than the provider's mutable global capability. Unavailable or legacy results use explicit synchronization and are retried without a Gateway restart. A confirmed legacy peer may create thread sandboxes only for the `default` no-auth user, because its primary user-data mount cannot isolate equal thread IDs across authenticated users; authenticated rolling deployments upgrade the Provisioner first. The Provisioner validates the request before its idempotent fast path, labels Pods with hashed user/thread identity and contract version, stores the exact identity plus a mount-signature annotation, and compares the live Pod specification before reuse. Every remote acquisition path, including active-cache and warm-pool reuse, deliberately replays the idempotent create request instead of trusting GET discovery or a liveness response. Contracts newer than v2 are accepted as forward-compatible, but the response must echo the exact frozen version, user, and thread; legacy responses may omit the version or return exact integer zero, never strings or booleans. The negotiated mount-contract version namespaces deterministic AIO sandbox IDs so a container created under another version cannot satisfy the acquisition; active reuse also replaces its client when validation returns a changed service URL. Reconciliation may still enumerate/adopt the old ID for orphan cleanup; it is not selected for the new identity.
- Agent receives uploaded file list via `UploadsMiddleware`

View File

@ -110,7 +110,7 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
}
```
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。远端失败且所有可能的副作用都成功补偿时,请求会返回错误并恢复宿主机文件;如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation此时请求仍返回错误。持续未确认时受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。Gateway 重启或后续嵌入式删除准备只会重新连接日志中记录的精确 sandbox ID不会把清理重定向到该线程后来获取的新 sandbox远端确认删除后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON``report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
删除 `document.pdf` 时,会先等待该实际文件名当前正在进行的上传、转换或沙箱同步生命周期结束。对于非挂载 provider系统在同一个 generation lease 内先暂存宿主机主文件与精确转换副本、持久化远端删除日志,再删除沙箱中的对应副本。远端失败且所有可能的副作用都成功补偿时,请求会返回错误并恢复宿主机文件;即使持久日志的中止清理失败,本地恢复也会在 `finally` 路径完成。如果补偿无法完整确认,系统会重试将全部远端目标收敛到删除状态并提交宿主机删除,避免重新暴露一个远端已经缺失的宿主机 generation此时请求仍返回错误。持续未确认时受保护的日志、提交标记和精确宿主机 tombstone 会一起保留;隐藏的主文件 tombstone 持续占用该文件名的可移植协调键,因此新的同名上传会得到 `_N` 名称,不会被旧日志的按路径重试误删。日志同时绑定 provider/backend 命名空间和原始 sandbox ID。Gateway 重启或后续嵌入式删除准备只能通过 provider 的非创建型接口精确重连该旧实例,或由后端明确确认它已经不存在;不会获取线程当前的新 sandbox也不会跨 provider/backend 命名空间清理。无法确认时继续保留日志与名称预留,远端确认删除或实例确认不存在后才清理 tombstone。随后只删除宿主机上它精确拥有的生成资产和主文件。系统不会推断或删除 `uploads/document.md`;该文件可能是用户独立上传的内容。其他文件名不会被这次等待阻塞;扫描期间已经完成收尾的无关删除事务会被视为已完成,而不是让当前上传失败。在 POSIX 部署上,升级前已经存在且能被列表接口返回的 Windows 非兼容文件名(例如 `CON``report?.pdf`、含反斜杠、仅由点/空格组成或末尾带空格的名称)仍可按返回的精确名称删除,远端日志重试也沿用同一精确旧名称;新上传仍执行严格的跨平台文件名校验。
## 支持的文档格式

View File

@ -43,7 +43,7 @@ from deerflow.integrations.lark_cli import INTEGRATION_ID as LARK_CLI_INTEGRATIO
from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI_SANDBOX_DATA_DIR, LARK_CLI_SANDBOX_RUNTIME_DIR, ensure_lark_cli_credential_tree, lark_skills_installed
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.sandbox.sandbox import Sandbox
from deerflow.sandbox.sandbox_provider import SandboxProvider
from deerflow.sandbox.sandbox_provider import SandboxProvider, SandboxReconciliationResult
from deerflow.uploads.layout import UPLOAD_CONVERSIONS_DIRNAME, ensure_conversion_dir
from .aio_sandbox import AioSandbox
@ -2571,6 +2571,51 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
self._last_activity[sandbox_id] = time.time()
return sandbox
def reconciliation_provider_key(self) -> str:
"""Bind durable journals to this provider's concrete backend namespace."""
provider_type = type(self)
backend = self._backend
backend_type = type(backend)
namespace_parts = [
f"{provider_type.__module__}.{provider_type.__qualname__}",
f"{backend_type.__module__}.{backend_type.__qualname__}",
]
if isinstance(backend, RemoteSandboxBackend):
namespace_parts.append(backend.provisioner_url)
elif isinstance(backend, LocalContainerBackend):
namespace_parts.extend([backend.runtime, backend._container_prefix])
namespace = "\0".join(namespace_parts).encode("utf-8")
return f"{namespace_parts[0]}:{hashlib.sha256(namespace).hexdigest()}"
def reconnect_sandbox_for_reconciliation(
self,
sandbox_id: str,
*,
thread_id: str,
user_id: str | None,
) -> SandboxReconciliationResult:
"""Reconnect to exactly *sandbox_id* without acquiring or creating."""
active = self.get(sandbox_id)
if active is not None:
return SandboxReconciliationResult.found(active)
discovery = self._backend.discover_for_reconciliation(sandbox_id)
if discovery.status == "absent":
return SandboxReconciliationResult.absent()
if discovery.status != "found" or discovery.info is None:
return SandboxReconciliationResult.unknown()
info = discovery.info
if info.sandbox_id != sandbox_id:
return SandboxReconciliationResult.unknown()
if info.thread_id is not None and info.thread_id != thread_id:
return SandboxReconciliationResult.unknown()
effective_user_id = self._effective_acquire_user_id(user_id)
if info.user_id is not None and info.user_id != effective_user_id:
return SandboxReconciliationResult.unknown()
return SandboxReconciliationResult.found(
AioSandbox(sandbox_id, info.sandbox_url),
close_after=True,
)
def release(self, sandbox_id: str) -> None:
"""Release a sandbox from active use into the warm pool.

View File

@ -7,6 +7,8 @@ import ipaddress
import logging
import time
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Literal
from urllib.parse import urlparse
import httpx
@ -17,6 +19,26 @@ from .sandbox_info import SandboxInfo
logger = logging.getLogger(__name__)
@dataclass(frozen=True, slots=True)
class SandboxDiscoveryResult:
"""Non-creating discovery outcome with explicit absence semantics."""
status: Literal["found", "absent", "unknown"]
info: SandboxInfo | None = None
@classmethod
def found(cls, info: SandboxInfo) -> SandboxDiscoveryResult:
return cls(status="found", info=info)
@classmethod
def absent(cls) -> SandboxDiscoveryResult:
return cls(status="absent")
@classmethod
def unknown(cls) -> SandboxDiscoveryResult:
return cls(status="unknown")
def sandbox_http_trust_env(sandbox_url: str) -> bool:
"""Whether HTTP clients for *sandbox_url* should inherit proxy settings.
@ -172,6 +194,20 @@ class SandboxBackend(ABC):
"""
...
def discover_for_reconciliation(self, sandbox_id: str) -> SandboxDiscoveryResult:
"""Discover one exact old instance without creating a replacement.
The base contract cannot distinguish a definite absence from a
transient discovery failure, so only a positive lookup is conclusive.
Backends with authoritative lookup APIs should override this method.
"""
try:
info = self.discover(sandbox_id)
except BaseException:
logger.warning("Sandbox discovery failed during reconciliation", exc_info=True)
return SandboxDiscoveryResult.unknown()
return SandboxDiscoveryResult.found(info) if info is not None else SandboxDiscoveryResult.unknown()
def list_running(self) -> list[SandboxInfo]:
"""Enumerate all running sandboxes managed by this backend.

View File

@ -15,7 +15,7 @@ from datetime import datetime
from deerflow.utils.network import get_free_port, release_port
from .backend import SandboxBackend, wait_for_sandbox_ready
from .backend import SandboxBackend, SandboxDiscoveryResult, wait_for_sandbox_ready
from .sandbox_info import SandboxInfo
logger = logging.getLogger(__name__)
@ -406,6 +406,19 @@ class LocalContainerBackend(SandboxBackend):
container_name=container_name,
)
def discover_for_reconciliation(self, sandbox_id: str) -> SandboxDiscoveryResult:
"""Resolve an exact deterministic container, preserving uncertainty."""
container_name = f"{self._container_prefix}-{sandbox_id}"
try:
running = self._is_container_running(container_name)
except RuntimeError:
logger.warning("Could not verify container %s during reconciliation", container_name, exc_info=True)
return SandboxDiscoveryResult.unknown()
if not running:
return SandboxDiscoveryResult.absent()
info = self.discover(sandbox_id)
return SandboxDiscoveryResult.found(info) if info is not None else SandboxDiscoveryResult.unknown()
def list_running(self) -> list[SandboxInfo]:
"""Enumerate all running containers matching the configured prefix.

View File

@ -30,7 +30,7 @@ import requests
from deerflow.runtime.user_context import DEFAULT_USER_ID, get_effective_user_id
from deerflow.skills.storage import user_should_see_legacy_skills
from .backend import SandboxBackend
from .backend import SandboxBackend, SandboxDiscoveryResult
from .sandbox_info import SandboxInfo
logger = logging.getLogger(__name__)
@ -341,6 +341,39 @@ class RemoteSandboxBackend(SandboxBackend):
"""
return self._provisioner_discover(sandbox_id)
def discover_for_reconciliation(self, sandbox_id: str) -> SandboxDiscoveryResult:
"""Look up an exact provisioner object without mount-version filtering."""
try:
resp = requests.get(
f"{self._provisioner_url}/api/sandboxes/{sandbox_id}",
headers=self._auth_headers(),
timeout=10,
)
if resp.status_code == 404:
return SandboxDiscoveryResult.absent()
resp.raise_for_status()
data = resp.json()
if not isinstance(data, dict):
raise ValueError("Provisioner reconciliation response is not an object")
returned_id = data.get("sandbox_id")
if returned_id is not None and returned_id != sandbox_id:
raise ValueError("Provisioner reconciliation returned a different sandbox ID")
sandbox_url = data.get("sandbox_url")
if not isinstance(sandbox_url, str) or not sandbox_url:
raise ValueError("Provisioner reconciliation response has no sandbox URL")
return SandboxDiscoveryResult.found(
SandboxInfo(
sandbox_id=sandbox_id,
sandbox_url=sandbox_url,
user_id=data.get("user_id"),
thread_id=data.get("thread_id"),
mount_contract_version=data.get("mount_contract_version"),
)
)
except (requests.RequestException, ValueError, TypeError):
logger.warning("Provisioner reconciliation lookup failed for %s", sandbox_id, exc_info=True)
return SandboxDiscoveryResult.unknown()
def list_running(self) -> list[SandboxInfo]:
"""Return all sandboxes currently managed by the provisioner.

View File

@ -2,12 +2,35 @@ import asyncio
import inspect
import threading
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Literal
from deerflow.config import get_app_config
from deerflow.reflection import resolve_class
from deerflow.sandbox.sandbox import Sandbox
@dataclass(frozen=True, slots=True)
class SandboxReconciliationResult:
"""Exact, non-creating lookup result for durable remote operations."""
status: Literal["found", "absent", "unknown"]
sandbox: Sandbox | None = None
close_after: bool = False
@classmethod
def found(cls, sandbox: Sandbox, *, close_after: bool = False) -> "SandboxReconciliationResult":
return cls(status="found", sandbox=sandbox, close_after=close_after)
@classmethod
def absent(cls) -> "SandboxReconciliationResult":
return cls(status="absent")
@classmethod
def unknown(cls) -> "SandboxReconciliationResult":
return cls(status="unknown")
class SandboxProvider(ABC):
"""Abstract base class for sandbox providers"""
@ -42,6 +65,30 @@ class SandboxProvider(ABC):
"""
pass
def reconciliation_provider_key(self) -> str:
"""Return a stable, non-secret backend namespace for durable journals."""
provider_type = type(self)
return f"{provider_type.__module__}.{provider_type.__qualname__}"
def reconnect_sandbox_for_reconciliation(
self,
sandbox_id: str,
*,
thread_id: str,
user_id: str | None,
) -> SandboxReconciliationResult:
"""Resolve exactly *sandbox_id* without creating or redirecting it.
Providers that can discover an old instance or prove it terminally
absent should override this method. The conservative default only
trusts an already-active exact ID; a cache miss remains ``unknown``.
"""
del thread_id, user_id
sandbox = self.get(sandbox_id)
if sandbox is None:
return SandboxReconciliationResult.unknown()
return SandboxReconciliationResult.found(sandbox)
@abstractmethod
def release(self, sandbox_id: str) -> None:
"""Release a sandbox environment.

View File

@ -317,37 +317,54 @@ def _pending_remote_deletion_reserves_name(base_dir: Path, coordination_key: str
with os.scandir(conversion_dir) as transactions:
for transaction in transactions:
if not (transaction.name.startswith(UPLOAD_DELETION_TRANSACTION_PREFIX) and transaction.name.endswith(UPLOAD_STAGING_SUFFIX) and transaction.is_dir(follow_symlinks=False)):
try:
is_transaction_dir = transaction.is_dir(follow_symlinks=False)
except FileNotFoundError:
continue
if not (transaction.name.startswith(UPLOAD_DELETION_TRANSACTION_PREFIX) and transaction.name.endswith(UPLOAD_STAGING_SUFFIX) and is_transaction_dir):
continue
transaction_dir = Path(transaction.path)
journal_path = transaction_dir / _UPLOAD_DELETION_REMOTE_JOURNAL
commit_path = transaction_dir / _UPLOAD_DELETION_COMMIT_MARKER
primary_dir = transaction_dir / _UPLOAD_DELETION_PRIMARY_DIRNAME
try:
journal_stat = os.lstat(journal_path)
commit_stat = os.lstat(commit_path)
primary_dir_stat = os.lstat(primary_dir)
except FileNotFoundError:
continue
if (
not stat.S_ISREG(journal_stat.st_mode)
or journal_stat.st_nlink != 1
or not stat.S_ISREG(commit_stat.st_mode)
or commit_stat.st_nlink != 1
or stat.S_ISLNK(primary_dir_stat.st_mode)
or not stat.S_ISDIR(primary_dir_stat.st_mode)
):
if stat.S_ISLNK(primary_dir_stat.st_mode) or not stat.S_ISDIR(primary_dir_stat.st_mode):
raise UnsafeUploadPathError("Unsafe pending remote upload deletion transaction")
with os.scandir(primary_dir) as primary_entries:
entries = list(primary_entries)
try:
with os.scandir(primary_dir) as primary_entries:
entries = list(primary_entries)
except FileNotFoundError:
# An unrelated transaction may be finalized while this name's
# lease is held. Its disappearance is a completed operation,
# not an upload failure.
continue
matching_entries = [entry for entry in entries if portable_name_coordination_key(entry.name) == coordination_key]
if not matching_entries:
continue
if len(entries) != 1:
raise UnsafeUploadPathError("Malformed pending remote upload deletion transaction")
entry = entries[0]
entry_stat = entry.stat(follow_symlinks=False)
entry = matching_entries[0]
try:
entry_stat = entry.stat(follow_symlinks=False)
journal_stat = os.lstat(journal_path)
commit_stat = os.lstat(commit_path)
except FileNotFoundError:
# Matching cleanup normally shares this name lease. Be robust
# to legacy cleanup paths: once the durable journal is gone,
# the old remote operation can no longer target a replacement.
try:
os.lstat(journal_path)
except FileNotFoundError:
continue
return True
if not stat.S_ISREG(journal_stat.st_mode) or journal_stat.st_nlink != 1 or not stat.S_ISREG(commit_stat.st_mode) or commit_stat.st_nlink != 1:
raise UnsafeUploadPathError("Unsafe pending remote upload deletion transaction")
if not stat.S_ISREG(entry_stat.st_mode) or entry_stat.st_nlink != 1:
raise UnsafeUploadPathError("Unsafe pending remote upload deletion tombstone")
if portable_name_coordination_key(entry.name) == coordination_key:
return True
return True
return False
@ -1483,9 +1500,11 @@ def delete_file_safe(
# generation.
_clear_staged_deletion_commit(staged_path)
abort_remote_delete = getattr(delete_remote_copy, "abort_prepared", None)
if callable(abort_remote_delete):
abort_remote_delete()
_restore_staged_deletion(staged_path, actual_file_path, identity)
try:
if callable(abort_remote_delete):
abort_remote_delete()
finally:
_restore_staged_deletion(staged_path, actual_file_path, identity)
raise
except RemoteDeletionCommitRequiredError:
# At least one remote mutation could not be rolled back. Restoring

View File

@ -3,6 +3,7 @@
from __future__ import annotations
import asyncio
import inspect
import json
import logging
import os
@ -14,6 +15,7 @@ from typing import Any
from deerflow.config.paths import get_paths
from deerflow.sandbox.sandbox_provider import (
SandboxReconciliationResult,
sandbox_provider_sandbox_uses_thread_data_mounts,
sandbox_provider_uses_thread_data_mounts,
sandbox_provider_uses_thread_data_mounts_async,
@ -112,6 +114,7 @@ class _SandboxDeletionHook:
sandbox_id: str | None = None
thread_id: str | None = None
user_id: str | None = None
provider_key: str | None = None
_journal_path: Path | None = None
_prepared_targets: tuple[tuple[str, bytes | None], ...] | None = None
@ -129,6 +132,8 @@ class _SandboxDeletionHook:
conversion_path: Path | None,
) -> None:
"""Persist enough context to retry an interrupted remote deletion."""
if not self.sandbox_id or not self.thread_id or not self.provider_key:
raise RuntimeError("Durable remote deletion requires sandbox, thread, and provider identities")
targets = (
(upload_virtual_path(filename), primary_path.read_bytes()),
(
@ -139,10 +144,11 @@ class _SandboxDeletionHook:
journal_path = _staged_deletion_remote_journal(primary_path)
payload = json.dumps(
{
"version": 1,
"version": 2,
"sandbox_id": self.sandbox_id,
"thread_id": self.thread_id,
"user_id": self.user_id,
"provider_key": self.provider_key,
"filename": filename,
"virtual_paths": [path for path, _bytes in targets],
},
@ -260,15 +266,67 @@ def _deletion_hook_for_sandbox(
sandbox_id: str | None = None,
thread_id: str | None = None,
user_id: str | None = None,
provider_key: str | None = None,
) -> _SandboxDeletionHook:
return _SandboxDeletionHook(
sandbox=sandbox,
sandbox_id=sandbox_id,
thread_id=thread_id,
user_id=user_id,
provider_key=provider_key,
)
def _declared_callable(instance: Any, name: str) -> Callable[..., Any] | None:
"""Return a real declared callable without trusting dynamic mock attributes."""
try:
inspect.getattr_static(instance, name)
except AttributeError:
return None
candidate = getattr(instance, name, None)
return candidate if callable(candidate) else None
def _sandbox_provider_reconciliation_key(sandbox_provider: Any) -> str:
"""Return the durable namespace that owns a provider's raw sandbox IDs."""
key_factory = _declared_callable(sandbox_provider, "reconciliation_provider_key")
if key_factory is None:
provider_type = type(sandbox_provider)
key = f"{provider_type.__module__}.{provider_type.__qualname__}"
else:
key = key_factory()
if not isinstance(key, str) or not key or len(key.encode("utf-8")) > 512:
raise ValueError("Sandbox reconciliation provider key must be a non-empty string of at most 512 bytes")
return key
def _reconnect_sandbox_for_reconciliation(
sandbox_provider: Any,
sandbox_id: str,
*,
thread_id: str,
user_id: str | None,
) -> SandboxReconciliationResult:
"""Resolve one exact old sandbox without creating or redirecting it."""
resolver = _declared_callable(sandbox_provider, "reconnect_sandbox_for_reconciliation")
if resolver is not None:
result = resolver(
sandbox_id,
thread_id=thread_id,
user_id=user_id,
)
else:
sandbox = sandbox_provider.get(sandbox_id)
result = SandboxReconciliationResult.found(sandbox) if sandbox is not None else SandboxReconciliationResult.unknown()
if not isinstance(result, SandboxReconciliationResult):
raise TypeError("Sandbox reconciliation returned an invalid result")
if result.status == "found" and result.sandbox is None:
raise ValueError("Found sandbox reconciliation result has no sandbox")
if result.status != "found" and (result.sandbox is not None or result.close_after):
raise ValueError("Non-found sandbox reconciliation result owns no sandbox")
return result
def _pending_remote_deletion_journals(base_dir: Path) -> list[Path]:
patterns = (
"threads/*/user-data/.upload-conversions/.upload-delete-*.part/.remote-delete.json",
@ -329,12 +387,13 @@ def reconcile_pending_remote_deletions(
try:
try:
data = _read_remote_delete_journal(journal_path)
if data.get("version") != 1:
if data.get("version") != 2:
raise ValueError("unsupported journal version")
filename = data.get("filename")
thread_id = data.get("thread_id")
user_id = data.get("user_id")
sandbox_id = data.get("sandbox_id")
provider_key = data.get("provider_key")
virtual_paths = data.get("virtual_paths")
expected_paths = [
upload_virtual_path(filename) if isinstance(filename, str) else None,
@ -347,6 +406,9 @@ def reconcile_pending_remote_deletions(
or (user_id is not None and not isinstance(user_id, str))
or not isinstance(sandbox_id, str)
or not sandbox_id
or not isinstance(provider_key, str)
or not provider_key
or len(provider_key.encode("utf-8")) > 512
or virtual_paths != expected_paths
):
raise ValueError("invalid remote deletion journal fields")
@ -356,6 +418,12 @@ def reconcile_pending_remote_deletions(
if sandbox_provider is None:
sandbox_provider = sandbox_provider_factory()
if _sandbox_provider_reconciliation_key(sandbox_provider) != provider_key:
logger.warning(
"Pending remote upload deletion belongs to provider namespace %r, not the active namespace; leaving journal pending",
provider_key,
)
continue
name_lease = UploadNameLease.try_acquire(
uploads_dir,
filename,
@ -364,30 +432,40 @@ def reconcile_pending_remote_deletions(
if name_lease is None:
continue
try:
sandbox = sandbox_provider.get(sandbox_id)
if sandbox is None:
acquired_sandbox_id = sandbox_provider.acquire(thread_id, user_id=user_id)
if acquired_sandbox_id != sandbox_id:
logger.warning(
"Pending remote upload deletion targets sandbox %r, but thread %r acquired %r; leaving journal pending",
sandbox_id,
thread_id,
acquired_sandbox_id,
)
continue
sandbox = sandbox_provider.get(sandbox_id)
if sandbox is None:
raise RuntimeError(f"Sandbox {sandbox_id!r} not found during upload deletion reconciliation")
pending = _SandboxDeletionHook(sandbox)._converge_to_deleted(tuple(virtual_paths))
if pending:
result = _reconnect_sandbox_for_reconciliation(
sandbox_provider,
sandbox_id,
thread_id=thread_id,
user_id=user_id,
)
if result.status == "unknown":
logger.warning(
"Remote upload deletion remains pending for %s: %s",
journal_path,
pending,
"Exact sandbox %r could not be resolved for pending remote upload deletion; leaving journal pending",
sandbox_id,
)
continue
_unlink_journal_durably(journal_path)
reconciled += 1
if result.status == "absent":
_unlink_journal_durably(journal_path)
reconciled += 1
continue
sandbox = result.sandbox
try:
pending = _SandboxDeletionHook(sandbox)._converge_to_deleted(tuple(virtual_paths))
if pending:
logger.warning(
"Remote upload deletion remains pending for %s: %s",
journal_path,
pending,
)
continue
_unlink_journal_durably(journal_path)
reconciled += 1
finally:
if result.close_after:
try:
sandbox.close()
except BaseException:
logger.warning("Failed to close a transient reconciliation sandbox client", exc_info=True)
finally:
name_lease.release()
except BaseException:
@ -423,6 +501,7 @@ def prepare_upload_deletion(
sandbox_id=sandbox_id,
thread_id=thread_id,
user_id=user_id,
provider_key=_sandbox_provider_reconciliation_key(sandbox_provider),
)
@ -452,6 +531,7 @@ async def prepare_upload_deletion_async(
sandbox_id=sandbox_id,
thread_id=thread_id,
user_id=user_id,
provider_key=_sandbox_provider_reconciliation_key(sandbox_provider),
)

View File

@ -310,6 +310,44 @@ def test_discover_returns_none_when_runtime_check_times_out(monkeypatch):
assert backend.discover("sandbox-timeout") is None
def test_reconciliation_discovery_confirms_exact_container_absent(monkeypatch):
backend = _backend_for_inspect_tests()
monkeypatch.setattr(backend, "_is_container_running", lambda _name: False)
result = backend.discover_for_reconciliation("missing")
assert result.status == "absent"
assert result.info is None
def test_reconciliation_discovery_preserves_runtime_uncertainty(monkeypatch):
backend = _backend_for_inspect_tests()
def fail(_name):
raise RuntimeError("daemon unavailable")
monkeypatch.setattr(backend, "_is_container_running", fail)
result = backend.discover_for_reconciliation("unknown")
assert result.status == "unknown"
assert result.info is None
def test_reconciliation_discovery_returns_exact_running_container(monkeypatch):
from deerflow.community.aio_sandbox.sandbox_info import SandboxInfo
backend = _backend_for_inspect_tests()
info = SandboxInfo(sandbox_id="existing", sandbox_url="http://localhost:18080")
monkeypatch.setattr(backend, "_is_container_running", lambda _name: True)
monkeypatch.setattr(backend, "discover", lambda sandbox_id: info if sandbox_id == "existing" else None)
result = backend.discover_for_reconciliation("existing")
assert result.status == "found"
assert result.info is info
def test_is_container_running_false_on_apple_container_not_found(monkeypatch):
"""Apple Container's generic "not found" is trusted when it names the container."""
backend = _backend_for_inspect_tests()

View File

@ -2024,3 +2024,74 @@ def test_reconcile_adopts_unready_container_when_no_teardown_is_in_flight(tmp_pa
provider._reconcile_orphans()
assert "adoptable" in provider._warm_pool, "reconcile must still adopt a genuinely unowned container"
def test_reconciliation_reconnects_exact_discovered_sandbox_without_acquiring(monkeypatch):
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
backend_mod = importlib.import_module("deerflow.community.aio_sandbox.backend")
info_mod = importlib.import_module("deerflow.community.aio_sandbox.sandbox_info")
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
info = info_mod.SandboxInfo(
sandbox_id="old-id",
sandbox_url="http://sandbox-old",
thread_id="thread-1",
user_id="alice",
)
provider._backend = SimpleNamespace(
discover_for_reconciliation=lambda sandbox_id: backend_mod.SandboxDiscoveryResult.found(info),
)
provider.get = lambda _sandbox_id: None
created: list[tuple[str, str]] = []
class TransientSandbox:
def __init__(self, sandbox_id, sandbox_url):
created.append((sandbox_id, sandbox_url))
monkeypatch.setattr(aio_mod, "AioSandbox", TransientSandbox)
result = provider.reconnect_sandbox_for_reconciliation(
"old-id",
thread_id="thread-1",
user_id="alice",
)
assert result.status == "found"
assert result.close_after is True
assert created == [("old-id", "http://sandbox-old")]
def test_reconciliation_rejects_discovered_sandbox_from_another_identity():
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
backend_mod = importlib.import_module("deerflow.community.aio_sandbox.backend")
info_mod = importlib.import_module("deerflow.community.aio_sandbox.sandbox_info")
provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
info = info_mod.SandboxInfo(
sandbox_id="old-id",
sandbox_url="http://sandbox-old",
thread_id="another-thread",
user_id="alice",
)
provider._backend = SimpleNamespace(
discover_for_reconciliation=lambda sandbox_id: backend_mod.SandboxDiscoveryResult.found(info),
)
provider.get = lambda _sandbox_id: None
result = provider.reconnect_sandbox_for_reconciliation(
"old-id",
thread_id="thread-1",
user_id="alice",
)
assert result.status == "unknown"
assert result.sandbox is None
def test_reconciliation_provider_key_changes_with_remote_backend_namespace():
aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
remote_mod = importlib.import_module("deerflow.community.aio_sandbox.remote_backend")
first = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
second = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
first._backend = remote_mod.RemoteSandboxBackend("http://provisioner-a:8002")
second._backend = remote_mod.RemoteSandboxBackend("http://provisioner-b:8002")
assert first.reconciliation_provider_key() != second.reconciliation_provider_key()

View File

@ -51,6 +51,54 @@ def test_list_running_delegates_to_provisioner_list(monkeypatch):
assert backend.list_running() == [sandbox_info]
def test_reconciliation_discovery_confirms_provisioner_absence(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002")
monkeypatch.setattr(requests, "get", lambda *_args, **_kwargs: _StubResponse(status_code=404))
result = backend.discover_for_reconciliation("old-id")
assert result.status == "absent"
assert result.info is None
def test_reconciliation_discovery_preserves_transport_uncertainty(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002")
def fail(*_args, **_kwargs):
raise requests.ConnectionError("offline")
monkeypatch.setattr(requests, "get", fail)
result = backend.discover_for_reconciliation("old-id")
assert result.status == "unknown"
assert result.info is None
def test_reconciliation_discovery_accepts_exact_legacy_mount_contract(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002")
monkeypatch.setattr(
requests,
"get",
lambda *_args, **_kwargs: _StubResponse(
payload={
"sandbox_id": "old-id",
"sandbox_url": "http://sandbox-old",
"thread_id": "thread-1",
"user_id": "alice",
"mount_contract_version": 1,
}
),
)
result = backend.discover_for_reconciliation("old-id")
assert result.status == "found"
assert result.info is not None
assert result.info.sandbox_id == "old-id"
assert result.info.mount_contract_version == 1
def test_provisioner_list_returns_sandbox_infos_and_filters_invalid_entries(monkeypatch):
backend = RemoteSandboxBackend("http://provisioner:8002")

View File

@ -15,6 +15,7 @@ from deerflow.uploads.manager import (
)
from deerflow.uploads.sandbox_sync import (
_deletion_hook_for_sandbox,
_sandbox_provider_reconciliation_key,
make_upload_paths_available,
make_upload_paths_available_async,
prepare_upload_deletion,
@ -111,6 +112,7 @@ def test_unconfirmed_remote_delete_is_persisted_and_retried_after_restart(tmp_pa
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(Provider(sandbox)),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
@ -166,6 +168,7 @@ def test_remote_delete_finalize_failure_never_restores_deleted_host_generation(t
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key="tests.Provider",
)
real_unlink = sandbox_sync_module._unlink_journal_durably
@ -191,6 +194,64 @@ def test_remote_delete_finalize_failure_never_restores_deleted_host_generation(t
assert cleanup_stale_upload_staging_files(tmp_path) == 0
@pytest.mark.parametrize("unlink_first", [False, True], ids=["unlink", "directory-fsync"])
def test_compensated_remote_delete_restores_host_even_when_journal_abort_fails(tmp_path, unlink_first):
import deerflow.uploads.sandbox_sync as sandbox_sync_module
primary_virtual = upload_virtual_path("report.pdf")
conversion_virtual = conversion_virtual_path("report.pdf")
class Sandbox:
def __init__(self) -> None:
self.files = {
primary_virtual: b"pdf",
conversion_virtual: b"generated",
}
self.failed = False
def remove_file(self, path: str) -> None:
self.files.pop(path, None)
if path == conversion_virtual and not self.failed:
self.failed = True
raise OSError("conversion deletion failed")
def update_file(self, path: str, content: bytes) -> None:
self.files[path] = content
uploads = tmp_path / "users" / "alice" / "threads" / "thread-1" / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"pdf")
conversion = conversion_path_for_upload(primary)
conversion.parent.mkdir()
conversion.write_bytes(b"generated")
sandbox = Sandbox()
deletion_hook = _deletion_hook_for_sandbox(
sandbox,
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key="tests.Provider",
)
real_unlink = sandbox_sync_module._unlink_journal_durably
def fail_abort(journal_path):
if unlink_first:
real_unlink(journal_path)
raise OSError("remote deletion journal abort failed")
with patch.object(sandbox_sync_module, "_unlink_journal_durably", side_effect=fail_abort):
with pytest.raises(OSError, match="journal abort failed"):
delete_file_safe(uploads, primary.name, delete_remote_copy=deletion_hook)
assert sandbox.files == {
primary_virtual: b"pdf",
conversion_virtual: b"generated",
}
assert primary.read_bytes() == b"pdf"
assert conversion.read_bytes() == b"generated"
def test_pending_remote_deletion_reserves_name_until_exact_remote_generation_is_gone(tmp_path):
old_primary_virtual = upload_virtual_path("report.pdf")
old_conversion_virtual = conversion_virtual_path("report.pdf")
@ -234,6 +295,7 @@ def test_pending_remote_deletion_reserves_name_until_exact_remote_generation_is_
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(Provider(sandbox)),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
@ -308,19 +370,22 @@ def test_reconciliation_never_redirects_old_journal_to_new_sandbox(tmp_path):
conversion = conversion_path_for_upload(primary)
conversion.parent.mkdir()
conversion.write_bytes(b"old conversion")
provider = RollingProvider(new_sandbox=None)
deletion_hook = _deletion_hook_for_sandbox(
OldSandbox(),
sandbox_id="sandbox-old",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(provider),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
delete_file_safe(uploads, primary.name, delete_remote_copy=deletion_hook)
new_sandbox = NewSandbox()
provider.new_sandbox = new_sandbox
assert (
reconcile_pending_remote_deletions(
sandbox_provider_factory=lambda: RollingProvider(new_sandbox),
sandbox_provider_factory=lambda: provider,
base_dir=tmp_path,
)
== 0
@ -333,6 +398,130 @@ def test_reconciliation_never_redirects_old_journal_to_new_sandbox(tmp_path):
assert cleanup_stale_upload_staging_files(tmp_path) == 0
def test_reconciliation_never_crosses_provider_boundary_when_raw_id_matches(tmp_path):
primary_virtual = upload_virtual_path("report.pdf")
conversion_virtual = conversion_virtual_path("report.pdf")
class OldProvider:
pass
class OldSandbox:
def remove_file(self, path: str) -> None:
if path == conversion_virtual:
raise OSError("old provider unavailable")
def update_file(self, _path: str, _content: bytes) -> None:
raise OSError("old provider compensation unavailable")
class NewSandbox:
def __init__(self) -> None:
self.files = {
primary_virtual: b"new primary",
conversion_virtual: b"new conversion",
}
def remove_file(self, path: str) -> None:
self.files.pop(path, None)
class NewProvider:
uses_thread_data_mounts = False
def __init__(self, sandbox) -> None:
self.sandbox = sandbox
def get(self, sandbox_id: str):
assert sandbox_id == "same-id"
return None
def acquire(self, thread_id: str, *, user_id: str | None = None) -> str:
assert (thread_id, user_id) == ("thread-1", "alice")
return "same-id"
uploads = tmp_path / "users" / "alice" / "threads" / "thread-1" / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"old primary")
conversion = conversion_path_for_upload(primary)
conversion.parent.mkdir()
conversion.write_bytes(b"old conversion")
deletion_hook = _deletion_hook_for_sandbox(
OldSandbox(),
sandbox_id="same-id",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(OldProvider()),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
delete_file_safe(uploads, primary.name, delete_remote_copy=deletion_hook)
new_sandbox = NewSandbox()
assert (
reconcile_pending_remote_deletions(
sandbox_provider_factory=lambda: NewProvider(new_sandbox),
base_dir=tmp_path,
)
== 0
)
assert new_sandbox.files == {
primary_virtual: b"new primary",
conversion_virtual: b"new conversion",
}
assert len(list(conversion.parent.glob(".upload-delete-*.part/.remote-delete.json"))) == 1
def test_reconciliation_finalizes_when_provider_confirms_exact_sandbox_absent(tmp_path):
conversion_virtual = conversion_virtual_path("report.pdf")
class Provider:
def reconnect_sandbox_for_reconciliation(
self,
sandbox_id: str,
*,
thread_id: str,
user_id: str | None,
):
assert (sandbox_id, thread_id, user_id) == ("sandbox-old", "thread-1", "alice")
from deerflow.sandbox.sandbox_provider import SandboxReconciliationResult
return SandboxReconciliationResult.absent()
class OldSandbox:
def remove_file(self, path: str) -> None:
if path == conversion_virtual:
raise OSError("old sandbox unavailable")
def update_file(self, _path: str, _content: bytes) -> None:
raise OSError("old sandbox compensation unavailable")
provider = Provider()
uploads = tmp_path / "users" / "alice" / "threads" / "thread-1" / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"old primary")
conversion = conversion_path_for_upload(primary)
conversion.parent.mkdir()
conversion.write_bytes(b"old conversion")
deletion_hook = _deletion_hook_for_sandbox(
OldSandbox(),
sandbox_id="sandbox-old",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(provider),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
delete_file_safe(uploads, primary.name, delete_remote_copy=deletion_hook)
assert (
reconcile_pending_remote_deletions(
sandbox_provider_factory=lambda: provider,
base_dir=tmp_path,
)
== 1
)
assert cleanup_stale_upload_staging_files(tmp_path) == 1
assert not list(conversion.parent.glob(".upload-delete-*.part"))
def test_reconciliation_rejects_journal_that_does_not_match_retained_generation(tmp_path):
report_virtual = upload_virtual_path("report.pdf")
report_conversion_virtual = conversion_virtual_path("report.pdf")
@ -380,6 +569,7 @@ def test_reconciliation_rejects_journal_that_does_not_match_retained_generation(
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(Provider(sandbox)),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
delete_file_safe(uploads, primary.name, delete_remote_copy=deletion_hook)
@ -448,6 +638,7 @@ def test_reconciliation_accepts_exact_legacy_posix_filename(tmp_path, filename):
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(Provider(sandbox)),
)
with pytest.raises(RemoteDeletionCommitRequiredError):
@ -488,6 +679,7 @@ def test_prepared_remote_journal_without_commit_marker_restores_host(tmp_path):
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key="tests.Provider",
)
hook.prepare(primary.name, staged_path, upload_manager_module._staged_conversion_path(staged_path))
@ -560,6 +752,7 @@ def test_reconciliation_finishes_crash_after_first_remote_delete(tmp_path):
sandbox_id="sandbox-1",
thread_id="thread-1",
user_id="alice",
provider_key=_sandbox_provider_reconciliation_key(Provider(sandbox)),
)
hook.prepare(
primary.name,

View File

@ -446,6 +446,57 @@ class TestUploadPublication:
finally:
publication.release()
def test_unrelated_publication_survives_pending_deletion_finalization(self, tmp_path):
import deerflow.uploads.manager as upload_manager_module
uploads = tmp_path / "users" / "alice" / "threads" / "thread-1" / "user-data" / "uploads"
uploads.mkdir(parents=True)
primary = uploads / "report.pdf"
primary.write_bytes(b"old")
identity = UploadIdentity.from_path(primary)
staged_path, stage_lease = upload_manager_module._stage_primary_deletion(
uploads,
primary,
identity,
recover_on_crash=True,
)
upload_manager_module._mark_staged_deletion_committed(staged_path)
journal = upload_manager_module._staged_deletion_remote_journal(staged_path)
journal.write_text("{}", encoding="utf-8")
transaction_dir = staged_path.parent.parent
primary_dir = staged_path.parent
stage_lease.release()
reached_primary_scan = threading.Event()
resume_primary_scan = threading.Event()
real_scandir = upload_manager_module.os.scandir
paused = False
pause_lock = threading.Lock()
def pause_first_primary_scan(path):
nonlocal paused
should_pause = False
with pause_lock:
if Path(path) == primary_dir and not paused:
paused = True
should_pause = True
if should_pause:
reached_primary_scan.set()
assert resume_primary_scan.wait(2)
return real_scandir(path)
with patch.object(upload_manager_module.os, "scandir", side_effect=pause_first_primary_scan):
with ThreadPoolExecutor(max_workers=1) as pool:
publication = pool.submit(publish_upload_bytes, uploads, "notes.txt", b"new")
assert reached_primary_scan.wait(2)
journal.unlink()
assert cleanup_stale_upload_staging_files(tmp_path) == 1
assert not transaction_dir.exists()
resume_primary_scan.set()
assert publication.result(timeout=2) == uploads / "notes.txt"
assert (uploads / "notes.txt").read_bytes() == b"new"
def test_held_existing_name_does_not_block_next_collision_candidate(self, tmp_path):
first = publish_upload_bytes_leased(tmp_path, "report.pdf", b"first")
pool = ThreadPoolExecutor(max_workers=1)