fix(sandbox): restore saved thread files when creating E2B sandboxes (#6824)

* fix(sandbox): restore saved thread files when creating E2B sandboxes

After an E2B sandbox expires between turns, a replacement starts without
the thread files already saved by the Gateway, so later reads fail.

Restore owner-scoped workspace, uploads, outputs and ACP files before a
new sandbox becomes available. Keep live sandbox reuse unchanged, share
the existing upload budget, and discard incomplete sandboxes on restore
failure. Skip links and unpublished temporary files during restoration.

Add offline regressions for release, expiry and recreation, sync and async
acquisition, owner isolation, live reuse, upload limits and failure cleanup.

* fix(sandbox): make thread restore failures actionable

* fix(sandbox): use neutral upload limit diagnostics

---------

Co-authored-by: conghua1013 <16472796+conghua1013@users.noreply.github.com>

README.md

@@ -2301,11 +2301,29 @@ changing the root.
 E2B acquisition uses a bounded executor. Waiting acquisitions do not use the
 default asyncio executor.
 
+When an E2B sandbox expires between conversation turns, its replacement restores
+the same user's saved thread files from `workspace`, `uploads`, `outputs`, and
+the ACP workspace before tools can use it. Live sandbox reuse does not overwrite
+remote files with older saved copies. Restoration failure or an exceeded upload
+limit rejects the new sandbox. Transient failures may succeed on a later attempt.
+Capacity breaches report the resource tree and exceeded cap: 100 MiB per file,
+2,000 files / 512 MiB per tree, and 2,000 files / 512 MiB across the creation pass.
+An operator must archive or remove unneeded files from the Gateway's persisted
+thread directories to fit those limits before retrying; retrying unchanged data
+cannot resolve a capacity breach. Deleting only the remote copies is insufficient.
+Other restoration errors identify the resource tree and cause category; Gateway
+logs retain the underlying exception for diagnosis.
+Recovery requires the Gateway's persisted thread directories to remain available;
+files that never synchronized before the VM disappeared cannot be recovered.
+The existing release sync saves workspace/output files but does not mirror file
+deletions, so saved copies of remotely deleted files can reappear on recreation.
+
 Each E2B mount upload pass accepts at most 512 MiB and 2,000 files. The pass
-also has a cooperative 120-second deadline. Skill projections and configured
-mounts share these limits. The provider checks the deadline before each mount
-and during directory preflight. The deadline stops new file uploads after it
-expires. It does not interrupt active filesystem or E2B SDK calls.
+also has a cooperative 120-second deadline. Thread restoration runs first and
+shares these limits with skill projections and configured mounts. The provider
+checks the deadline before each mount and during directory preflight. The deadline
+stops new file uploads after it expires. It does not interrupt active filesystem
+or E2B SDK calls.
 
 An E2B VM keeps its slot until E2B confirms destruction. This rule covers
 create and reclaim operations. Discovery can find a VM from another Gateway.

backend/packages/harness/deerflow/community/e2b_sandbox/AGENTS.md

@@ -1,5 +1,24 @@
 # E2B lifecycle
 
+Fresh VM creation restores owner/thread-scoped workspace, uploads, outputs and
+ACP files from Gateway-visible `Paths` directories before registering the VM.
+Do not use `host_*` paths: those address the Docker host, not Gateway storage.
+Restoration is required and shares the creation upload budget with optional
+skills and mounts. Errors discard the VM through ownership-fenced cleanup and
+retain capacity if destruction is unconfirmed. Active, warm and discovered VM
+reuse must never replay persisted files over live contents.
+Reject symlinked resource roots/ancestors; skip nested symlinks, special files,
+upload staging files and output-sync temporary files. Directory scan failures
+must propagate rather than masquerading as empty directories. Release sync is
+still copy-only; deletion reconciliation is a separate contract.
+Capacity breaches raise `_ThreadResourceRestoreLimitError` with the logical tree,
+cap and operator remediation; deadlines remain ordinary restore errors. Do not
+silently truncate required resources or imply unchanged over-limit data can succeed
+on retry. Log restore failures with owner/thread context and the exception chain;
+caller-facing errors identify the logical tree without exposing host paths.
+Regressions: `tests/test_e2b_sandbox_provider.py` exercises release, expiry and
+recreation using saved bytes, plus isolation, reuse and failed restoration.
+
 Reconciliation floors active-client TTL renewals at the SDK's default timeout
 and twice the sum of the configured reconciliation interval and pass budget
 (subject to the E2B timeout cap). Release still uses the configured idle timeout.

backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py

@@ -45,6 +45,7 @@ import os
 import posixpath
 import shlex
 import signal
+import stat
 import threading
 import time
 import uuid
@@ -124,6 +125,18 @@ class _E2BMaintenanceShutdownTimeout(RuntimeError):
     """A bounded maintenance-worker join did not prove worker termination."""
 
 
+class _ThreadResourceRestoreError(RuntimeError):
+    """A fresh VM cannot be used until persisted thread files are restored."""
+
+
+class _ThreadResourceRestoreLimitError(_ThreadResourceRestoreError):
+    """Persisted data exceeds a capacity cap; unchanged retries cannot succeed."""
+
+
+class _MountTreeLimitExceeded(ValueError):
+    """A source tree exceeds a per-file or per-tree capacity cap."""
+
+
 # Recursive skill projection replacement must never target an operating-system
 # tree. The configured E2B home is handled separately: an isolated descendant
 # such as /home/user/skills is supported, while the home directory itself and
@@ -198,6 +211,10 @@ class _MountPassLimitExceeded(Exception):
     """Stop the current mount upload pass at its aggregate resource limit."""
 
 
+class _MountPassCapacityExceeded(_MountPassLimitExceeded):
+    """A mount upload pass exceeds its file or byte cap, rather than its deadline."""
+
+
 @dataclass(frozen=True)
 class MountUploadResult:
     """Structured outcome of a mount upload pass.
@@ -1354,6 +1371,10 @@ class E2BSandboxProvider(SandboxProvider):
         mount_result: MountUploadResult | None = None
         try:
             mount_result = self._apply_mounts(client, user_id=user_id, thread_id=thread_id)
+        except _ThreadResourceRestoreError:
+            remote_destroyed = self._discard_unusable_sandbox(client, sandbox_id)
+            self._complete_reserved_remote_op(sandbox_id, remote_destroyed=remote_destroyed)
+            raise
         except Exception as e:
             logger.warning("Failed to apply some mounts to e2b sandbox %s: %s", sandbox_id, e)
 
@@ -2020,22 +2041,24 @@ class E2BSandboxProvider(SandboxProvider):
             self._bootstrap_sandbox_paths(client)
         except Exception as e:
             logger.exception("Failed to bootstrap e2b sandbox %s. Discarding the unusable sandbox.", sandbox_id)
-            remote_destroyed = False
-            if self._claim_ownership(sandbox_id, for_destroy=True):
-                kill_error = self._kill_client(client)
-                remote_destroyed = kill_error is None
-                if kill_error:
-                    logger.warning("Failed to kill e2b sandbox %s after bootstrap failure: %s", sandbox_id, kill_error)
-                self._release_ownership(sandbox_id)
-            else:
-                logger.info(
-                    "Not killing E2B sandbox %s after bootstrap failure because a peer owns it",
-                    sandbox_id,
-                )
-            self._safe_close_client(client)
+            remote_destroyed = self._discard_unusable_sandbox(client, sandbox_id)
             return e, remote_destroyed
         return None, True
 
+    def _discard_unusable_sandbox(self, client: E2BClientSandbox, sandbox_id: str) -> bool:
+        """Clean up failed initialization using the existing ownership fence."""
+        remote_destroyed = False
+        if self._claim_ownership(sandbox_id, for_destroy=True):
+            kill_error = self._kill_client(client)
+            remote_destroyed = kill_error is None
+            if kill_error:
+                logger.warning("Failed to kill e2b sandbox %s after initialization failure: %s", sandbox_id, kill_error)
+            self._release_ownership(sandbox_id)
+        else:
+            logger.info("Not killing E2B sandbox %s after initialization failure because a peer owns it", sandbox_id)
+        self._safe_close_client(client)
+        return remote_destroyed
+
     def _bootstrap_sandbox_paths(self, client: E2BClientSandbox) -> None:
         """Materialise DeerFlow's virtual path layout inside the e2b VM.
 
@@ -2146,6 +2169,47 @@ class E2BSandboxProvider(SandboxProvider):
             logger.warning("Could not ensure skills projection for user %s: %s", user_id, exc, exc_info=True)
             return []
 
+    def _restore_thread_resources(self, client: E2BClientSandbox, *, thread_id: str, user_id: str, budget: _MountUploadBudget) -> None:
+        """Hydrate only fresh VMs from Gateway-visible, owner-scoped paths."""
+        from deerflow.config.paths import get_paths
+
+        paths = get_paths()
+        home_dir = self._config["home_dir"].rstrip("/") or DEFAULT_E2B_HOME_DIR
+        sources = [
+            (paths.sandbox_work_dir(thread_id, user_id=user_id), "workspace", False),
+            (paths.sandbox_uploads_dir(thread_id, user_id=user_id), "uploads", False),
+            (paths.sandbox_outputs_dir(thread_id, user_id=user_id), "outputs", False),
+            (paths.acp_workspace_dir(thread_id, user_id=user_id), "acp-workspace", True),
+        ]
+        for source, subdir, read_only in sources:
+            try:
+                budget.check_deadline()
+                # Reject linked roots/ancestors: a thread resource must never read
+                # another owner's directory or arbitrary Gateway files.
+                for part in (source, *source.parents):
+                    if part == paths.base_dir:
+                        break
+                    if part.is_symlink():
+                        raise ValueError("Thread resource directory must not be a symlink")
+                try:
+                    mode = source.stat().st_mode
+                except FileNotFoundError:
+                    continue
+                if not stat.S_ISDIR(mode):
+                    raise ValueError("Thread resource path must be a directory")
+                self._upload_tree(client, source, f"{home_dir}/{subdir}", read_only, budget=budget, thread_resource=True)
+            except (_MountTreeLimitExceeded, _MountPassCapacityExceeded) as exc:
+                raise _ThreadResourceRestoreLimitError(
+                    f"Failed to restore persisted thread resources: tree={subdir}; {exc}. "
+                    "Operator action required: archive or remove unneeded files from the persisted thread directories "
+                    "to fit the stated limit before retrying. Retrying unchanged data will fail again."
+                ) from exc
+            except Exception as exc:
+                # Exception text can contain host paths or SDK details. Keep those
+                # in the operator log, while exposing the tree and cause category.
+                reason = str(exc) if isinstance(exc, _MountPassLimitExceeded) else type(exc).__name__
+                raise _ThreadResourceRestoreError(f"Failed to restore persisted thread resources: tree={subdir}; {reason}. See Gateway logs for the underlying cause.") from exc
+
     def _apply_mounts(
         self,
         client: E2BClientSandbox,
@@ -2174,6 +2238,22 @@ class E2BSandboxProvider(SandboxProvider):
             )
 
         effective_user_id = user_id or get_effective_user_id()
+        # Required files go first and share the creation upload budget with
+        # optional skills/mounts. Never hand out a partially restored VM.
+        if thread_id is not None:
+            try:
+                self._restore_thread_resources(client, thread_id=thread_id, user_id=effective_user_id, budget=budget)
+            except Exception as exc:
+                logger.warning(
+                    "Failed to restore persisted thread resources for user=%s thread=%s: %s",
+                    effective_user_id,
+                    thread_id,
+                    exc,
+                    exc_info=True,
+                )
+                if isinstance(exc, _ThreadResourceRestoreError):
+                    raise
+                raise _ThreadResourceRestoreError(f"Failed to restore persisted thread resources to e2b sandbox: {type(exc).__name__}. See Gateway logs for the underlying cause.") from exc
         projection_mounts = self._skill_projection_mounts(effective_user_id, thread_id) if thread_id is not None else self._skill_projection_mounts(effective_user_id)
         configured_mounts = self._config.get("mounts") or []
         skills_root = self._config["skills_container_path"]
@@ -2626,6 +2706,7 @@ class E2BSandboxProvider(SandboxProvider):
         read_only: bool,
         *,
         budget: _MountUploadBudget | None = None,
+        thread_resource: bool = False,
     ) -> None:
         """Recursively upload ``src`` into ``dest_dir`` inside the sandbox."""
         started_at = time.monotonic()
@@ -2638,17 +2719,37 @@ class E2BSandboxProvider(SandboxProvider):
             if budget is not None:
                 budget.check_deadline()
             file_size = path.stat().st_size
+            display_path = path.relative_to(src) if thread_resource else path
+            display_source = "thread resource tree" if thread_resource else str(src)
             if file_size > _MAX_MOUNT_FILE_SIZE:
-                raise ValueError(f"Mount file {path} is {file_size} bytes and exceeds the {_MAX_MOUNT_FILE_SIZE}-byte file limit")
+                raise _MountTreeLimitExceeded(f"Mount file {display_path} is {file_size} bytes and exceeds the {_MAX_MOUNT_FILE_SIZE}-byte file limit")
             if len(files) >= _MAX_MOUNT_FILES:
-                raise ValueError(f"Mount {src} contains more than {_MAX_MOUNT_FILES} files and exceeds the {_MAX_MOUNT_FILES}-file limit")
+                raise _MountTreeLimitExceeded(f"Mount {display_source} contains more than {_MAX_MOUNT_FILES} files and exceeds the {_MAX_MOUNT_FILES}-file limit")
             total_size += file_size
             if total_size > _MAX_MOUNT_TOTAL_SIZE:
-                raise ValueError(f"Mount {src} is at least {total_size} bytes and exceeds the {_MAX_MOUNT_TOTAL_SIZE}-byte total limit")
+                raise _MountTreeLimitExceeded(f"Mount {display_source} is at least {total_size} bytes and exceeds the {_MAX_MOUNT_TOTAL_SIZE}-byte total limit")
             files.append((path, target, file_size))
 
         if source_is_file:
             add_file(src, f"{dest_dir}/{src.name}")
+        elif thread_resource:
+            from deerflow.uploads.manager import is_upload_staging_file
+
+            def scan_error(error: OSError) -> None:
+                raise error
+
+            for directory, dirnames, filenames in os.walk(src, followlinks=False, onerror=scan_error):
+                if budget is not None:
+                    budget.check_deadline()
+                parent = Path(directory)
+                dirnames[:] = [name for name in dirnames if not (parent / name).is_symlink()]
+                for name in filenames:
+                    path = parent / name
+                    if is_upload_staging_file(name) or name.endswith(".e2bsync.tmp"):
+                        continue
+                    if not stat.S_ISREG(path.lstat().st_mode):
+                        continue
+                    add_file(path, f"{dest_dir}/{path.relative_to(src).as_posix()}")
         else:
             for path in src.rglob("*"):
                 if budget is not None:
@@ -2663,9 +2764,9 @@ class E2BSandboxProvider(SandboxProvider):
                 if budget is not None:
                     budget.check_deadline()
                 if budget is not None and budget.attempted_files >= _MAX_MOUNT_PASS_FILES:
-                    raise _MountPassLimitExceeded(f"file count cap {_MAX_MOUNT_PASS_FILES}")
+                    raise _MountPassCapacityExceeded(f"mount upload pass file count cap {_MAX_MOUNT_PASS_FILES}")
                 if budget is not None and budget.attempted_bytes + expected_size > _MAX_MOUNT_PASS_TOTAL_BYTES:
-                    raise _MountPassLimitExceeded(f"total byte budget {_MAX_MOUNT_PASS_TOTAL_BYTES}")
+                    raise _MountPassCapacityExceeded(f"mount upload pass total byte budget {_MAX_MOUNT_PASS_TOTAL_BYTES}")
                 try:
                     make_dir = getattr(client.files, "make_dir", None)
                     if callable(make_dir):
@@ -2674,7 +2775,16 @@ class E2BSandboxProvider(SandboxProvider):
                             make_dir(parent)
                 except Exception:
                     pass
-                with path.open("rb") as fh:
+                if thread_resource:
+                    if path.is_symlink() or not path.resolve().is_relative_to(src.resolve()):
+                        raise ValueError("Thread resource changed to a symlink during upload")
+                    fd = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_BINARY", 0))
+                    stream = os.fdopen(fd, "rb")
+                else:
+                    stream = path.open("rb")
+                with stream as fh:
+                    if thread_resource and not stat.S_ISREG(os.fstat(fh.fileno()).st_mode):
+                        raise ValueError("Thread resource is no longer a regular file")
                     actual_size = os.fstat(fh.fileno()).st_size
                     if actual_size != expected_size:
                         raise ValueError(f"Mount file {path} changed during upload preflight")

backend/tests/test_e2b_sandbox_provider.py

@@ -3391,6 +3391,298 @@ def _setup_paths(monkeypatch, tmp_path):
     monkeypatch.setattr(paths_mod, "get_paths", lambda: Paths(base_dir=tmp_path), raising=False)
 
 
+@pytest.mark.parametrize("home_dir", ["/home/user", "/home/custom"])
+@pytest.mark.parametrize("async_acquire", [False, True])
+def test_recreated_sandbox_restores_saved_thread_files(monkeypatch, tmp_path, home_dir, async_acquire):
+    """A second turn must read saved bytes after the first VM expires."""
+    _setup_paths(monkeypatch, tmp_path)
+    provider = _make_provider()
+    provider._config["home_dir"] = home_dir
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    sdk = _install_fake_sdk(monkeypatch, provider)
+    clients = []
+
+    def create(**kwargs):
+        client = FakeClient(sandbox_id=f"vm-{len(clients)}")
+        clients.append(client)
+        return client
+
+    sdk.create_factory = create
+
+    def acquire():
+        if async_acquire:
+            with ThreadPoolExecutor(max_workers=1) as executor:
+                provider._acquire_executor = executor
+                return asyncio.run(provider.acquire_async("t1", user_id="u1"))
+        return provider.acquire("t1", user_id="u1")
+
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    (paths.sandbox_uploads_dir("t1", user_id="u1") / "input.bin").write_bytes(b"\x00\xff")
+    (paths.acp_workspace_dir("t1", user_id="u1") / "result.md").write_text("ACP result", encoding="utf-8")
+    # Neither another owner nor another thread may supply the restored bytes.
+    for user_id, thread_id in [("u2", "t1"), ("u1", "t2")]:
+        paths.ensure_thread_dirs(thread_id, user_id=user_id)
+        (paths.sandbox_outputs_dir(thread_id, user_id=user_id) / "private.txt").write_text("private", encoding="utf-8")
+
+    first_id = acquire()
+    first = provider.get(first_id)
+    contents = {"outputs/test.md": "# Saved output\n", "workspace/nested/test.md": "# Saved work\n"}
+    for relative, content in contents.items():
+        first.write_file(f"/mnt/user-data/{relative}", content)
+    listing = "".join(f"{len(content.encode())}\t2.000000000\t{home_dir}/{relative}\x00" for relative, content in contents.items())
+    clients[0].commands._responses.append(SimpleNamespace(stdout=listing, stderr="", exit_code=0))
+    provider.release(first_id)
+    for relative, content in contents.items():
+        assert (paths.thread_dir("t1", user_id="u1") / "user-data" / relative).read_text(encoding="utf-8") == content
+
+    def expired(*args, **kwargs):
+        raise RuntimeError(FakeCommandsAPI.NOT_FOUND_MSG)
+
+    sdk.connect_factory = expired
+    second_id = acquire()
+    assert second_id != first_id
+    second = provider.get(second_id)
+    for relative, content in contents.items():
+        assert second.read_file(f"/mnt/user-data/{relative}") == content
+    assert clients[1].files.store[f"{home_dir}/uploads/input.bin"] == b"\x00\xff"
+    assert clients[1].files.store[f"{home_dir}/acp-workspace/result.md"] == b"ACP result"
+    assert not any("private.txt" in path for path in clients[1].files.store)
+    assert second.mount_upload_result.completed_files == 4
+    provider.shutdown()
+
+
+@pytest.mark.parametrize("reuse", ["active", "warm", "discovered"])
+def test_thread_restore_does_not_overwrite_live_sandbox(monkeypatch, tmp_path, reuse):
+    _setup_paths(monkeypatch, tmp_path)
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    sdk = _install_fake_sdk(monkeypatch, provider)
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    (paths.sandbox_outputs_dir("t1", user_id="u1") / "test.md").write_text("stale", encoding="utf-8")
+    client = FakeClient(files=FakeFilesAPI({"/home/user/outputs/test.md": b"newer"}))
+    sdk.connect_factory = lambda *args, **kwargs: client
+    if reuse == "active":
+        provider._sandboxes[client.sandbox_id] = _make_sandbox(client, sandbox_id=client.sandbox_id)
+        provider._thread_sandboxes[provider._thread_key("t1", "u1")] = client.sandbox_id
+    elif reuse == "warm":
+        provider._warm_pool[client.sandbox_id] = (provider._stable_seed("t1", "u1"), time.time())
+    else:
+        sdk.list_return = [_info(client.sandbox_id, "u1", "t1")]
+    sandbox_id = provider.acquire("t1", user_id="u1")
+    assert provider.get(sandbox_id).read_file("/mnt/user-data/outputs/test.md") == "newer"
+    assert client.files.write_calls == []
+    provider.shutdown()
+
+
+@pytest.mark.parametrize("failure", ["write", "file_limit", "byte_limit", "single_file_limit", "tree_file_limit", "tree_byte_limit", "deadline"])
+def test_thread_restore_failure_discards_unusable_sandbox(monkeypatch, tmp_path, caplog, failure):
+    _setup_paths(monkeypatch, tmp_path)
+    mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    sdk = _install_fake_sdk(monkeypatch, provider)
+    client = FakeClient()
+    sdk.create_factory = lambda **kwargs: client
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    saved = paths.sandbox_outputs_dir("t1", user_id="u1") / "test.md"
+    saved.write_bytes(b"saved")
+    if failure == "write":
+        monkeypatch.setattr(client.files, "write", MagicMock(side_effect=OSError("upload failed")))
+    elif failure == "file_limit":
+        monkeypatch.setattr(mod, "_MAX_MOUNT_PASS_FILES", 0)
+    elif failure == "byte_limit":
+        monkeypatch.setattr(mod, "_MAX_MOUNT_PASS_TOTAL_BYTES", 1)
+    elif failure == "single_file_limit":
+        monkeypatch.setattr(mod, "_MAX_MOUNT_FILE_SIZE", 1)
+    elif failure == "tree_file_limit":
+        monkeypatch.setattr(mod, "_MAX_MOUNT_FILES", 0)
+    elif failure == "tree_byte_limit":
+        monkeypatch.setattr(mod, "_MAX_MOUNT_TOTAL_SIZE", 1)
+    else:
+        provider._config["mount_upload_deadline_seconds"] = 0
+    with pytest.raises(RuntimeError, match="restore.*thread") as error:
+        provider.acquire("t1", user_id="u1")
+    message = str(error.value)
+    assert str(tmp_path) not in message
+    assert "tree=" in message
+    if failure.endswith("limit"):
+        assert isinstance(error.value, mod._ThreadResourceRestoreLimitError)
+        assert "tree=outputs" in message
+        assert "Operator action required" in message
+        assert "Retrying unchanged data" in message
+        expected_limit = {
+            "file_limit": "file count cap 0",
+            "byte_limit": "total byte budget 1",
+            "single_file_limit": "1-byte file limit",
+            "tree_file_limit": "0-file limit",
+            "tree_byte_limit": "1-byte total limit",
+        }[failure]
+        assert expected_limit in message
+    else:
+        assert not isinstance(error.value, mod._ThreadResourceRestoreLimitError)
+        assert "Operator action required" not in message
+        assert ("OSError" if failure == "write" else "time budget 0s") in message
+    records = [r for r in caplog.records if "Failed to restore" in r.message]
+    assert len(records) == 1
+    assert records[0].exc_info is not None
+    assert "u1" in records[0].message and "t1" in records[0].message
+    if failure == "write":
+        assert "upload failed" in caplog.text
+    assert client.killed and client.closed
+    assert not provider._sandboxes
+    assert not provider._thread_sandboxes
+    assert not provider._remote_ops_in_progress
+    assert provider._reserved_slots == 0
+    assert saved.read_bytes() == b"saved"
+    provider.shutdown()
+
+
+@pytest.mark.parametrize("cap", ["_MAX_MOUNT_PASS_FILES", "_MAX_MOUNT_PASS_TOTAL_BYTES"])
+def test_thread_restore_shared_limit_requires_saved_data_remediation(monkeypatch, tmp_path, cap):
+    _setup_paths(monkeypatch, tmp_path)
+    mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
+    monkeypatch.setattr(mod, cap, 1)
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    sdk = _install_fake_sdk(monkeypatch, provider)
+    clients = []
+
+    def create(**kwargs):
+        client = FakeClient(sandbox_id=f"restore-{len(clients)}")
+        clients.append(client)
+        return client
+
+    sdk.create_factory = create
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    first = paths.sandbox_work_dir("t1", user_id="u1") / "first.txt"
+    first.write_bytes(b"1")
+    (paths.sandbox_outputs_dir("t1", user_id="u1") / "second.txt").write_bytes(b"2")
+    try:
+        for _ in range(2):
+            with pytest.raises(mod._ThreadResourceRestoreLimitError, match="tree=outputs"):
+                provider.acquire("t1", user_id="u1")
+            assert clients[-1].killed and clients[-1].closed
+            assert not provider._sandboxes
+            assert provider._reserved_slots == 0
+        # Only changing the persisted data resolves a deterministic budget breach.
+        first.unlink()
+        sandbox_id = provider.acquire("t1", user_id="u1")
+        assert provider.get(sandbox_id) is not None
+        assert clients[-1].files.write_calls == [("/home/user/outputs/second.txt", b"2")]
+    finally:
+        provider.shutdown()
+
+
+def test_thread_restore_skips_links_and_unpublished_files(monkeypatch, tmp_path):
+    from support.symlinks import symlink_or_skip
+
+    _setup_paths(monkeypatch, tmp_path)
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    uploads = paths.sandbox_uploads_dir("t1", user_id="u1")
+    (uploads / "ready.txt").write_bytes(b"ready")
+    (uploads / ".upload-inflight.part").write_bytes(b"partial")
+    (uploads / "result.e2bsync.tmp").write_bytes(b"partial")
+    outside = tmp_path / "private"
+    outside.mkdir()
+    (outside / "secret.txt").write_bytes(b"secret")
+    symlink_or_skip(uploads / "linked.txt", outside / "secret.txt")
+    symlink_or_skip(uploads / "linked-dir", outside, target_is_directory=True)
+    client = FakeClient()
+
+    result = provider._apply_mounts(client, user_id="u1", thread_id="t1")
+
+    assert client.files.write_calls == [("/home/user/uploads/ready.txt", b"ready")]
+    assert result.completed_files == 1
+
+
+@pytest.mark.parametrize("root", ["user", "thread", "workspace"])
+def test_thread_restore_rejects_linked_resource_roots(monkeypatch, tmp_path, root):
+    from support.symlinks import symlink_or_skip
+
+    _setup_paths(monkeypatch, tmp_path)
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    sdk = _install_fake_sdk(monkeypatch, provider)
+    client = FakeClient()
+    sdk.create_factory = lambda **kwargs: client
+    paths = Paths(base_dir=tmp_path)
+    roots = {
+        "user": paths.user_dir("u1"),
+        "thread": paths.thread_dir("t1", user_id="u1"),
+        "workspace": paths.sandbox_work_dir("t1", user_id="u1"),
+    }
+    source = roots[root]
+    source.parent.mkdir(parents=True, exist_ok=True)
+    outside = tmp_path / "private"
+    outside.mkdir()
+    symlink_or_skip(source, outside, target_is_directory=True)
+    with pytest.raises(RuntimeError, match="restore.*thread"):
+        provider.acquire("t1", user_id="u1")
+    assert client.files.write_calls == []
+    assert client.killed and client.closed
+    provider.shutdown()
+
+
+def test_thread_restore_shares_budget_with_optional_mounts(monkeypatch, tmp_path):
+    _setup_paths(monkeypatch, tmp_path)
+    mod = importlib.import_module("deerflow.community.e2b_sandbox.e2b_sandbox_provider")
+    monkeypatch.setattr(mod, "_MAX_MOUNT_PASS_FILES", 1)
+    provider = _make_provider()
+    monkeypatch.setattr(provider, "_skill_projection_mounts", lambda *args: [])
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+    (paths.sandbox_outputs_dir("t1", user_id="u1") / "saved.txt").write_bytes(b"saved")
+    optional = tmp_path / "optional.txt"
+    optional.write_bytes(b"optional")
+    provider._config["mounts"] = [SimpleNamespace(host_path=str(optional), container_path="/mnt/extra", read_only=False)]
+    client = FakeClient()
+
+    result = provider._apply_mounts(client, user_id="u1", thread_id="t1")
+
+    assert client.files.write_calls == [("/home/user/outputs/saved.txt", b"saved")]
+    assert result.truncated
+    assert result.completed_files == 1
+
+
+@pytest.mark.parametrize(
+    "tree,path_method",
+    [
+        ("workspace", "sandbox_work_dir"),
+        ("uploads", "sandbox_uploads_dir"),
+        ("outputs", "sandbox_outputs_dir"),
+        ("acp-workspace", "acp_workspace_dir"),
+    ],
+)
+def test_thread_restore_scan_error_is_not_an_empty_directory(monkeypatch, tmp_path, caplog, tree, path_method):
+    _setup_paths(monkeypatch, tmp_path)
+    provider = _make_provider()
+    paths = Paths(base_dir=tmp_path)
+    paths.ensure_thread_dirs("t1", user_id="u1")
+
+    unreadable_root = getattr(paths, path_method)("t1", user_id="u1")
+    unreadable_root.mkdir(parents=True, exist_ok=True)
+    original_walk = os.walk
+
+    def unreadable(source, *, followlinks, onerror):
+        if source == unreadable_root:
+            onerror(PermissionError(13, "unreadable directory", str(source)))
+        return original_walk(source, followlinks=followlinks, onerror=onerror)
+
+    monkeypatch.setattr(os, "walk", unreadable)
+    with pytest.raises(RuntimeError, match=f"tree={tree}.*PermissionError") as error:
+        provider._apply_mounts(FakeClient(), user_id="u1", thread_id="t1")
+    assert isinstance(error.value.__cause__, PermissionError)
+    assert str(tmp_path) not in str(error.value)
+    assert "unreadable directory" in caplog.text
+
+
 def test_sync_outputs_to_host_writes_new_files(monkeypatch, tmp_path):
     p = _make_provider()
     _setup_paths(monkeypatch, tmp_path)