mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
b9a75b869d
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes Why this change: Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker capability probe collapsed "provisioner unreachable" into "broker not configured": one 1.5s timeout cached a negative for 300s, so new sandboxes were created without the broker sidecar while the per-user credential directories were still bind-mounted into the container, and existing Pods kept being reused without their actual mode ever being read. Second, an implicit-shell command that timed out or returned an unknown status only marked the current client object; warm-pool reclaim and rediscovery built a fresh client that re-entered a container whose shell state could not be proven clean, and nothing about that ambiguity survived a Gateway restart. What changed: - The capability cache stores confirmed observations only, keyed per provisioner endpoint/auth identity; an unknown answer (timeout, malformed payload) raises instead of selecting credential placement, while a confirmed broker requirement is retained through outages. One in-flight probe per identity shares success and failure, and a Pod-attested mode contradicting the cache forces one shared refresh. - Execution uses the mode attested for the actual sandbox: SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod observation into the client, so a probe outage delays drift detection but never denies reuse of an attested runtime, and never trusts an unattested one. - The provisioner reports each Pod's actual broker mode, generation (Pod UID) and shell capacity from one read, rejects lark requests conflicting with its configuration (409/503 carrying a capability-refresh marker the Gateway uses to drop the stale cache entry), rejects conflicting or Terminating Pods on coordination creates while reporting Terminating as data to status/get/list, and answers Pod presence independently of Service readiness. - A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by scope/sandbox/generation) fences containers after uncertain implicit shell outcomes across client rebuilds and restarts; in-process reuse, warm reclaim, discovery, create responses and orphan reconciliation all pass one policy gate, and replacement runs under the existing ownership and teardown fences. A fenced sandbox still tracked after a failed release-recycle is destroyed inline on the next acquire. - Quarantine records retire only after confirmed absence, retrying partial cleanup for the quarantined generation and releasing remembered ports under the same fences; local stop failures now propagate instead of being logged away, and remote destroy errors surface to callers. Problem solved: A provisioner blip could silently strip broker isolation from new lark sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted default shells persist across warm reuse, rediscovery and restarts. Docs updated: README, aio_sandbox and skills AGENTS.md. * docs(changelog): record sandbox attestation and quarantine fix (#6436) * fix(sandbox): preserve active leases and fail closed on unattested broker modes Concurrent uploads could recycle a healthy AIO runtime while another holder was creating or cleaning up a command session. Older provisioner responses could also trigger destructive replacement before failing admission. - Distinguish pending session creation from ambiguous outcomes and reserve idle teardown until execution and upload holders finish scope cleanup. - Reject stale fork lookups and reconcile cancelled acquires without releasing a sandbox still borrowed by another holder. - Default Lark broker mode to unverified and preserve unattested Pods until the provisioner is upgraded. - Recognize Docker 29 absence diagnostics while matching resource names exactly. - Preserve Gateway upload errors and cover admission and lifecycle regressions. * fix(sandbox): scope the unverified-broker refuse path to attested providers The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox without a lark_cli_broker attribute as unverified, but only AioSandbox carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no broker sidecar, no credential mounts, and no provisioner to upgrade, so every lark-cli command on those providers failed with an unactionable error. Declare the tri-state on the Sandbox base class (defaulting to False, the only mode brokerless providers have) so the None fail-closed path stays scoped to AioSandbox's deliberate unverified default. - Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring the persistent_shell_sessions tri-state pattern - Add a regression test pinning the non-broker overlay for a provider without the attribute * fix(sandbox): close review gaps in broker attestation and teardown lifecycle Address the latest #6436 review round: - The provisioner's post-build broker-mode 409 now carries the capability-refresh marker, so an incompatible-mode conflict surfaced by idempotent re-create, attestation drift or post-create verification drops the Gateway's cached capability observation instead of repeating the same 409 for the full negative-cache TTL. - AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False when teardown is deferred because an execution/upload holder or a peer teardown is still active, so callers can no longer mistake a deferred destroy for a completed one. - A live replacement under the same logical ID proves the fenced old generations are absent; retire_replaced_generations prunes only those stale records so they cannot strand future creates, while the live generation's fence and unversioned crash fences are never touched. - SandboxInfo.from_dict keeps a missing lark_cli_broker attestation unverified (None) instead of silently rehydrating as attested non-broker. - The restricted-network inspect path shares the strict _is_no_such_network_error parser, closing the substring loophole for context-name collisions and wrong-target errors. * docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate The three lease notes added in 26c307f3 pushed the effective guidance chain to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance (AG002). Fold them into two lines carrying the same rules. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
93 lines
4.2 KiB
Python
93 lines
4.2 KiB
Python
"""Quarantine follows a container generation across provider restarts."""
|
|
|
|
import pytest
|
|
|
|
from deerflow.community.aio_sandbox.quarantine import SandboxQuarantine
|
|
from deerflow.community.aio_sandbox.sandbox_info import SandboxInfo
|
|
|
|
|
|
def test_quarantine_survives_restart_and_does_not_taint_replacement(tmp_path):
|
|
old = SandboxInfo("thread-sandbox", "http://sandbox", container_id="old-generation")
|
|
replacement = SandboxInfo("thread-sandbox", "http://sandbox", container_id="new-generation")
|
|
SandboxQuarantine(tmp_path, "provisioner-a").mark(old)
|
|
|
|
restarted = SandboxQuarantine(tmp_path, "provisioner-a")
|
|
assert restarted.contains(old)
|
|
assert not restarted.contains(replacement)
|
|
assert not SandboxQuarantine(tmp_path, "provisioner-b").contains(old)
|
|
|
|
|
|
def test_discovery_without_generation_cannot_bypass_quarantine(tmp_path):
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
store.mark(SandboxInfo("thread-sandbox", "http://sandbox", container_id="old-generation"))
|
|
|
|
assert store.contains(SandboxInfo("thread-sandbox", "http://sandbox"))
|
|
assert not store.contains(SandboxInfo("another-thread", "http://sandbox"))
|
|
|
|
|
|
def test_quarantine_storage_failure_is_not_treated_as_safe(tmp_path, monkeypatch):
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
|
|
def unavailable(*_args, **_kwargs):
|
|
raise PermissionError("quarantine storage unavailable")
|
|
|
|
monkeypatch.setattr(type(tmp_path), "stat", unavailable)
|
|
with pytest.raises(PermissionError):
|
|
store.contains(SandboxInfo("thread-sandbox", "http://sandbox", container_id="generation"))
|
|
|
|
|
|
def test_crash_residue_between_directory_and_record_still_fences(tmp_path):
|
|
# mark() mkdirs the per-sandbox directory before writing the generation
|
|
# record; a process dying in between must not silently drop the fence.
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
directory = tmp_path / store._key("provisioner") / store._key("thread-sandbox")
|
|
directory.mkdir(parents=True)
|
|
|
|
assert store.contains(SandboxInfo("thread-sandbox", "http://sandbox"))
|
|
assert store.contains(SandboxInfo("thread-sandbox", "http://sandbox", container_id="crashed-generation"))
|
|
|
|
|
|
def test_retired_quarantine_allows_replacements_without_generation(tmp_path):
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
old = SandboxInfo("thread-sandbox", "http://sandbox")
|
|
store.mark(old)
|
|
store.mark(SandboxInfo(old.sandbox_id, old.sandbox_url, container_id="old-generation"))
|
|
other = SandboxInfo("another-thread", "http://sandbox")
|
|
store.mark(other)
|
|
|
|
store.retire(old)
|
|
|
|
restarted = SandboxQuarantine(tmp_path, "provisioner")
|
|
assert not restarted.contains(old)
|
|
assert not restarted.contains(SandboxInfo(old.sandbox_id, old.sandbox_url, container_id="new-generation"))
|
|
assert restarted.contains(other)
|
|
|
|
|
|
def test_retire_replaced_generations_drops_only_absent_fences(tmp_path):
|
|
"""A live replacement proves the fenced old generation is gone; its record
|
|
is pruned while the live generation's own fence survives."""
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
old = SandboxInfo("thread-sandbox", "http://sandbox", container_id="old-generation")
|
|
live = SandboxInfo("thread-sandbox", "http://sandbox", container_id="new-generation")
|
|
store.mark(old)
|
|
directory = tmp_path / store._key("provisioner") / store._key("thread-sandbox")
|
|
|
|
store.retire_replaced_generations(live)
|
|
|
|
assert not store.contains(old)
|
|
assert not directory.exists() # fully pruned, no stale record strands future creates
|
|
directory.mkdir() # a peer fences the live generation mid-flight
|
|
directory.joinpath(store._key("new-generation")).touch()
|
|
store.retire_replaced_generations(live)
|
|
assert store.contains(live)
|
|
|
|
|
|
def test_retire_replaced_generations_keeps_unversioned_crash_fence(tmp_path):
|
|
store = SandboxQuarantine(tmp_path, "provisioner")
|
|
store.mark(SandboxInfo("thread-sandbox", "http://sandbox")) # container_id-less mark → "unknown"
|
|
directory = tmp_path / store._key("provisioner") / store._key("thread-sandbox")
|
|
|
|
store.retire_replaced_generations(SandboxInfo("thread-sandbox", "http://sandbox", container_id="new-generation"))
|
|
|
|
assert (directory / "unknown").exists()
|