mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes Why this change: Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker capability probe collapsed "provisioner unreachable" into "broker not configured": one 1.5s timeout cached a negative for 300s, so new sandboxes were created without the broker sidecar while the per-user credential directories were still bind-mounted into the container, and existing Pods kept being reused without their actual mode ever being read. Second, an implicit-shell command that timed out or returned an unknown status only marked the current client object; warm-pool reclaim and rediscovery built a fresh client that re-entered a container whose shell state could not be proven clean, and nothing about that ambiguity survived a Gateway restart. What changed: - The capability cache stores confirmed observations only, keyed per provisioner endpoint/auth identity; an unknown answer (timeout, malformed payload) raises instead of selecting credential placement, while a confirmed broker requirement is retained through outages. One in-flight probe per identity shares success and failure, and a Pod-attested mode contradicting the cache forces one shared refresh. - Execution uses the mode attested for the actual sandbox: SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod observation into the client, so a probe outage delays drift detection but never denies reuse of an attested runtime, and never trusts an unattested one. - The provisioner reports each Pod's actual broker mode, generation (Pod UID) and shell capacity from one read, rejects lark requests conflicting with its configuration (409/503 carrying a capability-refresh marker the Gateway uses to drop the stale cache entry), rejects conflicting or Terminating Pods on coordination creates while reporting Terminating as data to status/get/list, and answers Pod presence independently of Service readiness. - A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by scope/sandbox/generation) fences containers after uncertain implicit shell outcomes across client rebuilds and restarts; in-process reuse, warm reclaim, discovery, create responses and orphan reconciliation all pass one policy gate, and replacement runs under the existing ownership and teardown fences. A fenced sandbox still tracked after a failed release-recycle is destroyed inline on the next acquire. - Quarantine records retire only after confirmed absence, retrying partial cleanup for the quarantined generation and releasing remembered ports under the same fences; local stop failures now propagate instead of being logged away, and remote destroy errors surface to callers. Problem solved: A provisioner blip could silently strip broker isolation from new lark sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted default shells persist across warm reuse, rediscovery and restarts. Docs updated: README, aio_sandbox and skills AGENTS.md. * docs(changelog): record sandbox attestation and quarantine fix (#6436) * fix(sandbox): preserve active leases and fail closed on unattested broker modes Concurrent uploads could recycle a healthy AIO runtime while another holder was creating or cleaning up a command session. Older provisioner responses could also trigger destructive replacement before failing admission. - Distinguish pending session creation from ambiguous outcomes and reserve idle teardown until execution and upload holders finish scope cleanup. - Reject stale fork lookups and reconcile cancelled acquires without releasing a sandbox still borrowed by another holder. - Default Lark broker mode to unverified and preserve unattested Pods until the provisioner is upgraded. - Recognize Docker 29 absence diagnostics while matching resource names exactly. - Preserve Gateway upload errors and cover admission and lifecycle regressions. * fix(sandbox): scope the unverified-broker refuse path to attested providers The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox without a lark_cli_broker attribute as unverified, but only AioSandbox carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no broker sidecar, no credential mounts, and no provisioner to upgrade, so every lark-cli command on those providers failed with an unactionable error. Declare the tri-state on the Sandbox base class (defaulting to False, the only mode brokerless providers have) so the None fail-closed path stays scoped to AioSandbox's deliberate unverified default. - Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring the persistent_shell_sessions tri-state pattern - Add a regression test pinning the non-broker overlay for a provider without the attribute * fix(sandbox): close review gaps in broker attestation and teardown lifecycle Address the latest #6436 review round: - The provisioner's post-build broker-mode 409 now carries the capability-refresh marker, so an incompatible-mode conflict surfaced by idempotent re-create, attestation drift or post-create verification drops the Gateway's cached capability observation instead of repeating the same 409 for the full negative-cache TTL. - AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False when teardown is deferred because an execution/upload holder or a peer teardown is still active, so callers can no longer mistake a deferred destroy for a completed one. - A live replacement under the same logical ID proves the fenced old generations are absent; retire_replaced_generations prunes only those stale records so they cannot strand future creates, while the live generation's fence and unversioned crash fences are never touched. - SandboxInfo.from_dict keeps a missing lark_cli_broker attestation unverified (None) instead of silently rehydrating as attested non-broker. - The restricted-network inspect path shares the strict _is_no_such_network_error parser, closing the substring loophole for context-name collisions and wrong-target errors. * docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate The three lease notes added in 26c307f3 pushed the effective guidance chain to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance (AG002). Fold them into two lines carrying the same rules. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
167 lines
7.3 KiB
Python
167 lines
7.3 KiB
Python
"""Regression tests for provisioner lark conflict markers and Terminating handling."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
|
|
def _pod(*, phase: str = "Running", deletion_timestamp=None, broker: bool = False) -> SimpleNamespace:
|
|
containers = [SimpleNamespace(name="lark-cli-broker")] if broker else []
|
|
init_containers = [SimpleNamespace(name="lark-cli-shim-init")] if broker else []
|
|
return SimpleNamespace(
|
|
metadata=SimpleNamespace(uid="pod-generation", deletion_timestamp=deletion_timestamp),
|
|
status=SimpleNamespace(phase=phase),
|
|
spec=SimpleNamespace(containers=containers, init_containers=init_containers),
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"broker_image, runtime, broker, expected_status",
|
|
[
|
|
("", False, True, 503), # broker requested but the image is not configured
|
|
("broker-image", True, False, 409), # broker configured but the request says non-broker
|
|
],
|
|
)
|
|
def test_lark_provisioning_conflicts_carry_capability_refresh_marker(monkeypatch, provisioner_module, broker_image, runtime, broker, expected_status):
|
|
"""Both lark config conflicts tell the Gateway to drop its cached capability."""
|
|
monkeypatch.setattr(provisioner_module, "LARK_CLI_BROKER_IMAGE", broker_image)
|
|
|
|
with pytest.raises(provisioner_module.HTTPException) as error:
|
|
provisioner_module._validate_lark_provisioning_request(runtime, broker)
|
|
|
|
assert error.value.status_code == expected_status
|
|
assert error.value.headers[provisioner_module.CAPABILITY_REFRESH_HEADER] == "lark-broker"
|
|
|
|
|
|
def test_lark_provisioning_request_without_conflict_passes(monkeypatch, provisioner_module):
|
|
monkeypatch.setattr(provisioner_module, "LARK_CLI_BROKER_IMAGE", "broker-image")
|
|
provisioner_module._validate_lark_provisioning_request(True, True)
|
|
provisioner_module._validate_lark_provisioning_request(False, False)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"requirements",
|
|
[
|
|
{"required_broker": False},
|
|
{"required_broker": True},
|
|
{"required_capacity": 5},
|
|
],
|
|
)
|
|
def test_sandbox_response_rejects_terminating_pod_when_create_requirements_present(monkeypatch, provisioner_module, requirements):
|
|
"""A create coordination request must not settle on a Terminating Pod."""
|
|
# The Pod must satisfy the requested mode, or the pre-existing mode check
|
|
# fires first; this test pins the Terminating check specifically.
|
|
pod = _pod(deletion_timestamp="2026-10-07T00:00:00Z", broker=requirements.get("required_broker") is True)
|
|
core = MagicMock()
|
|
core.read_namespaced_pod.return_value = pod
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
with pytest.raises(provisioner_module.HTTPException) as error:
|
|
provisioner_module._sandbox_response("sandbox-1", "http://sandbox", **requirements)
|
|
|
|
assert error.value.status_code == 409
|
|
assert "terminating" in error.value.detail
|
|
|
|
|
|
def test_sandbox_response_broker_conflict_carries_capability_refresh_marker(monkeypatch, provisioner_module):
|
|
"""A post-build broker-mode 409 must also tell the Gateway to drop its cached
|
|
capability observation, or the change lives out the negative cache TTL."""
|
|
core = MagicMock()
|
|
core.read_namespaced_pod.return_value = _pod(broker=False)
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
with pytest.raises(provisioner_module.HTTPException) as error:
|
|
provisioner_module._sandbox_response("sandbox-1", "http://sandbox", required_broker=True)
|
|
|
|
assert error.value.status_code == 409
|
|
assert error.value.headers[provisioner_module.CAPABILITY_REFRESH_HEADER] == provisioner_module.CAPABILITY_REFRESH_LARK_BROKER
|
|
|
|
|
|
@pytest.mark.parametrize("broker", [False, True])
|
|
def test_sandbox_response_attests_pattern_a_and_b_modes(monkeypatch, provisioner_module, broker):
|
|
"""Pod observation attests both real layouts: Pattern A (init image only)
|
|
reads attested non-broker; Pattern B (shim + sidecar) reads broker."""
|
|
if broker:
|
|
pod = _pod(broker=True)
|
|
else:
|
|
# Pattern A: the init container is `lark-cli-init` (not the broker's
|
|
# shim-init), no sidecar, credentials mounted on the sandbox container.
|
|
pod = SimpleNamespace(
|
|
metadata=SimpleNamespace(uid="pod-generation", deletion_timestamp=None),
|
|
status=SimpleNamespace(phase="Running"),
|
|
spec=SimpleNamespace(
|
|
containers=[
|
|
SimpleNamespace(
|
|
name="sandbox",
|
|
volume_mounts=[SimpleNamespace(mount_path=f"{provisioner_module.LARK_CLI_CONFIG_CONTAINER_PATH}/u")],
|
|
)
|
|
],
|
|
init_containers=[SimpleNamespace(name="lark-cli-init")],
|
|
),
|
|
)
|
|
core = MagicMock()
|
|
core.read_namespaced_pod.return_value = pod
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox", required_broker=broker)
|
|
|
|
assert response.lark_cli_broker is broker
|
|
|
|
|
|
def test_sandbox_response_reports_terminating_status_without_requirements(monkeypatch, provisioner_module):
|
|
"""get/list pass no requirements: the phase is data, not an error."""
|
|
core = MagicMock()
|
|
core.read_namespaced_pod.return_value = _pod(deletion_timestamp="2026-10-07T00:00:00Z")
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox")
|
|
|
|
assert response.status == "Terminating"
|
|
assert response.sandbox_url == "http://sandbox"
|
|
|
|
|
|
def test_list_sandboxes_keeps_terminating_entries_visible(monkeypatch, provisioner_module):
|
|
"""One Terminating Pod must not fail or empty the whole fleet listing."""
|
|
core = MagicMock()
|
|
|
|
def read_pod(name, namespace):
|
|
assert namespace == provisioner_module.K8S_NAMESPACE
|
|
if name == "sandbox-sandbox-terminating":
|
|
return _pod(deletion_timestamp="2026-10-07T00:00:00Z")
|
|
return _pod(phase="Running")
|
|
|
|
core.read_namespaced_pod.side_effect = read_pod
|
|
core.list_namespaced_service.return_value = SimpleNamespace(
|
|
items=[
|
|
SimpleNamespace(
|
|
metadata=SimpleNamespace(labels={"sandbox-id": "sandbox-terminating"}),
|
|
spec=SimpleNamespace(ports=[SimpleNamespace(name="http", node_port=31001)]),
|
|
),
|
|
SimpleNamespace(
|
|
metadata=SimpleNamespace(labels={"sandbox-id": "sandbox-running"}),
|
|
spec=SimpleNamespace(ports=[SimpleNamespace(name="http", node_port=31002)]),
|
|
),
|
|
]
|
|
)
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
payload = provisioner_module.list_sandboxes()
|
|
|
|
assert payload["count"] == 2
|
|
statuses = {entry.sandbox_id: entry.status for entry in payload["sandboxes"]}
|
|
assert statuses == {"sandbox-terminating": "Terminating", "sandbox-running": "Running"}
|
|
|
|
|
|
def test_sandbox_response_running_pod_without_broker_requirement_still_succeeds(monkeypatch, provisioner_module):
|
|
core = MagicMock()
|
|
core.read_namespaced_pod.return_value = _pod(phase="Running")
|
|
monkeypatch.setattr(provisioner_module, "core_v1", core)
|
|
|
|
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox")
|
|
|
|
assert response.status == "Running"
|
|
assert response.sandbox_url == "http://sandbox"
|