deer-flow/backend/tests/test_provisioner_sandbox_responses.py
Storm b9a75b869d
fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes (#6436)
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes

Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.

What changed:
- The capability cache stores confirmed observations only, keyed per
  provisioner endpoint/auth identity; an unknown answer (timeout, malformed
  payload) raises instead of selecting credential placement, while a
  confirmed broker requirement is retained through outages. One in-flight
  probe per identity shares success and failure, and a Pod-attested mode
  contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
  SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
  observation into the client, so a probe outage delays drift detection but
  never denies reuse of an attested runtime, and never trusts an unattested
  one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
  UID) and shell capacity from one read, rejects lark requests conflicting
  with its configuration (409/503 carrying a capability-refresh marker the
  Gateway uses to drop the stale cache entry), rejects conflicting or
  Terminating Pods on coordination creates while reporting Terminating as
  data to status/get/list, and answers Pod presence independently of
  Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
  scope/sandbox/generation) fences containers after uncertain implicit
  shell outcomes across client rebuilds and restarts; in-process reuse,
  warm reclaim, discovery, create responses and orphan reconciliation all
  pass one policy gate, and replacement runs under the existing ownership
  and teardown fences. A fenced sandbox still tracked after a failed
  release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
  cleanup for the quarantined generation and releasing remembered ports
  under the same fences; local stop failures now propagate instead of being
  logged away, and remote destroy errors surface to callers.

Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.

* docs(changelog): record sandbox attestation and quarantine fix (#6436)

* fix(sandbox): preserve active leases and fail closed on unattested broker modes

Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.

- Distinguish pending session creation from ambiguous outcomes and reserve
  idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
  releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
  the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.

* fix(sandbox): scope the unverified-broker refuse path to attested providers

The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.

Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.

- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
  the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
  without the attribute

* fix(sandbox): close review gaps in broker attestation and teardown lifecycle

Address the latest #6436 review round:

- The provisioner's post-build broker-mode 409 now carries the
  capability-refresh marker, so an incompatible-mode conflict surfaced by
  idempotent re-create, attestation drift or post-create verification drops
  the Gateway's cached capability observation instead of repeating the same
  409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
  when teardown is deferred because an execution/upload holder or a peer
  teardown is still active, so callers can no longer mistake a deferred
  destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
  generations are absent; retire_replaced_generations prunes only those
  stale records so they cannot strand future creates, while the live
  generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
  unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
  _is_no_such_network_error parser, closing the substring loophole for
  context-name collisions and wrong-target errors.

* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate

The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 07:56:09 +08:00

167 lines
7.3 KiB
Python

"""Regression tests for provisioner lark conflict markers and Terminating handling."""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
def _pod(*, phase: str = "Running", deletion_timestamp=None, broker: bool = False) -> SimpleNamespace:
containers = [SimpleNamespace(name="lark-cli-broker")] if broker else []
init_containers = [SimpleNamespace(name="lark-cli-shim-init")] if broker else []
return SimpleNamespace(
metadata=SimpleNamespace(uid="pod-generation", deletion_timestamp=deletion_timestamp),
status=SimpleNamespace(phase=phase),
spec=SimpleNamespace(containers=containers, init_containers=init_containers),
)
@pytest.mark.parametrize(
"broker_image, runtime, broker, expected_status",
[
("", False, True, 503), # broker requested but the image is not configured
("broker-image", True, False, 409), # broker configured but the request says non-broker
],
)
def test_lark_provisioning_conflicts_carry_capability_refresh_marker(monkeypatch, provisioner_module, broker_image, runtime, broker, expected_status):
"""Both lark config conflicts tell the Gateway to drop its cached capability."""
monkeypatch.setattr(provisioner_module, "LARK_CLI_BROKER_IMAGE", broker_image)
with pytest.raises(provisioner_module.HTTPException) as error:
provisioner_module._validate_lark_provisioning_request(runtime, broker)
assert error.value.status_code == expected_status
assert error.value.headers[provisioner_module.CAPABILITY_REFRESH_HEADER] == "lark-broker"
def test_lark_provisioning_request_without_conflict_passes(monkeypatch, provisioner_module):
monkeypatch.setattr(provisioner_module, "LARK_CLI_BROKER_IMAGE", "broker-image")
provisioner_module._validate_lark_provisioning_request(True, True)
provisioner_module._validate_lark_provisioning_request(False, False)
@pytest.mark.parametrize(
"requirements",
[
{"required_broker": False},
{"required_broker": True},
{"required_capacity": 5},
],
)
def test_sandbox_response_rejects_terminating_pod_when_create_requirements_present(monkeypatch, provisioner_module, requirements):
"""A create coordination request must not settle on a Terminating Pod."""
# The Pod must satisfy the requested mode, or the pre-existing mode check
# fires first; this test pins the Terminating check specifically.
pod = _pod(deletion_timestamp="2026-10-07T00:00:00Z", broker=requirements.get("required_broker") is True)
core = MagicMock()
core.read_namespaced_pod.return_value = pod
monkeypatch.setattr(provisioner_module, "core_v1", core)
with pytest.raises(provisioner_module.HTTPException) as error:
provisioner_module._sandbox_response("sandbox-1", "http://sandbox", **requirements)
assert error.value.status_code == 409
assert "terminating" in error.value.detail
def test_sandbox_response_broker_conflict_carries_capability_refresh_marker(monkeypatch, provisioner_module):
"""A post-build broker-mode 409 must also tell the Gateway to drop its cached
capability observation, or the change lives out the negative cache TTL."""
core = MagicMock()
core.read_namespaced_pod.return_value = _pod(broker=False)
monkeypatch.setattr(provisioner_module, "core_v1", core)
with pytest.raises(provisioner_module.HTTPException) as error:
provisioner_module._sandbox_response("sandbox-1", "http://sandbox", required_broker=True)
assert error.value.status_code == 409
assert error.value.headers[provisioner_module.CAPABILITY_REFRESH_HEADER] == provisioner_module.CAPABILITY_REFRESH_LARK_BROKER
@pytest.mark.parametrize("broker", [False, True])
def test_sandbox_response_attests_pattern_a_and_b_modes(monkeypatch, provisioner_module, broker):
"""Pod observation attests both real layouts: Pattern A (init image only)
reads attested non-broker; Pattern B (shim + sidecar) reads broker."""
if broker:
pod = _pod(broker=True)
else:
# Pattern A: the init container is `lark-cli-init` (not the broker's
# shim-init), no sidecar, credentials mounted on the sandbox container.
pod = SimpleNamespace(
metadata=SimpleNamespace(uid="pod-generation", deletion_timestamp=None),
status=SimpleNamespace(phase="Running"),
spec=SimpleNamespace(
containers=[
SimpleNamespace(
name="sandbox",
volume_mounts=[SimpleNamespace(mount_path=f"{provisioner_module.LARK_CLI_CONFIG_CONTAINER_PATH}/u")],
)
],
init_containers=[SimpleNamespace(name="lark-cli-init")],
),
)
core = MagicMock()
core.read_namespaced_pod.return_value = pod
monkeypatch.setattr(provisioner_module, "core_v1", core)
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox", required_broker=broker)
assert response.lark_cli_broker is broker
def test_sandbox_response_reports_terminating_status_without_requirements(monkeypatch, provisioner_module):
"""get/list pass no requirements: the phase is data, not an error."""
core = MagicMock()
core.read_namespaced_pod.return_value = _pod(deletion_timestamp="2026-10-07T00:00:00Z")
monkeypatch.setattr(provisioner_module, "core_v1", core)
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox")
assert response.status == "Terminating"
assert response.sandbox_url == "http://sandbox"
def test_list_sandboxes_keeps_terminating_entries_visible(monkeypatch, provisioner_module):
"""One Terminating Pod must not fail or empty the whole fleet listing."""
core = MagicMock()
def read_pod(name, namespace):
assert namespace == provisioner_module.K8S_NAMESPACE
if name == "sandbox-sandbox-terminating":
return _pod(deletion_timestamp="2026-10-07T00:00:00Z")
return _pod(phase="Running")
core.read_namespaced_pod.side_effect = read_pod
core.list_namespaced_service.return_value = SimpleNamespace(
items=[
SimpleNamespace(
metadata=SimpleNamespace(labels={"sandbox-id": "sandbox-terminating"}),
spec=SimpleNamespace(ports=[SimpleNamespace(name="http", node_port=31001)]),
),
SimpleNamespace(
metadata=SimpleNamespace(labels={"sandbox-id": "sandbox-running"}),
spec=SimpleNamespace(ports=[SimpleNamespace(name="http", node_port=31002)]),
),
]
)
monkeypatch.setattr(provisioner_module, "core_v1", core)
payload = provisioner_module.list_sandboxes()
assert payload["count"] == 2
statuses = {entry.sandbox_id: entry.status for entry in payload["sandboxes"]}
assert statuses == {"sandbox-terminating": "Terminating", "sandbox-running": "Running"}
def test_sandbox_response_running_pod_without_broker_requirement_still_succeeds(monkeypatch, provisioner_module):
core = MagicMock()
core.read_namespaced_pod.return_value = _pod(phase="Running")
monkeypatch.setattr(provisioner_module, "core_v1", core)
response = provisioner_module._sandbox_response("sandbox-1", "http://sandbox")
assert response.status == "Running"
assert response.sandbox_url == "http://sandbox"