2 Commits

Author SHA1 Message Date
万家良
c324f351ee
fix(sandbox): truncate quarantine path keys to stay under Windows MAX_PATH (#6880)
* fix(sandbox): truncate quarantine path keys to stay under Windows MAX_PATH

The quarantine nests three hashed path components (scope, sandbox id,
container id). Three full 64-char SHA-256 hexdigests exceed Windows'
260-char MAX_PATH once the quarantine root lives under a long path, so
mark() fails with ENOENT creating the generation record. Truncate the
digest to 16 hex chars, matching the sandbox-naming convention used
elsewhere in this module.

* test(sandbox): pin quarantine key truncation to 16 hex chars

Guard against a regression that reverts the key digest to a full 64-char
SHA-256 hexdigest: three nested 64-char components exceed Windows' 260-char
MAX_PATH and re-break record creation, while Linux CI still passes.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-15 11:00:01 +08:00
Storm
b9a75b869d
fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes (#6436)
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes

Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.

What changed:
- The capability cache stores confirmed observations only, keyed per
  provisioner endpoint/auth identity; an unknown answer (timeout, malformed
  payload) raises instead of selecting credential placement, while a
  confirmed broker requirement is retained through outages. One in-flight
  probe per identity shares success and failure, and a Pod-attested mode
  contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
  SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
  observation into the client, so a probe outage delays drift detection but
  never denies reuse of an attested runtime, and never trusts an unattested
  one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
  UID) and shell capacity from one read, rejects lark requests conflicting
  with its configuration (409/503 carrying a capability-refresh marker the
  Gateway uses to drop the stale cache entry), rejects conflicting or
  Terminating Pods on coordination creates while reporting Terminating as
  data to status/get/list, and answers Pod presence independently of
  Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
  scope/sandbox/generation) fences containers after uncertain implicit
  shell outcomes across client rebuilds and restarts; in-process reuse,
  warm reclaim, discovery, create responses and orphan reconciliation all
  pass one policy gate, and replacement runs under the existing ownership
  and teardown fences. A fenced sandbox still tracked after a failed
  release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
  cleanup for the quarantined generation and releasing remembered ports
  under the same fences; local stop failures now propagate instead of being
  logged away, and remote destroy errors surface to callers.

Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.

* docs(changelog): record sandbox attestation and quarantine fix (#6436)

* fix(sandbox): preserve active leases and fail closed on unattested broker modes

Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.

- Distinguish pending session creation from ambiguous outcomes and reserve
  idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
  releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
  the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.

* fix(sandbox): scope the unverified-broker refuse path to attested providers

The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.

Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.

- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
  the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
  without the attribute

* fix(sandbox): close review gaps in broker attestation and teardown lifecycle

Address the latest #6436 review round:

- The provisioner's post-build broker-mode 409 now carries the
  capability-refresh marker, so an incompatible-mode conflict surfaced by
  idempotent re-create, attestation drift or post-create verification drops
  the Gateway's cached capability observation instead of repeating the same
  409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
  when teardown is deferred because an execution/upload holder or a peer
  teardown is still active, so callers can no longer mistake a deferred
  destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
  generations are absent; retire_replaced_generations prunes only those
  stale records so they cannot strand future creates, while the live
  generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
  unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
  _is_no_such_network_error parser, closing the substring loophole for
  context-name collisions and wrong-target errors.

* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate

The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-10 07:56:09 +08:00