fix(sandbox): add FOWNER for AIO 1.11 startup (#5163)

* fix(sandbox): add FOWNER for AIO 1.11 startup

* test(sandbox): cover FOWNER startup capability

* docs(sandbox): document FOWNER capability

* test(sandbox): pin FOWNER regression smoke

* ci(sandbox): allow pinning FOWNER smoke image

* style(sandbox): format FOWNER smoke test

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Jun 2026-09-04 00:03:49 +08:00 committed by GitHub
parent 69c160ba77
commit 83cb6767b3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 117 additions and 77 deletions

View File

@ -1,16 +1,14 @@
name: Sandbox Image Smoke
# Real-image validation of the Docker sandbox hardening: pulls the shipped
# AIO image and drives it through the production readiness deadline
# (SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT). The offline unit suite (-m "not
# live") never runs this, so this dedicated job is the only place the
# known-bad capability/startup regressions are caught before merge.
# Real-image validation of the Docker sandbox hardening. The job preserves
# baseline coverage for the repository-configured/default AIO image and also
# exercises the 1.11.0 startup path that requires FOWNER for chmod on
# /run/user/1000. The offline unit suite (-m "not live") never runs these.
#
# Reproducibility: the image reference comes from the repository variable
# SANDBOX_SMOKE_IMAGE (pin a digest there, e.g.
# registry/.../all-in-one-sandbox@sha256:...). When unset it falls back to
# the mutable :latest tag and the resolved digest is printed to the job
# summary so a failure can be reproduced against the exact image tested.
# Reproducibility: SANDBOX_SMOKE_IMAGE can pin the baseline image. Both that
# image (or the :latest fallback) and the explicit 1.11.0 regression image are
# pulled first and converted to immutable repo@sha256 references; the exact
# digests are printed to the job summary before either live test runs.
on:
workflow_dispatch:
@ -31,27 +29,36 @@ permissions:
jobs:
sandbox-image-smoke:
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 25
env:
SANDBOX_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest' }}
FOWNER_SMOKE_IMAGE_REF: ${{ vars.SANDBOX_FOWNER_SMOKE_IMAGE || 'enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:1.11.0' }}
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Resolve an immutable image reference for this run
# Pull once on the runner's platform and export the immutable
# repo@sha256 reference through GITHUB_ENV: the test then runs the
# exact image recorded here, and a tag moving between steps cannot
# make the summary name a different image than the one executed.
# (docker manifest inspect + jq is not used because picking a
# manifest from the index by hand can miss the runner's platform.)
- name: Resolve immutable image references for this run
# Pull on the runner's platform, then pass only immutable repo@sha256
# references to pytest. This keeps both the baseline and the 1.11.0
# regression run reproducible even if either source tag moves later.
run: |
set -euo pipefail
docker pull "$SANDBOX_SMOKE_IMAGE_REF" >/dev/null
repo_digest="$(docker image inspect "$SANDBOX_SMOKE_IMAGE_REF" --format '{{index .RepoDigests 0}}')"
echo "DEER_FLOW_SANDBOX_SMOKE_IMAGE=$repo_digest" >> "$GITHUB_ENV"
echo "Smoke-testing immutable reference: $repo_digest" | tee -a "$GITHUB_STEP_SUMMARY"
resolve_image() {
local source_ref="$1"
local output_var="$2"
local label="$3"
local repo_digest
docker pull "$source_ref" >/dev/null
repo_digest="$(docker image inspect "$source_ref" --format '{{index .RepoDigests 0}}')"
echo "$output_var=$repo_digest" >> "$GITHUB_ENV"
printf '%s: `%s`\n' "$label" "$repo_digest" | tee -a "$GITHUB_STEP_SUMMARY"
}
resolve_image "$SANDBOX_SMOKE_IMAGE_REF" DEER_FLOW_SANDBOX_SMOKE_IMAGE "Configured/default smoke image"
resolve_image "$FOWNER_SMOKE_IMAGE_REF" DEER_FLOW_SANDBOX_FOWNER_SMOKE_IMAGE "FOWNER regression image (1.11.0)"
- name: Set up Python
uses: actions/setup-python@v6
@ -68,6 +75,6 @@ jobs:
working-directory: backend
run: uv sync --group dev
- name: Run the live real-image smoke test
- name: Run the live real-image smoke tests
working-directory: backend
run: uv run pytest -m live tests/test_aio_sandbox_local_backend.py -v

View File

@ -668,15 +668,15 @@ When you configure `sandbox.mounts`, DeerFlow exposes those `container_path` val
The sandbox HTTP API (`/v1/shell/*` and friends) has no authentication: anyone who can reach a published sandbox port can execute arbitrary commands in that sandbox. For bare-metal Docker sandbox runs that use localhost, DeerFlow binds the sandbox port to `127.0.0.1` so it is not exposed on other host interfaces. For Docker-outside-of-Docker deployments that connect through `host.docker.internal`, the port is bound to the address that hostname actually resolves to — the daemon's `host-gateway-ip` mapping (customizable, possibly IPv6) — so the published port and the address the gateway connects to always match, and the port is no longer published on external network interfaces (previously it was bound to `0.0.0.0`). If resolution fails, the Docker default bridge gateway (via `docker network inspect bridge`, falling back to `172.17.0.1`) is used as a best-effort bind and a warning is logged. Set `DEER_FLOW_SANDBOX_BIND_HOST` explicitly if your deployment needs a different bind address; setting it to `0.0.0.0` restores the legacy broad bind, which re-exposes the unauthenticated exec API on every interface and should be paired with an external firewall.
Local Docker sandbox containers are also hardened by default: all Linux capabilities are dropped (`--cap-drop=ALL`) except the minimum four the shipped image needs — `CHOWN` (the entrypoint chowns /opt/jupyter), `SETUID`/`SETGID` (it creates the gem user and drops to it via `su`), and `DAC_OVERRIDE` (the root nginx master writes gem-owned logs under /var/log/nginx, a per-request runtime need) — privilege escalation is blocked across exec (`no-new-privileges`), and CPU/memory/PID resources are bounded.
Local Docker sandbox containers are also hardened by default: all Linux capabilities are dropped (`--cap-drop=ALL`) except a five-capability compatibility allowlist — `CHOWN`, `FOWNER`, `SETUID`, `SETGID`, and `DAC_OVERRIDE` — while privilege escalation across exec stays blocked with `no-new-privileges` and CPU/memory/PID resources are bounded. `CHOWN`/`SETUID`/`SETGID` support the runtime user handoff and `DAC_OVERRIDE` supports the root nginx master's writes to gem-owned logs. `FOWNER` is specifically required by the newer AIO 1.11.x startup path (regression-tested against the recommended 1.11.0 image), which runs `chmod /run/user/1000` after capabilities are dropped. Images that do not perform that `chmod` do not need `FOWNER`; DeerFlow deliberately does not guess a smaller set from mutable tags, digests, or arbitrary custom images, so the default compatibility allowlist remains version-agnostic.
A custom image that is already fully initialized as a non-root user (no runtime root handoff) should set `DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0` to drop every capability including those three: leaving them on would let sandboxed code chown bind-mounted paths or impersonate mounted-file UIDs/GIDs for the container's lifetime. Note that `no-new-privileges` does **not** mitigate that risk — it only blocks gaining privileges across exec; the risk comes from the retained `CAP_SETUID`/`CAP_SETGID` themselves. One hardening knob is relaxed by default: the shipped AIO image runs with `seccomp=unconfined` because its Chromium browser does not start under Docker's default seccomp profile (syscall filtering is disabled — see the two seccomp variables below to change that). The following environment variables (set them in the gateway process, e.g. via `.env` loaded by docker-compose, or the gateway service `environment:`) tune or disable each knob:
A custom image that is already fully initialized as a non-root user and needs none of those compatibility capabilities should set `DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0` to drop the whole set. This is an all-or-nothing opt-out, not a per-capability selector: an older or custom root-initialized image that does not need `FOWNER` may still require `CHOWN`, `SETUID`, `SETGID`, or `DAC_OVERRIDE` and should therefore leave the compatibility set enabled. Retained capabilities remain available for the container's lifetime and can let sandboxed code change ownership or mode on accessible bind-mounted paths, impersonate mounted-file UIDs/GIDs, or bypass discretionary access checks. `no-new-privileges` does **not** mitigate that existing-capability risk — it only blocks gaining new privileges across exec. One hardening knob is relaxed by default: the shipped AIO image runs with `seccomp=unconfined` because its Chromium browser does not start under Docker's default seccomp profile (syscall filtering is disabled — see the two seccomp variables below to change that). The following environment variables (set them in the gateway process, e.g. via `.env` loaded by docker-compose, or the gateway service `environment:`) tune or disable each knob:
| Environment variable | Default | Purpose |
| --- | --- | --- |
| `DEER_FLOW_SANDBOX_BIND_HOST` | loopback / bridge gateway (see above) | Host interface for the sandbox `-p` publish. Must be an IP literal (bare or bracketed IPv6) or a hostname, which is resolved to an address first — Docker publish specs do not accept hostnames. `0.0.0.0` restores the legacy broad bind (risky). |
| `DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED` | on | The shipped AIO image's Chromium browser does not start under Docker's default seccomp profile (see the upstream agent-infra sandbox FAQ), so `seccomp=unconfined` remains the default. Set to `0` to run with the built-in profile — passed explicitly as `seccomp=builtin`, so a daemon configured with a different default cannot weaken the opt-out — and only for images verified to start and pass browser checks with it. |
| `DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS` | on | Keeps the four capabilities (`CHOWN`/`SETUID`/`SETGID`/`DAC_OVERRIDE`) that the shipped image needs: three for the entrypoint's runtime user handoff, plus `DAC_OVERRIDE` because the root nginx master writes gem-owned log files for the container's lifetime. Set to `0` for images already fully initialized as a non-root user — every capability is then dropped, so sandboxed code cannot chown bind-mounted paths or impersonate mounted-file UIDs/GIDs. |
| `DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS` | on | Keeps the five-capability compatibility set (`CHOWN`/`FOWNER`/`SETUID`/`SETGID`/`DAC_OVERRIDE`). `FOWNER` specifically covers the newer AIO 1.11.x startup `chmod /run/user/1000` path (tested with 1.11.0); images without that step do not need `FOWNER`, but DeerFlow does not infer per-image capability subsets from tags/digests/custom images. Set to `0` only for images that need none of the five — the switch drops the entire set. |
| `DEER_FLOW_SANDBOX_SECCOMP_PROFILE` | unset | Path to a custom seccomp profile (e.g. a restricted, Chromium-compatible one built from Docker's default plus the namespace syscalls Chromium needs). Takes precedence over the unconfined default. |
| `DEER_FLOW_SANDBOX_MEMORY` | `2g` | `--memory` limit per sandbox container. `0`/`none` disables the limit. |
| `DEER_FLOW_SANDBOX_CPUS` | `2` | `--cpus` limit per sandbox container. `0`/`none` disables the limit. |

View File

@ -822,36 +822,39 @@ class LocalContainerBackend(SandboxBackend):
# Docker-only security hardening. The sandbox container executes
# untrusted, model-authored code, so it must not run with the
# daemon's permissive defaults: all Linux capabilities are dropped
# except the minimum the shipped image's entrypoint needs to
# initialize itself, privilege escalation (setuid/sudo) is blocked,
# except a small compatibility set needed across supported AIO image
# startup/runtime paths, privilege escalation (setuid/sudo) is blocked,
# and CPU/memory/PID footprints are bounded so one runaway sandbox
# cannot exhaust the host or fork-bomb it. Each knob has an env
# escape hatch documented in backend/docs/CONFIGURATION.md. Apple
# Container's CLI does not support these flags, so they are
# Docker-only.
if self._runtime == "docker":
# The default image (/opt/gem/run.sh) starts as root, creates the
# gem account at runtime, chown -R's /opt/jupyter, and drops to
# that user via su before starting the services. That needs
# CHOWN/SETUID/SETGID; additionally the root nginx master writes
# logs under /var/log/nginx that belong to the gem user, which
# requires DAC_OVERRIDE — without it nginx dies with
# "open() .../access.log failed (13: Permission denied)" on every
# start (a runtime need, not just startup: access.log is written
# per request). Dropping ALL of them makes the image fail before
# the readiness endpoint exists.
# Supported shipped/recommended AIO images start as root, create
# the gem account at runtime, chown -R /opt/jupyter, and drop to
# that user via su. CHOWN/SETUID/SETGID cover that ownership
# handoff. FOWNER is specifically required by the newer 1.11.x
# startup path (regression-tested against 1.11.0), which chmods
# /run/user/1000 after capabilities are dropped. Images that do
# not perform that chmod do not need FOWNER; DeerFlow deliberately
# keeps this compatibility allowlist version-agnostic instead of
# guessing from mutable tags/digests or arbitrary custom images.
# The root nginx master also writes gem-owned logs under
# /var/log/nginx, which requires DAC_OVERRIDE — without it nginx
# dies with "open() .../access.log failed (13: Permission denied)"
# on every start (a runtime need, not just startup). Dropping ALL
# of these can make root-initialized images fail before readiness.
# no-new-privileges stays: it only blocks *gaining* privileges
# through exec, it does not revoke the capabilities added here,
# and su from the already-root entrypoint does not need to gain
# anything. Everything else (NET_RAW, SYS_PTRACE, ...) stays
# dropped, which is the bulk of the attack-surface reduction.
# For a pre-initialized non-root image nothing ever runs as
# root, so the handoff capabilities are not needed — and leaving
# them available for the container's lifetime would let
# sandboxed code chown bind-mounted paths or impersonate
# mounted-file UIDs/GIDs. Such images opt out with
# DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0 (see CONFIGURATION.md),
# which drops every capability including these three.
# A pre-initialized non-root image that needs none of these
# compatibility capabilities should opt out with
# DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0 (see CONFIGURATION.md).
# That switch drops the whole set; it is intentionally not used
# to infer or trim individual capabilities for older/custom root-
# initialized images that may still need the remaining entries.
if _env_flag_disabled("DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS"):
cmd.extend(["--cap-drop=ALL", "--security-opt", "no-new-privileges"])
else:
@ -859,6 +862,7 @@ class LocalContainerBackend(SandboxBackend):
[
"--cap-drop=ALL",
"--cap-add=CHOWN",
"--cap-add=FOWNER",
"--cap-add=SETUID",
"--cap-add=SETGID",
"--cap-add=DAC_OVERRIDE",

View File

@ -447,12 +447,13 @@ def test_start_container_hardens_docker_run_by_default(monkeypatch):
assert "--cap-drop=ALL" in captured_cmd
# The shipped image's entrypoint starts as root, creates the gem user,
# chowns /opt/jupyter, and drops to that user via su — CHOWN/SETUID/SETGID
# must survive the drop or the container exits before readiness. The root
# nginx master also writes gem-owned logs under /var/log/nginx for the
# container's lifetime, which needs DAC_OVERRIDE.
# chowns /opt/jupyter, prepares /run/user/1000 with chmod, and drops to
# that user via su. CHOWN/FOWNER/SETUID/SETGID must survive the drop or
# the container exits before readiness. The root nginx master also writes
# gem-owned logs under /var/log/nginx for the container's lifetime, which
# needs DAC_OVERRIDE.
cap_adds = [arg.split("=", 1)[1] for arg in captured_cmd if arg.startswith("--cap-add=")]
assert cap_adds == ["CHOWN", "SETUID", "SETGID", "DAC_OVERRIDE"]
assert cap_adds == ["CHOWN", "FOWNER", "SETUID", "SETGID", "DAC_OVERRIDE"]
security_opts = [captured_cmd[i + 1] for i, arg in enumerate(captured_cmd) if arg == "--security-opt"]
assert "no-new-privileges" in security_opts
# The shipped AIO image needs seccomp=unconfined for its Chromium
@ -957,9 +958,12 @@ def test_effective_network_target_last_name_field_wins():
assert target("1f2a" * 16) == "1f2a" * 16 # network ID passes through
# ── Real-image startup smoke test (docker-gated) ────────────────────────────
# Keep in sync with aio_sandbox_provider.DEFAULT_IMAGE.
# ── Real-image startup smoke tests (docker-gated) ────────────────────────────
# Keep the baseline default in sync with aio_sandbox_provider.DEFAULT_IMAGE.
_DEFAULT_AIO_IMAGE = "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest"
# Dedicated regression target for #5161. The workflow resolves this tag to an
# immutable digest before pytest runs so the CI result records the exact image.
_FOWNER_REGRESSION_AIO_IMAGE = "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:1.11.0"
def _docker_daemon_available() -> bool:
@ -970,21 +974,13 @@ def _docker_daemon_available() -> bool:
return False
# `live`: pulls and runs a mutable external image, so the default offline
# suite (`make test` = `-m "not live"`) never touches the network. The
# daemon probe happens inside the test body — never at collection time.
@pytest.mark.live
def test_default_image_starts_under_hardened_capabilities(monkeypatch):
"""Real smoke test against the shipped default image — no subprocess mock.
The image's entrypoint (/opt/gem/run.sh) starts as root, creates the gem
account at runtime, chown -R's /opt/jupyter, and drops to that user via
su before starting the services. Under the default hardened argv
(--cap-drop=ALL + no-new-privileges) that initialization needs
CHOWN/SETUID/SETGID to be re-added, or the container exits (set -e)
before the readiness endpoint exists. Reaching readiness through the
real docker run proves the whole startup chain survives the hardening.
"""
def _assert_image_starts_under_hardened_capabilities(
monkeypatch,
*,
image: str,
sandbox_id: str,
failure_label: str,
) -> None:
from deerflow.community.aio_sandbox.backend import SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT
from deerflow.community.aio_sandbox.local_backend import wait_for_sandbox_ready
@ -992,9 +988,7 @@ def test_default_image_starts_under_hardened_capabilities(monkeypatch):
pytest.skip("requires a running Docker daemon")
backend = LocalContainerBackend(
# Pin via this override when wiring a dedicated integration job, so
# the run does not depend on a mutable :latest tag.
image=os.environ.get("DEER_FLOW_SANDBOX_SMOKE_IMAGE", _DEFAULT_AIO_IMAGE),
image=image,
base_port=18210,
container_prefix="sandbox-smoke",
config_mounts=[],
@ -1002,7 +996,7 @@ def test_default_image_starts_under_hardened_capabilities(monkeypatch):
)
_clear_hardening_env(monkeypatch)
info = backend.create(thread_id="smoke", sandbox_id="caps-smoke")
info = backend.create(thread_id="smoke", sandbox_id=sandbox_id)
try:
# The production deadline, single-sourced: the sync and async
# provider paths destroy the container after exactly this budget, so
@ -1012,8 +1006,8 @@ def test_default_image_starts_under_hardened_capabilities(monkeypatch):
ready = wait_for_sandbox_ready(info.sandbox_url, timeout=SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT)
if not ready:
# Fail diagnosably: the entrypoint's own log tells us whether the
# capability set is still incomplete (chown/useradd/su errors) or
# the services are merely slow.
# capability set is still incomplete (chown/chmod/useradd/su
# errors) or the services are merely slow.
logs = subprocess.run(
["docker", "logs", info.container_name],
capture_output=True,
@ -1037,17 +1031,52 @@ def test_default_image_starts_under_hardened_capabilities(monkeypatch):
timeout=30,
)
tail = "\n".join((logs.stdout + logs.stderr).splitlines()[-40:]) + "\n" + (prog_logs.stdout or "")
pytest.fail(f"default image never became ready under the hardened capabilities: {info.sandbox_url}\n--- last 40 container log lines ---\n{tail}")
pytest.fail(f"{failure_label} never became ready under the hardened capabilities: {info.sandbox_url}\n--- last 40 container log lines ---\n{tail}")
assert backend.is_alive(info)
finally:
backend.destroy(info)
# `live`: pulls and runs external images, so the default offline suite
# (`make test` = `-m "not live"`) never touches the network. The daemon probe
# happens inside the helper — never at collection time.
@pytest.mark.live
def test_default_image_starts_under_hardened_capabilities(monkeypatch):
"""Preserve real-image coverage for the configured/default AIO image.
This stays separate from the 1.11.0 regression because older images may
not execute the FOWNER-gated chmod added by the newer startup path.
"""
_assert_image_starts_under_hardened_capabilities(
monkeypatch,
image=os.environ.get("DEER_FLOW_SANDBOX_SMOKE_IMAGE", _DEFAULT_AIO_IMAGE),
sandbox_id="caps-smoke-default",
failure_label="configured/default image",
)
@pytest.mark.live
def test_aio_1_11_image_starts_with_fowner_capability(monkeypatch):
"""Regression smoke for #5161 against the recommended AIO 1.11.0 image.
Its entrypoint chmods /run/user/1000 after capabilities are dropped.
Without FOWNER that startup path exits before readiness; reaching the
endpoint proves the five-capability compatibility set covers the bug.
"""
_assert_image_starts_under_hardened_capabilities(
monkeypatch,
image=os.environ.get("DEER_FLOW_SANDBOX_FOWNER_SMOKE_IMAGE", _FOWNER_REGRESSION_AIO_IMAGE),
sandbox_id="caps-smoke-fowner-1-11",
failure_label="AIO 1.11.0 FOWNER regression image",
)
def test_start_container_preinitialized_image_can_drop_startup_caps(monkeypatch):
"""A custom, pre-initialized non-root image never runs the root handoff,
so CHOWN/SETUID/SETGID must not stay available for the container's
lifetime (chown on bind mounts, UID/GID impersonation). Opting out with
DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0 drops every capability."""
so CHOWN/FOWNER/SETUID/SETGID/DAC_OVERRIDE must not stay available for
the container's lifetime (chown/chmod on bind mounts, UID/GID
impersonation). Opting out with DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0
drops every capability."""
backend = LocalContainerBackend(
image="my-preinitialized-sandbox:latest",
base_port=8080,