ChatDev/execorchestra/docs/apptainer-deployment.md
2026-11-05 16:22:12 +08:00

142 lines
7.5 KiB
Markdown

# Apptainer deployment branch
Development branch: `codex/cluster-apptainer`.
Baseline: `main` at `8ea74254d03d10c556cc5608667e501805f16555`.
This branch adds an opt-in execution substrate. Existing Docker executors and
their default entrypoints remain unchanged. Changes are developed in a separate
checkout; neither the source archive nor the original working checkout is used
as the development directory. No merge or public push is implied.
## Execution boundary
`ApptainerExecutionSubstrate` implements the transport used by the existing
external execution boundary. Resource wrapper staging, declared dependency
checks, typed dispatch and artifact publication remain framework operations.
The container contains the original tool image, identified by its SIF SHA-256.
Each run must have a new state directory. Only its `app` and `task-tmp`
subdirectories are writable container mounts. File operations execute in the
container so task symlinks are never followed by host-side file reads/writes.
The source tree, model gateway credentials, and other users' directories are not
mounted. Output is bounded and timed-out process groups are stopped.
Offline calls use a separate network namespace. A network-enabled call requires
both the operator's `declared` policy and the resource's network requirement.
These calls use the compute node's network, not a Docker bridge. Tool proxy and
service variables are explicitly selected; ambient model keys and container
runtime overrides are not inherited. This is user-space containment, not a VM
or a promise of identical Docker isolation. Slurm supplies resource limits.
The configuration follows the official Apptainer documentation for
[bind mounts](https://apptainer.org/docs/user/1.4/bind_paths_and_mounts.html) and
[environment handling](https://apptainer.org/docs/user/1.4/environment_and_metadata.html).
The host helper PATH must include both the container-tools `bin` and `sbin`
directories, including `squashfuse_ll` and `fusermount3`.
## Contract check
Run inside an allocated Slurm job, from the framework directory:
```bash
python -m scripts.run_apptainer \
--mode contract \
--image /path/to/sgar-runtime-rc1.sif \
--image-sha256 VERIFIED_SIF_SHA256 \
--apptainer /path/to/apptainer \
--tools-prefix /path/to/container-tools \
--run-id unique-contract-id \
--output-dir runs/unique-contract-id
```
This runs a real resource wrapper, publishes its artifact and verifies that a
dependent resource consumes it. It makes no model or embedding API calls and
does not claim task-solving success. Unit regressions can be run with
`python -m unittest discover -s tests -p 'test_apptainer_runtime.py'`.
## Pipeline integration
`--mode pipeline --query '...'` uses the same `run_pipeline` and existing
readiness, model-health, control-probe and retrieval-index gates. Nothing is
marked ready merely because the container starts. The default network policy
is `disabled`; use `--network-policy declared` only for the intended online task.
Explicit tool settings are passed by name, e.g. `--tool-env SEARXNG_BASE_URL`.
Load private credentials outside version control before starting the process.
Input-file staging is exposed through `--query-file`, `--input NAME=PATH` and
`--input-root PATH` using the same immutable public-input contract. Generated-code
dependency provisioning still needs parity verification with the Docker
runtime; this transport does not silently install missing task packages.
## Fresh tool and service evidence
`python -m scripts.smoke_tools_apptainer --help` runs the existing fixtures,
argument builders, source checks and success classifier in the pinned SIF.
Every row includes the SIF digest and implementation hash. Reports preserve
failures. Task-specific browser/compiler images remain a separate prerequisite.
An empty selection fails rather than producing a vacuous pass.
`python -m scripts.with_web_services --help` wraps a command with pinned
SearXNG and Zotero containers in the allocated job. Supply the preserved
SearXNG configuration as a private file. Search engine settings are retained;
the cluster proxy and dynamically allocated loopback ports replace the old
Docker network addresses. Zotero starts its installed server without updating
translators. Both services are stopped when the command exits or times out.
No gateway key is passed to either service. Startup alone is not a functional
check: run the search and DOI wrappers while the services are alive.
For readiness, write verified combined tool evidence to the existing local
`tool_smoke_latest.json` location and an ignored
`sgar_mvp/config/apptainer_runtime_lock.json` containing `runtime_backend`,
`runtime_image_sha256`, and `source_oci_digest`. Then run:
```bash
python -m scripts.prepare_release readiness --runtime-backend apptainer
```
Docker remains the default. Apptainer evidence cannot reuse Docker image IDs,
and mismatched per-resource SIF digests are rejected. The existing strict
preparation gate preserves the configured candidate pool when resources are
unavailable. Do not manufacture ready rows or use `--allow-unavailable` to
claim a full-pool reproduction. Index refresh and a complete task are later
stages, after the required configuration and availability decisions are made.
Three pre-existing wrappers (CORS headers, HTTP security headers, DNS latency)
actually perform network requests while their manifests declared offline use.
This branch corrects only their `network_required` flags, consistently in the
individual and combined catalogs. Their implementations and interfaces are
unchanged. All other resource definitions remain identical to the baseline.
## Verified checkpoint (2026-10-30)
On the target A800 compute cluster, job `267052` passed all seven formal
resource/artifact contract checks. Job `267063` passed eight unit boundary
checks and six real-container integration checks, including UTF-8/binary file
identity, hidden host files and ambient credentials, a read-only image,
symlink containment, offline/declared networking, and timed-out child cleanup.
Both allocations completed and released their GPUs. No model API requests were
made by these checks, and complete framework task execution is still pending.
The existing production-contract fixture needed the current
`authorized_materials` descriptors for its downstream artifact. The fixture
now supplies the recorded artifact identity; production validators are unchanged.
The next checks verified all five configured control roles (`267115`) and all
20 Agent wrappers (`267137`). Ten of the 12 selected model resources passed
every declared capability probe. The other two remain unavailable to strict
preparation; a failed probe is not replaced with a synthetic ready record.
Across the original fixtures, 127 of 133 tools passed actual execution. Three
require task-specific images; dictionary access, exchange-rate authentication,
and the Hypothesis service remain separate availability issues in snapshot
`267214`. That strict preparation check failed as expected and verified that
the configured 238-resource pool and effective catalogs were unchanged.
Job `267138` passed eight unit and seven real-container boundary checks,
including the new task-owned HOME/cache needed by Go. Job `267174` passed
six readiness regression tests, rechecked the three corrected network tools,
and verified all 102 declared Python package versions in the original image.
Job `267155` verified real SearXNG search and Zotero DOI translation, followed
by service shutdown and port release. These are component results; index
construction and complete task execution have not yet been verified.