ChatDev/execorchestra/docs/SERVICES.md

8.1 KiB

Models and services

Run these commands from the project directory (ChatDev/execorchestra) after installing the host environment and running python -m scripts.prepare_release init. Keep all service credentials in .env, and export them with set -a; source .env; set +a in each shell. Use shell-compatible values; quote values containing spaces or shell characters.

Model endpoint

Set LLM_BASE_URL to the OpenAI-compatible API base URL and LLM_API_KEY to its credential. The endpoint must support strict JSON-schema responses, configured reasoning settings, tool continuation, and usage metadata.

Control role API model ID Reasoning
Planner gpt-6.1-sol high
Profiler gemini-3.8-flash high
Composer and plan adaptation gpt-6-luna high
Optional evaluator gpt-5.6-sol high

The optional evaluator is disabled in the supplied execution configuration. gpt-6.1-sol and gpt-6-luna are registered for control roles; adding them does not add them to the task candidate pool. The distributed research pool contains 10 models, 38 tools, 4 skills, and 5 agents. The candidate model API IDs are:

deepseek-v4.1-flash
deepseek-v4-pro-0813
gemini-3.8-flash
glm-5.3
gpt-5.6-luna
gpt-5.6-sol
gpt-6-astra
qwen3.5-35b-a3b
qwen3.8-flash
qwen3-coder-next

Health probes cover the model registry in model_selection.json; readiness requires the configured candidate pool. Map API identifiers to the corresponding model versions.

Embedding service

The reference deployment uses Qwen/Qwen3-Embedding-4B, revision 5cf2132abc99cad020ac570b19d031efec650f2b, bfloat16 inference, and 2,560-dimensional vectors.

Download the model and tokenizer

For a local deployment, prepare its separate GPU environment and download the weights and matching tokenizer. These commands do not start the service:

conda create -n execorchestra-embedding python=3.11 -y
conda run -n execorchestra-embedding python -m pip install -r requirements-embedding.txt
python -m scripts.download_embedding

For an existing remote service, download only the matching tokenizer. No GPU environment or weights are needed on the framework host:

python -m scripts.download_embedding --tokenizer-only

Start the local service

After configuration, open a separate terminal at the project directory (ChatDev/execorchestra). Set a private, nonempty SGAR_EMBEDDING_API_KEY in .env and use an NVIDIA GPU supporting BF16. On a dedicated GPU server, start the prepared service with:

conda activate execorchestra-embedding
set -a
source .env
set +a
export SGAR_EMBEDDING_MODEL_PATH="$PWD/.cache/embedding-model"
# On a dedicated GPU server only; Slurm users instead run inside an allocation.
export SGAR_EMBEDDING_DEDICATED_GPU=1
python -m scripts.embedding_server

The service binds to 127.0.0.1:8001. It uses BF16, SDPA, last-token pooling, L2 normalization, and a fixed microbatch of one. It does not silently truncate inputs or add a second retrieval instruction. Stop the foreground process after the task; the module does not start automatically during installation or checks. The model directory must contain the weights and tokenizer for the pinned revision.

An NVIDIA driver and NVIDIA Container Toolkit are required for the alternative vLLM container deployment:

export EMBEDDING_GPU=0
VLLM_API_KEY="$SGAR_EMBEDDING_API_KEY" docker run -d \
  --name execorchestra-embedding --gpus "device=${EMBEDDING_GPU}" \
  --network host --ipc host -e VLLM_API_KEY \
  -v "$PWD/.cache/embedding-model:/models/embedding-4b:ro" \
  vllm/vllm-openai:v0.27.1-cu129-ubuntu2404 \
  /models/embedding-4b --served-model-name sgar-embedding-4b \
  --host 127.0.0.1 --port 8001 --dtype bfloat16 \
  --gpu-memory-utilization 0.80 --max-model-len 32768

The client template points to http://127.0.0.1:8001/v1/embeddings with API model ID sgar-embedding-4b. For a remote service, update runtime.api_endpoint_url and, if necessary, runtime.api_model_id in sgar_mvp/config/embedding_release.json. Keep the model revision, dimension, instructions, and normalization settings unchanged. The host client adds retrieval instructions. Size GPU memory for the configured sequence length and service concurrency.

The framework sends {"model": "sgar-embedding-4b", "input": ["..."]} and expects data entries with unique zero-based index values and 2,560-element embedding arrays. Export the API key even for a local service.

Tool services

The complete candidate list includes the three services below. These endpoints must be reachable from tool containers; 127.0.0.1 inside a container does not refer to the host.

Environment variable Required interface
SEARXNG_BASE_URL SearXNG root URL, with JSON search enabled at /search.
ZOTERO_TRANSLATION_SERVER_URL Translation Server root URL, accepting text identifiers at /search.
HYPOTHESIS_API_URL Hypothesis API root, including /api, accepting annotation creation at /annotations.
HYPOTHESIS_API_TOKEN Write-capable token for an isolated Hypothesis instance.

For SearXNG and Zotero, the following starts separate services on the default Docker bridge gateway. Use an administrator-approved bind address if your host uses different networking:

export DOCKER_GATEWAY="$(docker network inspect bridge -f '{{(index .IPAM.Config 0).Gateway}}')"
mkdir -p .cache/services/searxng
python - <<'PY'
from pathlib import Path
import secrets
p = Path(".cache/services/searxng/settings.yml")
if not p.exists():
    p.write_text("use_default_settings: true\nsearch:\n  formats: [html, json]\n"
                 "server:\n  limiter: false\n  secret_key: " + secrets.token_hex(32) + "\n")
PY
docker run -d --name execorchestra-search \
  -p "${DOCKER_GATEWAY}:8888:8080" \
  -v "$PWD/.cache/services/searxng:/etc/searxng" \
  searxng/searxng@sha256:394bc18556af80cd1cd448daf5b69ead7f65d990912851a47ba1ec79c85fec44
docker run -d --name execorchestra-translation \
  -p "${DOCKER_GATEWAY}:1969:1969" zotero/translation-server:2.0.3

Set SEARXNG_BASE_URL=http://<bridge-gateway>:8888 and ZOTERO_TRANSLATION_SERVER_URL=http://<bridge-gateway>:1969 in .env, replacing <bridge-gateway> with the value above. Include that address in proxy bypass lists. SearXNG also needs outbound access to its configured search engines. See the SearXNG deployment documentation and Zotero Translation Server for service administration.

Use an isolated Hypothesis server and its token for annotation tools. The retained environment used server version v0.39.0. Its upstream deployment includes a database, search backend, and worker; those services and their state are not part of the framework repository. The tool check creates an annotation; use a private test instance. This service is required by the complete candidate pool.

Network configuration

Framework requests use explicit routes:

Setting Applies to
SGAR_MODEL_PROXY Model API traffic and the optional cold-image downloader. Leave empty only when these routes work directly or images are pre-pulled.
SGAR_TOOL_HTTP_PROXY, SGAR_TOOL_HTTPS_PROXY Network-enabled tool containers.
SGAR_TOOL_NO_PROXY Local/internal tool-service addresses bypassing those proxies.
SGAR_DOCKER_PROXY TB task containers; use an HTTP CONNECT proxy reachable from the Docker bridge.
SGAR_NO_PROXY TB container proxy exclusions.
HTTP_PROXY, HTTPS_PROXY, NO_PROXY Host downloads, image builds, and preparation tool subprocesses.

Use container-reachable proxy addresses for tools and TB. Configure SGAR_DOCKER_PROXY explicitly: the adapter's default is a bridge-gateway relay on port 7894. Model clients use SGAR_MODEL_PROXY; embedding requests to loopback or private IP literals bypass it.

Export .env before preparation and execution. Public-network tools require --network-policy declared in the framework/GAIA launcher or --network in the TB launcher; these flags do not override a task's own network restrictions.