mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-05-12 11:43:49 +00:00
94da8f67d7
`make dev` ran `uv sync` unconditionally on every restart, wiping any
optional extras the user had installed manually with
`uv sync --all-packages --extra postgres`. The Docker image-build path
already solved this via the `UV_EXTRAS` build-arg in backend/Dockerfile;
the local serve.sh path and the docker-compose-dev startup command
were the remaining outliers.
`scripts/serve.sh` now resolves extras before `uv sync`:
1. honors `UV_EXTRAS` (parity with backend/Dockerfile and
docker/docker-compose.yaml — no new convention introduced);
2. falls back to parsing config.yaml — `database.backend: postgres`
or legacy `checkpointer.type: postgres` auto-pins
`--extra postgres`, so the common case needs zero extra config.
3. detector stderr is no longer suppressed, so whitelist warnings or
crashes surface to the dev terminal (review feedback).
Detection lives in `scripts/detect_uv_extras.py` (stdlib-only — has to
run before the venv exists). Extra names are validated against
`^[A-Za-z][A-Za-z0-9_-]*$` so a stray shell metacharacter in `.env`
cannot reach `uv sync` downstream (defense in depth).
`docker/docker-compose-dev.yaml`'s startup command is now extracted to
`docker/dev-entrypoint.sh` (review feedback — the inline command had
grown to a ~350-char one-liner). The script:
- parses comma/whitespace-separated UV_EXTRAS, applying the same
`^[A-Za-z][A-Za-z0-9_-]*$` whitelist as the local detector;
- emits one `--extra X` flag per token, so `UV_EXTRAS=postgres,ollama`
works in Docker dev too (harmonized with local — review feedback);
- calls `uv sync --all-packages` (PR #2584) so workspace member
extras (deerflow-harness's postgres extra) are installed;
- keeps the existing self-heal `(uv sync || (recreate venv && retry))`
branch;
- exposes `--print-extras` for dry-run testing.
The compose file mounts the script read-only at runtime, so script
edits take effect on `make docker-restart` without an image rebuild.
The `--no-sync` alternative (a separate suggestion in the issue thread)
was considered but rejected for dev paths because it would drop the
self-heal branch and the auto-pickup of new pyproject deps. `--no-sync`
is already in use for the production CMD (`backend/Dockerfile:101`)
where it's appropriate.
Updates the asyncpg-missing error message to include the
`--all-packages` flag (matching #2584) plus the persistent install flow,
and expands `config.example.yaml` so all three install paths
(local / docker dev / docker image build) are documented with their
multi-extra capabilities.
Tests:
- `tests/test_detect_uv_extras.py` (21 tests) — local-path env parsing,
YAML edge cases, env-vs-config precedence, whitelist rejection of
shell metacharacters.
- `tests/test_dev_entrypoint.py` (15 tests) — docker-path validation
via `--print-extras`, multi-extra parsing, metacharacter abort.
- `tests/test_persistence_scaffold.py` (22 tests, unchanged) — passes
with the merged `--all-packages --extra postgres` error message.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
86 lines
3.6 KiB
Bash
Executable File
86 lines
3.6 KiB
Bash
Executable File
#!/usr/bin/env sh
|
|
#
|
|
# DeerFlow gateway dev entrypoint — runs inside the docker-compose-dev gateway
|
|
# container. Extracted from docker/docker-compose-dev.yaml's inline `command:`
|
|
# (PR #2767, addressing review on Issue #2754).
|
|
#
|
|
# Responsibilities:
|
|
# 1. Resolve `--extra X` flags from UV_EXTRAS (comma- or whitespace-separated,
|
|
# mirroring scripts/detect_uv_extras.py for parity with local `make dev`).
|
|
# 2. Validate each extra against [A-Za-z][A-Za-z0-9_-]* so a stray shell
|
|
# metacharacter in `.env` cannot reach `uv sync`.
|
|
# 3. `uv sync --all-packages` so workspace member extras (deerflow-harness's
|
|
# postgres extra in particular) are installed — see PR #2584.
|
|
# 4. Self-heal: if the first sync fails, recreate .venv and retry once.
|
|
# 5. Hand off to uvicorn with reload, replacing this shell so uvicorn becomes
|
|
# PID 1 inside the container.
|
|
#
|
|
# Anchored at /bin/sh (not bash) since alpine-based base images may not ship
|
|
# bash. Uses POSIX-only constructs throughout.
|
|
|
|
set -e
|
|
|
|
# `--print-extras` is a dry-run hook: parse + validate UV_EXTRAS, print the
|
|
# resulting `--extra X` flags to stdout, and exit. Used by the unit test in
|
|
# backend/tests/test_dev_entrypoint.py and useful for ad-hoc debugging.
|
|
PRINT_EXTRAS_ONLY=0
|
|
if [ "${1:-}" = "--print-extras" ]; then
|
|
PRINT_EXTRAS_ONLY=1
|
|
fi
|
|
|
|
# Mirror the legacy command's behavior: redirect both stdout and stderr to the
|
|
# host-mounted log file (../logs/gateway.log → /app/logs/gateway.log). Skip
|
|
# the redirect under --print-extras so the test runner can capture stdout.
|
|
if [ "$PRINT_EXTRAS_ONLY" = "0" ]; then
|
|
exec >/app/logs/gateway.log 2>&1
|
|
fi
|
|
|
|
# ── Resolve extras ──────────────────────────────────────────────────────────
|
|
|
|
EXTRAS_FLAGS=""
|
|
if [ -n "${UV_EXTRAS:-}" ]; then
|
|
# Normalize comma → space, then split on whitespace via the unquoted `for`.
|
|
for raw in $(printf '%s' "$UV_EXTRAS" | tr ',' ' '); do
|
|
[ -z "$raw" ] && continue
|
|
# Reject anything that does not look like an identifier.
|
|
# Two patterns: leading non-letter, or any non-[A-Za-z0-9_-] character.
|
|
case "$raw" in
|
|
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
|
|
echo "[startup] UV_EXTRAS entry '$raw' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $raw"
|
|
done
|
|
fi
|
|
|
|
if [ "$PRINT_EXTRAS_ONLY" = "1" ]; then
|
|
# Trim leading space for tidier output, then exit.
|
|
printf '%s\n' "${EXTRAS_FLAGS# }"
|
|
exit 0
|
|
fi
|
|
|
|
if [ -n "$EXTRAS_FLAGS" ]; then
|
|
echo "[startup] uv extras:$EXTRAS_FLAGS"
|
|
fi
|
|
|
|
# ── Sync dependencies (with self-heal) ──────────────────────────────────────
|
|
|
|
cd /app/backend
|
|
|
|
# `--all-packages` propagates extras into workspace members (PR #2584).
|
|
# `$EXTRAS_FLAGS` intentionally unquoted so each `--extra X` becomes its own arg.
|
|
# shellcheck disable=SC2086 # word-splitting is intentional here
|
|
if ! uv sync --all-packages $EXTRAS_FLAGS; then
|
|
echo "[startup] uv sync failed; recreating .venv and retrying once"
|
|
uv venv --allow-existing .venv
|
|
# shellcheck disable=SC2086
|
|
uv sync --all-packages $EXTRAS_FLAGS
|
|
fi
|
|
|
|
# ── Hand off to uvicorn ─────────────────────────────────────────────────────
|
|
|
|
PYTHONPATH=. exec uv run uvicorn app.gateway.app:app \
|
|
--host 0.0.0.0 --port 8001 \
|
|
--reload --reload-include='*.yaml .env'
|