diff --git a/CHANGELOG.md b/CHANGELOG.md index 216d230292..9c3f28dd57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -678,6 +678,32 @@ This release closes that milestone with **439 merged pull requests**. before a trailing `Exit Code: N` or `Command exited with code N`, an output that is only `Command exited with code N` is left unchanged, and the result keeps `deerflow_tool_meta`, `artifact` and `id`. ([#6307]) +- **deploy:** The Helm chart and docker-compose no longer ship a multi-replica + trap. The chart's default `config` enables `run_ownership.heartbeat_enabled` + and `run_events.backend: db` (so a starting Pod only reclaims runs whose + owner is really gone, and message history is shared across Pods), exports + `DEER_FLOW_MULTI_INSTANCE` for the Gateway's startup gate from + `gateway.replicas` or an explicit `gateway.multiInstance` (set it before + scaling with `kubectl scale` or an HPA, which the chart cannot see), + generates and injects a shared `AUTH_JWT_SECRET` (per-Pod `.jwt_secret` + files logged users out across replicas; the key is required whenever the + gateway is multi-instance so concurrently booting Pods cannot race to write + different keys), declares a surge-then-drain rollout strategy, renders a + `PodDisruptionBudget` for a multi-instance gateway, and bounds uvicorn's + graceful shutdown (`gateway.uvicornGracefulShutdownSeconds`, 10s; `0` is + honored) inside a 90s termination grace period so an idle SSE stream cannot + push the memory drain into SIGKILL. `existingAppSecret` is now honored by + every consumer (the gateway and frontend kept referencing the generated + `-app` Secret, which was not rendered, so Pods never started). + Both compose files bound the same uvicorn timeout and set + `stop_grace_period: 90s` (Docker's 10s default cut the 30s memory flush + short; the lifespan's worst case is about 61s once every bounded teardown + hook is counted, and the tests read those bounds from the Gateway). The chart README and compose comments stop citing the long-closed + issue #3948 as the reason to stay at one replica and list what actually + remains single-instance (IM channels, WeChat QR login, browser tools). + Upgrading a release that predates `AUTH_JWT_SECRET` generates a new key and + signs every browser session out once; the chart README shows how to seed + the previous key into the Secret first to keep sessions. ([#6347]) - **persistence:** A second Gateway instance no longer fails startup with `TimeoutError` while another instance runs a PostgreSQL schema migration. The bootstrap advisory lock was taken with a blocking `pg_advisory_lock` on the @@ -8782,3 +8808,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag. [#6338]: https://github.com/bytedance/deer-flow/pull/6338 [#6343]: https://github.com/bytedance/deer-flow/pull/6343 [#6344]: https://github.com/bytedance/deer-flow/pull/6344 +[#6347]: https://github.com/bytedance/deer-flow/pull/6347 diff --git a/backend/tests/_gateway_shutdown_budget.py b/backend/tests/_gateway_shutdown_budget.py new file mode 100644 index 0000000000..5538873077 --- /dev/null +++ b/backend/tests/_gateway_shutdown_budget.py @@ -0,0 +1,68 @@ +"""Shutdown work every deployment's stop grace period must cover. + +The chart's ``terminationGracePeriodSeconds`` and both compose files' +``stop_grace_period`` must outlast the Gateway's lifespan shutdown. Besides the +memory queue flush, ``app.gateway.app.lifespan`` runs several *sequential*, +individually bounded hooks around it, and ``langgraph_runtime`` drains in-flight +runs on exit. These numbers are read from the Gateway rather than copied into +the tests, so raising a drain budget or adding a bounded hook here fails the +grace-period tests instead of silently outgrowing the deployments. +""" + +from __future__ import annotations + +from app.gateway.deps import _RUN_DRAIN_TIMEOUT_SECONDS +from deerflow.config.memory_config import MemoryConfig + +#: Teardown steps in ``app.gateway.app.lifespan`` that each wait up to +#: ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS`` before proceeding. They run one after +#: another, so a pathological shutdown can spend the full budget on every one. +#: Keep this list in step with the ``asyncio.wait_for(..., timeout= +#: _SHUTDOWN_HOOK_TIMEOUT_SECONDS)`` sites in that function. +HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT: tuple[str, ...] = ( + "startup trash sweep", + "notification delivery worker stop", + "channel service stop", + "browser session close", + "MCP session pool close", +) + + +def _app_constants() -> tuple[float, float]: + """Return ``(_SHUTDOWN_HOOK_TIMEOUT_SECONDS, _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS)``.""" + # Importing the FastAPI module builds nothing, but it is heavy; keep it lazy. + from app.gateway.app import _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS, _SHUTDOWN_HOOK_TIMEOUT_SECONDS + + return float(_SHUTDOWN_HOOK_TIMEOUT_SECONDS), float(_RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS) + + +def bounded_hooks_seconds() -> float: + """Worst case for the sequential hooks bounded by ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS``.""" + hook_timeout, _ = _app_constants() + return len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT) * hook_timeout + + +def retrieval_warm_wait_seconds() -> float: + """Brief wait for the derived retrieval-index rebuild before the flush.""" + _, retrieval_warm_timeout = _app_constants() + return retrieval_warm_timeout + + +def run_drain_seconds() -> float: + """Bound on draining in-flight runs before the checkpointer is torn down.""" + return float(_RUN_DRAIN_TIMEOUT_SECONDS) + + +def memory_flush_seconds() -> float: + """Default ``memory.shutdown_flush_timeout_seconds``.""" + return float(MemoryConfig.model_fields["shutdown_flush_timeout_seconds"].default) + + +def lifespan_shutdown_seconds() -> float: + """Worst-case lifespan shutdown: every bounded hook + retrieval-warm wait + run drain + memory flush. + + Unbounded steps (OIDC close, scheduler / task-service stops, backend close) + are expected to be quick and are not modeled; deployments keep slack on top + of this figure for them. + """ + return bounded_hooks_seconds() + retrieval_warm_wait_seconds() + run_drain_seconds() + memory_flush_seconds() diff --git a/backend/tests/test_compose_default_workers.py b/backend/tests/test_compose_default_workers.py index 93f173a4fa..2c404ba1bc 100644 --- a/backend/tests/test_compose_default_workers.py +++ b/backend/tests/test_compose_default_workers.py @@ -16,9 +16,13 @@ import re from pathlib import Path import yaml +from _gateway_shutdown_budget import HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT, lifespan_shutdown_seconds REPO_ROOT = Path(__file__).resolve().parents[2] COMPOSE_PATH = REPO_ROOT / "docker" / "docker-compose.yaml" +DEV_COMPOSE_PATH = REPO_ROOT / "docker" / "docker-compose-dev.yaml" +DEV_ENTRYPOINT_PATH = REPO_ROOT / "docker" / "dev-entrypoint.sh" +GATEWAY_APP_PATH = REPO_ROOT / "backend" / "app" / "gateway" / "app.py" def _gateway_command() -> str: @@ -43,3 +47,46 @@ def test_gateway_worker_count_remains_overridable(): """The worker count must stay configurable, not hard-coded to 1.""" command = _gateway_command() assert "${GATEWAY_WORKERS:-1}" in command, f"worker count must use ${{GATEWAY_WORKERS:-1}} so operators can override it; got: {command}" + + +def _graceful_shutdown_bound(launch: str) -> int: + """Return uvicorn's ``--timeout-graceful-shutdown`` bound from a launch command or script.""" + match = re.search(r"--timeout-graceful-shutdown (\d+)", launch) + assert match is not None, f"uvicorn launch must bound its graceful shutdown; got: {launch}" + return int(match.group(1)) + + +def _stop_grace_seconds(compose_path: Path) -> int: + compose = yaml.safe_load(compose_path.read_text(encoding="utf-8")) + grace = compose["services"]["gateway"]["stop_grace_period"] + match = re.fullmatch(r"(\d+)s", str(grace)) + assert match is not None, f"stop_grace_period must be expressed in seconds, got {grace!r}" + return int(match.group(1)) + + +def test_gateway_bounds_uvicorn_graceful_shutdown(): + """Open SSE connections must not hold lifespan shutdown (bounded hooks, run drain, memory flush) past the stop grace period.""" + assert _graceful_shutdown_bound(_gateway_command()) <= 30 + + +def test_gateway_stop_grace_period_covers_the_shutdown_work(): + """Docker's 10s default SIGKILLed the 30s memory queue flush on every restart.""" + assert _stop_grace_seconds(COMPOSE_PATH) >= _graceful_shutdown_bound(_gateway_command()) + lifespan_shutdown_seconds() + + +def test_dev_gateway_bounds_uvicorn_graceful_shutdown_and_covers_it(): + """The dev stack launches uvicorn from dev-entrypoint.sh and is restarted far more often.""" + bound = _graceful_shutdown_bound(DEV_ENTRYPOINT_PATH.read_text(encoding="utf-8")) + assert bound <= 30 + assert _stop_grace_seconds(DEV_COMPOSE_PATH) >= bound + lifespan_shutdown_seconds() + + +def test_shutdown_budget_models_every_bounded_lifespan_hook(): + """Each ``wait_for(..., timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS)`` in the lifespan is a sequential 5s worst case the grace periods must cover.""" + source = GATEWAY_APP_PATH.read_text(encoding="utf-8") + bounded_sites = source.count("timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS") + assert bounded_sites == len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT), ( + f"app.gateway.app bounds {bounded_sites} teardown hook(s) with _SHUTDOWN_HOOK_TIMEOUT_SECONDS but " + f"_gateway_shutdown_budget models {len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT)}; update HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT " + "so the chart and compose grace periods keep covering the worst case" + ) diff --git a/backend/tests/test_helm_multi_replica_prereqs.py b/backend/tests/test_helm_multi_replica_prereqs.py new file mode 100644 index 0000000000..1e5e295cb0 --- /dev/null +++ b/backend/tests/test_helm_multi_replica_prereqs.py @@ -0,0 +1,170 @@ +"""Regression tests for the Helm chart's multi-replica prerequisites. + +The chart used to ship ``gateway.replicas: 1`` with a README that cited the +long-closed issue #3948, while its rendered config lacked the two settings a +second Pod actually needs: run-ownership heartbeats (without a lease, a +starting Pod writes every peer's live run off as an orphan) and +database-backed run events. These tests pin the config block, the +``DEER_FLOW_MULTI_INSTANCE`` declaration the Gateway's startup gate reads +(derived from ``gateway.replicas`` or declared with ``gateway.multiInstance`` +for deployments scaled by kubectl/HPA), the shared ``AUTH_JWT_SECRET`` that is +required once more than one instance runs, the ``existingAppSecret`` wiring, +the rollout strategy, the PodDisruptionBudget, and a termination grace period +that covers the shutdown work. + +The ``helm template`` tests skip when helm is not installed; CI's runner has it. +""" + +from __future__ import annotations + +import re +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml +from _gateway_shutdown_budget import lifespan_shutdown_seconds + +REPO_ROOT = Path(__file__).resolve().parents[2] +CHART = REPO_ROOT / "deploy" / "helm" / "deer-flow" +VALUES = CHART / "values.yaml" +APP_SECRET_TEMPLATE = CHART / "templates" / "secret-app.yaml" +README = CHART / "README.md" + + +def _values() -> dict: + return yaml.safe_load(VALUES.read_text(encoding="utf-8")) + + +def _rendered_config() -> dict: + return yaml.safe_load(_values()["config"]) + + +def _render_chart(*settings: str) -> list[dict]: + helm = shutil.which("helm") + if helm is None: + pytest.skip("helm is unavailable") + command = [helm, "template", "deer-flow", str(CHART)] + for setting in settings: + command.extend(["--set", setting]) + rendered = subprocess.run(command, check=True, capture_output=True, text=True).stdout + return [document for document in yaml.safe_load_all(rendered) if isinstance(document, dict)] + + +def _by_kind(documents: list[dict], kind: str, name_suffix: str) -> dict: + return next(document for document in documents if document.get("kind") == kind and document["metadata"]["name"].endswith(name_suffix)) + + +def _has_kind(documents: list[dict], kind: str) -> bool: + return any(document.get("kind") == kind for document in documents) + + +def _container(documents: list[dict], deployment_suffix: str, container_name: str) -> dict: + containers = _by_kind(documents, "Deployment", deployment_suffix)["spec"]["template"]["spec"]["containers"] + return next(container for container in containers if container["name"] == container_name) + + +def _env(container: dict) -> dict[str, dict]: + return {item["name"]: item for item in container["env"]} + + +def _secret_ref(container: dict, env_name: str) -> dict: + return _env(container)[env_name]["valueFrom"]["secretKeyRef"] + + +def test_default_config_enables_the_multi_replica_prerequisites() -> None: + config = _rendered_config() + assert config["database"]["backend"] == "postgres" + assert config["checkpointer"]["type"] == "postgres" + assert config["stream_bridge"]["type"] == "redis" + assert config["run_ownership"]["heartbeat_enabled"] is True, "without a lease every peer run is reclaimed as an orphan on Pod start" + assert config["run_events"]["backend"] == "db", "memory run events are process-local" + + +def test_default_replica_count_stays_conservative_with_a_rollout_budget() -> None: + values = _values() + gateway = values["gateway"] + assert gateway["replicas"] == 1 + assert gateway["multiInstance"] is None, "derived from replicas unless the operator declares it" + assert gateway["strategy"]["type"] == "RollingUpdate" + assert gateway["strategy"]["rollingUpdate"]["maxUnavailable"] == 0 + assert gateway["strategy"]["rollingUpdate"]["maxSurge"] == 1 + assert gateway["podDisruptionBudget"]["enabled"] is True + assert gateway["podDisruptionBudget"]["minAvailable"] == 1 + assert values["existingAppSecret"] == "" + + +def test_termination_grace_period_covers_the_shutdown_work() -> None: + gateway = _values()["gateway"] + required = gateway["preStopSleepSeconds"] + gateway["uvicornGracefulShutdownSeconds"] + lifespan_shutdown_seconds() + assert gateway["terminationGracePeriodSeconds"] > required, f"grace period must exceed {required}s of shutdown work" + + +def test_app_secret_template_preserves_the_jwt_secret_across_upgrades() -> None: + """``helm template`` never sees ``lookup`` results, so pin the preservation in the source.""" + template = APP_SECRET_TEMPLATE.read_text(encoding="utf-8") + assert 'index $prev.data "AUTH_JWT_SECRET"' in template, "must survive upgrades like the other app secrets" + assert "AUTH_JWT_SECRET: {{ $jwtSecret | quote }}" in template + + +def test_readme_documents_the_multi_instance_knobs() -> None: + readme = README.read_text(encoding="utf-8") + assert re.search(r"do not raise\s+`gateway\.replicas` past 1", readme) is None + for needle in ("DEER_FLOW_MULTI_INSTANCE", "AUTH_JWT_SECRET", "gateway.multiInstance", "existingAppSecret", ".jwt_secret"): + assert needle in readme, needle + + +@pytest.mark.parametrize("replicas", [1, 2, 3]) +def test_rendered_declaration_follows_the_replica_count(replicas: int) -> None: + values = _values()["gateway"] + documents = _render_chart(f"gateway.replicas={replicas}") + gateway = _container(documents, "-gateway", "gateway") + multi_instance = replicas > 1 + + assert _env(gateway)["DEER_FLOW_MULTI_INSTANCE"]["value"] == str(multi_instance).lower() + jwt = _secret_ref(gateway, "AUTH_JWT_SECRET") + assert jwt["name"].endswith("-app") + assert jwt["key"] == "AUTH_JWT_SECRET" + assert jwt["optional"] is (not multi_instance), "the shared key is required once Pods could race to write .jwt_secret" + assert f"--timeout-graceful-shutdown {values['uvicornGracefulShutdownSeconds']}" in " ".join(gateway["args"]) + + assert _has_kind(documents, "PodDisruptionBudget") is multi_instance + if multi_instance: + assert _by_kind(documents, "PodDisruptionBudget", "-gateway")["spec"]["minAvailable"] == values["podDisruptionBudget"]["minAvailable"] + + deployment = _by_kind(documents, "Deployment", "-gateway") + assert deployment["spec"]["strategy"] == values["strategy"] + assert deployment["spec"]["template"]["spec"]["terminationGracePeriodSeconds"] == values["terminationGracePeriodSeconds"] + + +def test_explicit_multi_instance_declaration_wins_over_the_replica_count() -> None: + """kubectl scale and HPAs change the count without a Helm upgrade, so the operator declares it.""" + documents = _render_chart("gateway.replicas=1", "gateway.multiInstance=true") + gateway = _container(documents, "-gateway", "gateway") + assert _env(gateway)["DEER_FLOW_MULTI_INSTANCE"]["value"] == "true" + assert _secret_ref(gateway, "AUTH_JWT_SECRET")["optional"] is False + assert _has_kind(documents, "PodDisruptionBudget") + + +def test_pdb_can_be_disabled_and_a_zero_graceful_shutdown_is_honored() -> None: + documents = _render_chart("gateway.replicas=2", "gateway.podDisruptionBudget.enabled=false", "gateway.uvicornGracefulShutdownSeconds=0") + assert not _has_kind(documents, "PodDisruptionBudget") + assert "--timeout-graceful-shutdown 0" in " ".join(_container(documents, "-gateway", "gateway")["args"]) + + +def test_existing_app_secret_is_honored_by_every_consumer() -> None: + """The helper used to ignore the value, so Pods referenced a Secret the chart no longer rendered.""" + documents = _render_chart("existingAppSecret=my-app-secret", "gateway.replicas=2") + assert not any(document.get("kind") == "Secret" and document["metadata"]["name"].endswith("-app") for document in documents) + gateway = _container(documents, "-gateway", "gateway") + assert _secret_ref(gateway, "DEER_FLOW_INTERNAL_AUTH_TOKEN")["name"] == "my-app-secret" + assert _secret_ref(gateway, "AUTH_JWT_SECRET")["name"] == "my-app-secret" + frontend = _container(documents, "-frontend", "frontend") + assert _secret_ref(frontend, "BETTER_AUTH_SECRET")["name"] == "my-app-secret" + + +def test_rendered_app_secret_carries_the_jwt_secret() -> None: + documents = _render_chart() + secret = _by_kind(documents, "Secret", "-app") + assert secret["stringData"]["AUTH_JWT_SECRET"] diff --git a/deploy/helm/deer-flow/README.md b/deploy/helm/deer-flow/README.md index 69e4b4f2d2..71b58f9775 100644 --- a/deploy/helm/deer-flow/README.md +++ b/deploy/helm/deer-flow/README.md @@ -267,17 +267,57 @@ kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002 `%40`). The chart uses an external `databaseUrl` verbatim and does not rewrite the DSN in a user-managed Secret. -- **Graceful shutdown & memory drain.** The gateway pod sets `terminationGracePeriodSeconds` (default 45s, overridable via `gateway.terminationGracePeriodSeconds`) plus an optional `preStop` sleep (`gateway.preStopSleepSeconds`, default 5s). The grace period MUST exceed the Gateway's graceful-shutdown work — channel stop (~5s) plus the memory-queue drain (`memory.shutdown_flush_timeout_seconds`, default 30s) plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. **When you raise `memory.shutdown_flush_timeout_seconds`, raise `gateway.terminationGracePeriodSeconds` to match** (channel stop + drain + buffer). -- **Gateway replicas.** Postgres + the Redis stream bridge together make the - gateway's *persisted* state (checkpointer + run/thread metadata) and *live - stream* path cross-pod-safe. The default is still 1 replica: **do not raise - `gateway.replicas` past 1 yet.** Run control — `create_or_reject` dedup, - `cancel`, and orphan reconciliation — is still worker-local (in-process - `asyncio.Lock` + in-memory `record.task`), tracked by [issue - #3948](https://github.com/bytedance/deer-flow/issues/3948). With >1 replica a - double-submit can create two runs on one thread (checkpoint corruption), a - cancel can land on a non-owner pod (409), and a crashed pod's runs stay - `pending`/`running` forever. Stay on 1 replica until that work lands. +- **Graceful shutdown & memory drain.** The gateway pod sets `terminationGracePeriodSeconds` (default 90s, overridable via `gateway.terminationGracePeriodSeconds`) plus an optional `preStop` sleep (`gateway.preStopSleepSeconds`, default 5s), and bounds uvicorn's `--timeout-graceful-shutdown` (`gateway.uvicornGracefulShutdownSeconds`, default 10s) so an idle SSE connection cannot hold up lifespan shutdown indefinitely. The grace period MUST exceed the Gateway's graceful-shutdown work — the preStop sleep, the uvicorn timeout, and the lifespan's worst case: five hooks bounded at 5s each (startup trash sweep, notification delivery worker, channel service, browser sessions, MCP session pool), the 1s retrieval-warm wait, the in-flight run drain (5s) and the memory-queue drain (`memory.shutdown_flush_timeout_seconds`, default 30s), about 61s in total, plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. `backend/tests/_gateway_shutdown_budget.py` reads these bounds from the Gateway and pins the chart and compose budgets against them. **When you raise `memory.shutdown_flush_timeout_seconds`, raise `gateway.terminationGracePeriodSeconds` to match** (preStop + uvicorn timeout + ~31s of bounded hooks and drains + memory drain + buffer). +- **Gateway replicas.** Run control is cross-pod-safe since the work tracked + by [issue #3948](https://github.com/bytedance/deer-flow/issues/3948) landed + (#4003, #4064, #4500): admission is a durable one-active-run-per-thread + constraint with `Idempotency-Key` reuse, a cancel that lands on a non-owner + Pod is recorded durably and executed by the owner on its next lease renewal, + and a crashed Pod's runs are reclaimed by a peer once their lease expires. + The chart's default `config` enables what that needs — Postgres for + `database` and `checkpointer`, `run_events.backend: db`, + `run_ownership.heartbeat_enabled: true`, and the Redis stream bridge + (sandbox ownership leases are inferred from it) — and exports + `DEER_FLOW_MULTI_INSTANCE=true` whenever `gateway.replicas > 1`, so the + Gateway's startup gate verifies those prerequisites instead of staying inert + (its worker count only sees one Pod). If you scale with `kubectl scale` or + an HPA instead of `gateway.replicas`, set `gateway.multiInstance: true` + first: those change the replica count without a Helm upgrade, so the + derived declaration, the PodDisruptionBudget and the required shared + `AUTH_JWT_SECRET` would otherwise keep their single-replica rendering. + Mind the window between setting it and the actual scale-up: the + PodDisruptionBudget (`minAvailable: 1`) is rendered immediately and blocks + every voluntary eviction and node drain while only one Pod exists, so set + `gateway.podDisruptionBudget.enabled: false` for that interim if you need + to drain nodes first. `gateway.replicas: 1` stays the + default because the following are still single-instance: **IM channels** + (every Pod would connect to every platform — Telegram polling conflicts and + Discord double-processes; keep 1 replica while channels are enabled), the + WeChat QR login, and browser tools. Before raising replicas, also set + `persistence.home.accessMode: ReadWriteMany` on multi-node clusters (thread + uploads, outputs, memory and `extensions_config.json` live on that volume) + and `agent_storage.backend: db` so custom agents are visible on every Pod. + A `PodDisruptionBudget` (`minAvailable: 1`) is rendered automatically for a + multi-instance gateway (same rule), and the rollout strategy is + surge-then-drain (`maxSurge: 1`, `maxUnavailable: 0`). +- **App secret.** `-app` holds `BETTER_AUTH_SECRET`, + `DEER_FLOW_INTERNAL_AUTH_TOKEN` and `AUTH_JWT_SECRET` (the session-cookie + signing key), each generated once and preserved across upgrades via + `lookup`. `existingAppSecret` points the gateway and frontend at a Secret + you manage instead (no `-app` is generated); it must carry all + three keys. `AUTH_JWT_SECRET` is required whenever the gateway is + multi-instance — without it, concurrently booting Pods race to write their + own `.jwt_secret` on the home volume and sign sessions with different keys — + and only a single Pod may omit it and fall back to that file. **Upgrading a + release that predates `AUTH_JWT_SECRET`** generates a new key, so every + existing browser session is signed out once. To keep sessions, copy the key + the gateway has been using into the Secret before upgrading; `lookup` then + preserves it like the other app secrets: + + ```bash + JWT=$(kubectl -n deer-flow exec deploy/deer-flow-gateway -- cat /app/backend/.deer-flow/.jwt_secret) + kubectl -n deer-flow patch secret deer-flow-app -p "{\"stringData\":{\"AUTH_JWT_SECRET\":\"$JWT\"}}" + ``` - **Scheduled task recovery.** If a deployment explicitly enables `scheduler.multi_instance: true`, it must use shared Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`. @@ -285,8 +325,8 @@ kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002 atomically takes over only expired leases, and fences stale post-launch bookkeeping. `max_concurrent_runs` is a shared global cap across Pods, including pre-launch dispatch reservations. Restart all Gateway Pods after - changing these startup-only settings. This does not remove the broader - Gateway replica limitations described above. + changing these startup-only settings. IM channels remain single-instance + regardless (see Gateway replicas above). - **Redis stream bridge.** A bundled single-instance redis StatefulSet (`redis.enabled: true`, `redis:7-alpine`) runs in the namespace and the gateway connects via the in-cluster Service. Per-run SSE events are stored in diff --git a/deploy/helm/deer-flow/templates/NOTES.txt b/deploy/helm/deer-flow/templates/NOTES.txt index 2003abfc03..b2e09b7123 100644 --- a/deploy/helm/deer-flow/templates/NOTES.txt +++ b/deploy/helm/deer-flow/templates/NOTES.txt @@ -69,7 +69,8 @@ Provisioner / sandbox notes: Generated secrets (persisted across upgrades): - • BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN are stored in the + • BETTER_AUTH_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and AUTH_JWT_SECRET (the + session-cookie signing key shared by every gateway Pod) are stored in the Secret `{{ include "deer-flow.appSecret" . }}`. Database ({{ if .Values.postgresql.enabled }}bundled postgres{{ else }}external postgres{{ end }}): diff --git a/deploy/helm/deer-flow/templates/_helpers.tpl b/deploy/helm/deer-flow/templates/_helpers.tpl index 45b9d3ef37..a6a41d0f17 100644 --- a/deploy/helm/deer-flow/templates/_helpers.tpl +++ b/deploy/helm/deer-flow/templates/_helpers.tpl @@ -70,9 +70,26 @@ imagePullSecrets: {{- else -}}{{- printf "%s-provider" (include "deer-flow.fullname" .) -}}{{- end -}} {{- end -}} -{{/* Name of the Secret holding generated app secrets (auth token, better-auth). */}} +{{/* Name of the Secret holding the app secrets (BETTER_AUTH_SECRET, + DEER_FLOW_INTERNAL_AUTH_TOKEN, AUTH_JWT_SECRET). `existingAppSecret` points + every consumer (gateway, frontend) at a user-managed Secret instead and + skips generating one; it must carry all three keys for more than one + gateway instance. */}} {{- define "deer-flow.appSecret" -}} -{{- printf "%s-app" (include "deer-flow.fullname" .) -}} +{{- if .Values.existingAppSecret -}}{{- .Values.existingAppSecret -}} +{{- else -}}{{- printf "%s-app" (include "deer-flow.fullname" .) -}}{{- end -}} +{{- end -}} + +{{/* "true" when more than one gateway instance shares the database. An explicit + boolean `gateway.multiInstance` wins -- set it before scaling with + `kubectl scale` or an HPA, which change the replica count without a Helm + upgrade -- otherwise it is derived from `gateway.replicas`. */}} +{{- define "deer-flow.gatewayMultiInstance" -}} +{{- if kindIs "invalid" .Values.gateway.multiInstance -}} +{{- gt (int .Values.gateway.replicas) 1 -}} +{{- else -}} +{{- eq (toString .Values.gateway.multiInstance | lower) "true" -}} +{{- end -}} {{- end -}} {{/* Name of the postgres StatefulSet/Service. */}} diff --git a/deploy/helm/deer-flow/templates/gateway-deployment.yaml b/deploy/helm/deer-flow/templates/gateway-deployment.yaml index 095f059e84..1cd2a51ba0 100644 --- a/deploy/helm/deer-flow/templates/gateway-deployment.yaml +++ b/deploy/helm/deer-flow/templates/gateway-deployment.yaml @@ -8,6 +8,10 @@ metadata: app.kubernetes.io/component: gateway spec: replicas: {{ .Values.gateway.replicas }} + {{- with .Values.gateway.strategy }} + strategy: + {{- toYaml . | nindent 4 }} + {{- end }} selector: matchLabels: {{- include "deer-flow.selectorLabels" . | nindent 6 }} @@ -24,10 +28,14 @@ spec: checksum/extensions: {{ include "deer-flow.extensionsChecksum" . }} spec: {{- include "deer-flow.imagePullSecrets" . | nindent 6 }} - # MUST exceed channel stop (~5s) + memory shutdown_flush_timeout_seconds - # (default 30s) + buffer, or K8s SIGKILLs the memory drain mid-flight - # (values.yaml gateway.terminationGracePeriodSeconds). K8s defaults to 30s. - terminationGracePeriodSeconds: {{ .Values.gateway.terminationGracePeriodSeconds | default 45 }} + # MUST exceed preStop sleep + uvicorn --timeout-graceful-shutdown + the + # lifespan's worst case -- five hooks bounded at 5s each (trash sweep, + # notification worker, channel service, browser sessions, MCP session + # pool), the 1s retrieval-warm wait, the 5s run drain and the memory + # shutdown_flush_timeout_seconds (default 30s): ~61s -- plus buffer, or + # K8s SIGKILLs the memory drain mid-flight (values.yaml + # gateway.terminationGracePeriodSeconds). K8s defaults to 30s. + terminationGracePeriodSeconds: {{ .Values.gateway.terminationGracePeriodSeconds | default 90 }} securityContext: # Non-root: the gateway image has no USER directive and would otherwise # run as root. uid/gid 1000; fsGroup 1000 so the .deer-flow PVC is @@ -83,7 +91,7 @@ spec: workingDir: /app command: ["sh", "-c"] args: - - cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers 1 + - cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers 1 --timeout-graceful-shutdown {{ .Values.gateway.uvicornGracefulShutdownSeconds | int }} env: - name: CI value: "true" @@ -118,6 +126,26 @@ spec: secretKeyRef: name: {{ include "deer-flow.appSecret" . }} key: DEER_FLOW_INTERNAL_AUTH_TOKEN + # Session-cookie signing key shared by every Pod. Without it each + # Pod runs _load_or_create_secret(), which read-then-writes + # .jwt_secret on the home volume: concurrently booting replicas race + # and sign sessions with different keys. The key is therefore + # required whenever more than one instance runs; only a single Pod + # with a user-managed existingAppSecret that lacks it may fall back + # to the file. + - name: AUTH_JWT_SECRET + valueFrom: + secretKeyRef: + name: {{ include "deer-flow.appSecret" . }} + key: AUTH_JWT_SECRET + optional: {{ ne (include "deer-flow.gatewayMultiInstance" .) "true" }} + # Tells the Gateway's startup gate that peers share the database; + # its GATEWAY_WORKERS count only sees this Pod. Derived from + # gateway.replicas unless gateway.multiInstance is set explicitly: + # kubectl scale and HPAs change the replica count without a Helm + # upgrade, so set multiInstance: true before scaling that way. + - name: DEER_FLOW_MULTI_INSTANCE + value: {{ include "deer-flow.gatewayMultiInstance" . | quote }} {{- /* PostgreSQL DSN — only mounted when postgres is configured (bundled StatefulSet, external databaseUrl, or an existing Secret). In sqlite/memory mode the env is omitted and diff --git a/deploy/helm/deer-flow/templates/gateway-pdb.yaml b/deploy/helm/deer-flow/templates/gateway-pdb.yaml new file mode 100644 index 0000000000..29c0d0b45b --- /dev/null +++ b/deploy/helm/deer-flow/templates/gateway-pdb.yaml @@ -0,0 +1,24 @@ +{{- /* + PodDisruptionBudget for the gateway. Rendered only for a multi-instance + gateway (gateway.replicas > 1, or gateway.multiInstance: true for + deployments scaled with kubectl/HPA): with a single replica a budget of 1 + would block every node drain, and with several it keeps at least one Pod + serving (and renewing its run leases) while nodes are drained. +*/ -}} +{{- $pdb := .Values.gateway.podDisruptionBudget | default dict -}} +{{- if and $pdb.enabled (eq (include "deer-flow.gatewayMultiInstance" .) "true") }} +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: {{ include "deer-flow.fullname" . }}-gateway + namespace: {{ include "deer-flow.namespace" . }} + labels: + {{- include "deer-flow.labels" . | nindent 4 }} + app.kubernetes.io/component: gateway +spec: + minAvailable: {{ $pdb.minAvailable | default 1 | int }} + selector: + matchLabels: + {{- include "deer-flow.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: gateway +{{- end }} diff --git a/deploy/helm/deer-flow/templates/secret-app.yaml b/deploy/helm/deer-flow/templates/secret-app.yaml index 03c48fa66e..5fb67023ca 100644 --- a/deploy/helm/deer-flow/templates/secret-app.yaml +++ b/deploy/helm/deer-flow/templates/secret-app.yaml @@ -2,12 +2,18 @@ {{- $prev := lookup "v1" "Secret" (include "deer-flow.namespace" .) (include "deer-flow.appSecret" .) -}} {{- $betterAuth := "" -}} {{- $internalToken := "" -}} +{{- $jwtSecret := "" -}} {{- if $prev -}} {{- $betterAuth = index $prev.data "BETTER_AUTH_SECRET" | default "" | b64dec -}} {{- $internalToken = index $prev.data "DEER_FLOW_INTERNAL_AUTH_TOKEN" | default "" | b64dec -}} +{{- $jwtSecret = index $prev.data "AUTH_JWT_SECRET" | default "" | b64dec -}} {{- end -}} {{- if not $betterAuth -}}{{- $betterAuth = randAlphaNum 48 | lower -}}{{- end -}} {{- if not $internalToken -}}{{- $internalToken = randAlphaNum 40 -}}{{- end -}} +{{- /* Session-cookie signing key. Generated once and preserved across upgrades + so sessions survive, and shared by every gateway Pod so a session signed + on one replica validates on another. */ -}} +{{- if not $jwtSecret -}}{{- $jwtSecret = randAlphaNum 48 -}}{{- end -}} apiVersion: v1 kind: Secret metadata: @@ -19,4 +25,5 @@ type: Opaque stringData: BETTER_AUTH_SECRET: {{ $betterAuth | quote }} DEER_FLOW_INTERNAL_AUTH_TOKEN: {{ $internalToken | quote }} + AUTH_JWT_SECRET: {{ $jwtSecret | quote }} {{- end -}} diff --git a/deploy/helm/deer-flow/values.yaml b/deploy/helm/deer-flow/values.yaml index 943664cc72..c7f867a18d 100644 --- a/deploy/helm/deer-flow/values.yaml +++ b/deploy/helm/deer-flow/values.yaml @@ -25,18 +25,63 @@ image: # -- Gateway (backend) deployment. gateway: - replicas: 1 # Safe default. Postgres + the Redis stream bridge are - # wired, but multi-replica needs issue #3948's run-control - # work (cancel/dedup/reconcile). Scheduler multi_instance - # recovery is safe only when its config prerequisites are met; - # max_concurrent_runs is shared across Pods in that mode. + # Replica count. 1 is the conservative default; >1 is supported for the web + # and API path once the prerequisites below hold, all of which this chart + # sets in `config` by default: Postgres (`database` + `checkpointer`), + # `run_events.backend: db`, `run_ownership.heartbeat_enabled: true` and the + # Redis stream bridge (sandbox ownership leases are inferred from it). The + # chart also exports DEER_FLOW_MULTI_INSTANCE=true when replicas > 1 so the + # Gateway's startup gate enforces those prerequisites instead of staying + # inert (its worker count only sees one Pod). Still single-instance: IM + # channels (every Pod connects to every platform -- keep replicas at 1 + # while channels are enabled), the WeChat QR login, and browser tools. + # Multi-node clusters also need `persistence.home.accessMode: ReadWriteMany`, + # and `agent_storage.backend: db` keeps custom agents visible on every Pod. + # Scheduler multi_instance recovery is safe only when its config + # prerequisites are met; max_concurrent_runs is shared across Pods then. + replicas: 1 + # Explicit multi-instance declaration (boolean). Leave unset (null) to derive + # it from `replicas` above. Set `true` before scaling the gateway with + # `kubectl scale` or an HPA: those change the replica count without a Helm + # upgrade, so the derived DEER_FLOW_MULTI_INSTANCE, the PodDisruptionBudget + # and the required shared AUTH_JWT_SECRET would otherwise keep their + # single-replica rendering. Note the PodDisruptionBudget (minAvailable: 1) + # is rendered as soon as this is true: until the scale-up actually happens + # it blocks voluntary evictions and node drains of the single Pod. Set + # podDisruptionBudget.enabled: false for that interim if you need to drain. + multiInstance: null + # Rollout strategy. Surge-then-drain keeps one Pod serving while a peer + # starts; with run-ownership heartbeats the new Pod leaves the old Pod's live + # runs alone (their leases keep renewing) while the old Pod drains them. + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + # PodDisruptionBudget, rendered only for a multi-instance gateway (replicas + # > 1 or multiInstance: true; a budget of 1 on a single replica would block + # node drains). minAvailable falls back to 1 when unset; disable the budget + # rather than setting it to 0. + podDisruptionBudget: + enabled: true + minAvailable: 1 + # uvicorn --timeout-graceful-shutdown (seconds): how long open HTTP + # connections (long SSE streams included) may hold up lifespan shutdown + # after SIGTERM before uvicorn closes them. Without a bound one idle SSE + # client can push the whole shutdown past the grace period into SIGKILL. + # 0 closes open connections immediately on SIGTERM. + uvicornGracefulShutdownSeconds: 10 # Pod termination grace period (seconds). MUST exceed the Gateway's graceful - # shutdown work: channel stop (~5s) + memory queue drain - # (memory.shutdown_flush_timeout_seconds, default 30s) + buffer. K8s defaults + # shutdown work: preStopSleepSeconds + uvicornGracefulShutdownSeconds + the + # lifespan's worst case (five 5s-bounded hooks: trash sweep, notification + # worker, channel service, browser sessions, MCP session pool; the 1s + # retrieval-warm wait; the 5s run drain; the memory queue drain, + # memory.shutdown_flush_timeout_seconds, default 30s -- about 61s) + buffer. + # backend/tests/_gateway_shutdown_budget.py pins the arithmetic. K8s defaults # to 30s, which SIGKILLs the memory drain mid-flight and silently re-introduces # the memory loss that shutdown_flush is fixing. Raise this when you raise the # memory timeout. See gateway-deployment.yaml + README "Graceful shutdown". - terminationGracePeriodSeconds: 45 + terminationGracePeriodSeconds: 90 # preStop hook sleep (seconds) before SIGTERM, so the Service/ingress # deregisters the pod and in-flight HTTP requests drain before graceful # shutdown begins (which runs the memory queue drain). 0 disables the hook. @@ -242,6 +287,11 @@ skills: secrets: {} # -- Use an existing Secret instead of creating one from `secrets` above. existingSecret: "" +# -- Use an existing Secret for the app secrets instead of generating +# `-app`. It must hold BETTER_AUTH_SECRET (frontend), +# DEER_FLOW_INTERNAL_AUTH_TOKEN and, for more than one gateway instance, +# AUTH_JWT_SECRET (the session-cookie signing key every Pod must share). +existingAppSecret: "" # -- Ingress in front of nginx (port 2026). nginx preserves all internal routing. ingress: @@ -319,6 +369,25 @@ config: | stream_bridge: type: redis + # Multi-replica prerequisites (harmless at 1 replica). Heartbeats give each + # run a lease, so a starting Pod only reclaims runs whose owner is really + # gone instead of writing every peer's live run off as an orphan; db run + # events share message history and delivery receipts across Pods. Sandbox + # ownership leases (#4206) are inferred from the Redis stream bridge above. + # Both are startup-only: restart every Gateway Pod together after changing + # them. + run_ownership: + heartbeat_enabled: true + run_events: + backend: db + + # Custom agents and managed subagents default to per-Pod files under the + # home volume. Switch to the shared database when running more than one + # replica (import existing files once with + # `python backend/scripts/migrate_agents_to_db.py`): + # agent_storage: + # backend: db + memory: storage_path: memory.json diff --git a/docker/dev-entrypoint.sh b/docker/dev-entrypoint.sh index e6cb1d04e7..f5669eea6e 100755 --- a/docker/dev-entrypoint.sh +++ b/docker/dev-entrypoint.sh @@ -162,8 +162,13 @@ fi # ── Hand off to uvicorn ───────────────────────────────────────────────────── +# --timeout-graceful-shutdown bounds how long open SSE connections may delay +# lifespan shutdown (memory flush, run drain) after SIGTERM; the compose +# service's stop_grace_period (90s) covers this bound plus the lifespan's +# worst-case drain work (~61s). PYTHONPATH=. exec uv run --no-sync uvicorn app.gateway.app:app \ --host 0.0.0.0 --port 8001 \ + --timeout-graceful-shutdown 10 \ --reload \ --reload-include='*.yaml' \ --reload-include='.env' \ diff --git a/docker/docker-compose-dev.yaml b/docker/docker-compose-dev.yaml index 18cf69b320..7bf4e0590a 100644 --- a/docker/docker-compose-dev.yaml +++ b/docker/docker-compose-dev.yaml @@ -194,6 +194,12 @@ services: # this file readable and lets the script be linted (shellcheck-clean). # See PR #2767 / Issue #2754. command: ["sh", "/usr/local/bin/dev-entrypoint.sh"] + # Docker's 10s default SIGKILLed the 30s memory queue flush + # (memory.shutdown_flush_timeout_seconds) on every `make docker-restart`; + # dev-entrypoint.sh bounds uvicorn's graceful shutdown to 10s inside this. + # Must cover that bound plus the lifespan's worst case (~61s, see the + # gateway comment in docker-compose.yaml) with slack. + stop_grace_period: 90s volumes: # Mount the dev entrypoint as a read-only file so edits to the script # take effect on `make docker-restart` without requiring an image rebuild. diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index 043befbc15..cb10792330 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -109,13 +109,24 @@ services: NPM_REGISTRY: ${NPM_REGISTRY:-} LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65} container_name: deer-flow-gateway - # Gateway hosts the agent runtime with in-process RunManager + StreamBridge - # singletons -- run state lives in this worker's memory. Default to a single - # worker: the Redis stream bridge solves cross-worker SSE delivery and - # reconnect, but run cancel, request dedup, and per-worker IM channel - # services remain worker-local. Override GATEWAY_WORKERS only when the - # Redis stream bridge is enabled and those limitations are acceptable. - command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}" + # Gateway hosts the agent runtime (RunManager + StreamBridge) in process. + # Default to a single worker: GATEWAY_WORKERS > 1 is supported only with + # Postgres (database.backend: postgres, run_events.backend: db, + # run_ownership.heartbeat_enabled: true) and the Redis stream bridge wired + # below -- the Gateway refuses to start otherwise. IM channels, the WeChat + # QR login and browser tools stay worker-local, so keep one worker while + # they are enabled. --timeout-graceful-shutdown bounds how long open SSE + # connections may delay lifespan shutdown after SIGTERM; stop_grace_period + # below must cover that bound plus the lifespan's worst case: five hooks + # bounded at 5s each (trash sweep, notification worker, channel service, + # browser sessions, MCP session pool), the 1s retrieval-warm wait, the 5s + # run drain and the 30s memory flush (memory.shutdown_flush_timeout_seconds) + # -- about 61s -- plus slack. backend/tests/_gateway_shutdown_budget.py + # pins the arithmetic. + command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1} --timeout-graceful-shutdown 10" + # Docker's 10s default SIGKILLed the 30s memory queue flush + # (memory.shutdown_flush_timeout_seconds) on every restart. + stop_grace_period: 90s volumes: - ${DEER_FLOW_CONFIG_PATH}:/app/backend/config.yaml:ro # Writable on purpose: the Gateway edits this file at runtime (MCP diff --git a/frontend/src/content/en/application/deployment-guide.mdx b/frontend/src/content/en/application/deployment-guide.mdx index 9cadef0e48..6ed26dda32 100644 --- a/frontend/src/content/en/application/deployment-guide.mdx +++ b/frontend/src/content/en/application/deployment-guide.mdx @@ -145,6 +145,19 @@ run_events: Keep SQLite deployments to local, single-node, or short-lived evaluation environments. +### Running more than one Gateway replica + +The Helm chart ships `gateway.replicas: 1`. Raising it is supported for the web and API path once every replica shares Postgres and Redis. The chart's default `config` already sets what the Gateway's startup gate checks: + +- `database.backend: postgres`, with the `checkpointer` section pointing at the same database +- `run_events.backend: db`, so message history and delivery receipts are shared +- `run_ownership.heartbeat_enabled: true`, so a starting Pod only reclaims runs whose owner is really gone +- `stream_bridge.type: redis`, so SSE streams and reconnects work from any Pod (sandbox ownership leases are inferred from it) + +The chart exports `DEER_FLOW_MULTI_INSTANCE=true` when `gateway.replicas > 1`, and the Gateway refuses to start if any of these is missing. If you scale with `kubectl scale` or an HPA instead of `gateway.replicas`, set `gateway.multiInstance: true` first so the declaration, the PodDisruptionBudget and the required shared `AUTH_JWT_SECRET` do not keep their single-replica rendering (the PodDisruptionBudget then blocks node drains of the single Pod until the scale-up happens; disable it with `gateway.podDisruptionBudget.enabled: false` for that interim if needed). On multi-node clusters also set `persistence.home.accessMode: ReadWriteMany` (uploads, outputs, memory and `extensions_config.json` live on that volume) and `agent_storage.backend: db` so custom agents are visible on every Pod. + +Still single-instance: IM channels (every Pod would connect to every platform), the WeChat QR login, and browser tools. Keep one replica while those are enabled. + ### K8s Provisioner setup The provisioner manages sandbox Pods in a Kubernetes cluster. It is included in `docker/docker-compose-dev.yaml`. diff --git a/frontend/src/content/zh/application/deployment-guide.mdx b/frontend/src/content/zh/application/deployment-guide.mdx index d939024461..a77b19013e 100644 --- a/frontend/src/content/zh/application/deployment-guide.mdx +++ b/frontend/src/content/zh/application/deployment-guide.mdx @@ -142,6 +142,19 @@ run_events: SQLite 部署应限制在本地、单节点或短期评估场景。 +### 运行多个 Gateway 副本 + +Helm chart 默认 `gateway.replicas: 1`。当所有副本共享同一个 Postgres 与 Redis 时,Web 与 API 路径支持调高副本数。chart 默认的 `config` 已经设置了 Gateway 启动检查所要求的项: + +- `database.backend: postgres`,且 `checkpointer` 段指向同一个数据库 +- `run_events.backend: db`,消息历史与投递回执跨 Pod 共享 +- `run_ownership.heartbeat_enabled: true`,新启动的 Pod 只回收属主确实已死的 run +- `stream_bridge.type: redis`,SSE 流与断线重连在任意 Pod 上都可用(沙箱所有权租约由此推断) + +当 `gateway.replicas > 1` 时,chart 会导出 `DEER_FLOW_MULTI_INSTANCE=true`,缺少上述任一项时 Gateway 会拒绝启动。如果你通过 `kubectl scale` 或 HPA 而不是 `gateway.replicas` 扩容,请先设置 `gateway.multiInstance: true`:它们在不执行 Helm upgrade 的情况下改变副本数,否则该声明、PodDisruptionBudget 和必需的共享 `AUTH_JWT_SECRET` 都会停留在单副本的渲染结果上(在真正扩容之前,PodDisruptionBudget 会阻止驱逐唯一的 Pod 和排空节点;如有需要,可在此期间设置 `gateway.podDisruptionBudget.enabled: false`)。多节点集群还需要把 `persistence.home.accessMode` 设为 `ReadWriteMany`(上传、产物、记忆与 `extensions_config.json` 都在该卷上),并设置 `agent_storage.backend: db` 让自定义 agent 在每个 Pod 上可见。 + +仍然只能单实例运行的部分:IM 通道(每个 Pod 都会连接所有平台)、WeChat 扫码登录、浏览器工具。启用这些功能时请保持 1 个副本。 + ### K8s Provisioner 配置