fix(deploy): wire the multi-replica prerequisites into the Helm chart and compose (#6347)

* fix(deploy): wire the multi-replica prerequisites into the Helm chart and compose

The chart shipped gateway.replicas: 1 with a README citing the long-closed
issue #3948, while its rendered config lacked run_ownership heartbeats and
db run events: raising replicas made every starting Pod write its peers'
lease-less runs off as orphans, and the Gateway's startup gate could not
notice because its worker count only sees one Pod.

Enable heartbeats and db run events in the default config, export
DEER_FLOW_MULTI_INSTANCE from the replica count for the startup gate,
generate and inject a shared AUTH_JWT_SECRET (per-Pod .jwt_secret files
logged users out across replicas), declare a surge-then-drain rollout
strategy, render a PodDisruptionBudget when replicas > 1, and bound
uvicorn's graceful shutdown inside a 90s termination grace period.
docker-compose gets the same uvicorn bound and a 60s stop_grace_period so
the 30s memory flush is no longer SIGKILLed. Docs and comments now list
what actually remains single-instance: IM channels, WeChat QR login and
browser tools.

* fix(deploy): honor existingAppSecret, declare multi-instance explicitly, cover the dev stack's shutdown

Follow-up to the multi-replica prerequisites wiring, addressing review:

- `deer-flow.appSecret` returns `existingAppSecret` when it is set. The
  gateway and frontend kept referencing the generated `<release>-app`
  Secret, which the chart stopped rendering, so every Pod sat in
  CreateContainerConfigError. The value is now declared in values.yaml.
- `gateway.multiInstance` (boolean, default null) overrides the
  replica-derived DEER_FLOW_MULTI_INSTANCE, PodDisruptionBudget and
  AUTH_JWT_SECRET requirement for deployments scaled with kubectl or an
  HPA, which change the replica count without a Helm upgrade.
- AUTH_JWT_SECRET is required, not optional, whenever the gateway is
  multi-instance: concurrently booting Pods otherwise race to write
  different `.jwt_secret` files on the home volume and sign sessions with
  different keys.
- `gateway.uvicornGracefulShutdownSeconds: 0` is honored instead of being
  swallowed by `default 10`.
- docker-compose-dev.yaml and dev-entrypoint.sh get the same uvicorn
  graceful-shutdown bound and `stop_grace_period: 60s`.
- The chart README documents the one-time sign-out when upgrading a
  release that predates AUTH_JWT_SECRET and how to seed the previous key
  into the Secret first.
- Tests read the shutdown budget from the Gateway through a shared helper
  instead of hard-coding 5/5/30, the raw-template string matches give way
  to `helm template` renders, and new renders cover multiInstance,
  existingAppSecret, a disabled PDB and a zero graceful-shutdown bound.

* docs(changelog): link the multi-replica deploy entry to #6347

* fix(deploy): budget every bounded teardown hook and flag the multiInstance PDB window

Review follow-ups on #6347:

- The shutdown budget modeled only channel stop + run drain + memory flush
  (40s). The lifespan also runs the startup trash sweep, the notification
  delivery worker stop, browser-session close and the MCP session-pool
  close, each bounded at _SHUTDOWN_HOOK_TIMEOUT_SECONDS, plus a 1s
  retrieval-warm wait: about 61s worst case. _gateway_shutdown_budget.py
  now reads all of them from the Gateway, a drift guard pins the hook
  count against app.py, and both compose files move stop_grace_period
  from 60s to 90s (Helm's 90s already had room: 5 + 10 + 61). Comments
  and the chart README cite the full arithmetic.
- gateway.multiInstance: true renders the PodDisruptionBudget immediately,
  which blocks drains of the single Pod until the scale-up happens; the
  README, values comment and both deployment guides say so and name
  podDisruptionBudget.enabled: false as the interim escape hatch.

CHANGELOG.md

@@ -678,6 +678,32 @@ This release closes that milestone with **439 merged pull requests**.
   before a trailing `Exit Code: N` or `Command exited with code N`, an output
   that is only `Command exited with code N` is left unchanged, and the result
   keeps `deerflow_tool_meta`, `artifact` and `id`. ([#6307])
+- **deploy:** The Helm chart and docker-compose no longer ship a multi-replica
+  trap. The chart's default `config` enables `run_ownership.heartbeat_enabled`
+  and `run_events.backend: db` (so a starting Pod only reclaims runs whose
+  owner is really gone, and message history is shared across Pods), exports
+  `DEER_FLOW_MULTI_INSTANCE` for the Gateway's startup gate from
+  `gateway.replicas` or an explicit `gateway.multiInstance` (set it before
+  scaling with `kubectl scale` or an HPA, which the chart cannot see),
+  generates and injects a shared `AUTH_JWT_SECRET` (per-Pod `.jwt_secret`
+  files logged users out across replicas; the key is required whenever the
+  gateway is multi-instance so concurrently booting Pods cannot race to write
+  different keys), declares a surge-then-drain rollout strategy, renders a
+  `PodDisruptionBudget` for a multi-instance gateway, and bounds uvicorn's
+  graceful shutdown (`gateway.uvicornGracefulShutdownSeconds`, 10s; `0` is
+  honored) inside a 90s termination grace period so an idle SSE stream cannot
+  push the memory drain into SIGKILL. `existingAppSecret` is now honored by
+  every consumer (the gateway and frontend kept referencing the generated
+  `<release>-app` Secret, which was not rendered, so Pods never started).
+  Both compose files bound the same uvicorn timeout and set
+  `stop_grace_period: 90s` (Docker's 10s default cut the 30s memory flush
+  short; the lifespan's worst case is about 61s once every bounded teardown
+  hook is counted, and the tests read those bounds from the Gateway). The chart README and compose comments stop citing the long-closed
+  issue #3948 as the reason to stay at one replica and list what actually
+  remains single-instance (IM channels, WeChat QR login, browser tools).
+  Upgrading a release that predates `AUTH_JWT_SECRET` generates a new key and
+  signs every browser session out once; the chart README shows how to seed
+  the previous key into the Secret first to keep sessions. ([#6347])
 - **persistence:** A second Gateway instance no longer fails startup with
   `TimeoutError` while another instance runs a PostgreSQL schema migration. The
   bootstrap advisory lock was taken with a blocking `pg_advisory_lock` on the
@@ -8782,3 +8808,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
 [#6338]: https://github.com/bytedance/deer-flow/pull/6338
 [#6343]: https://github.com/bytedance/deer-flow/pull/6343
 [#6344]: https://github.com/bytedance/deer-flow/pull/6344
+[#6347]: https://github.com/bytedance/deer-flow/pull/6347

backend/tests/_gateway_shutdown_budget.py

@@ -0,0 +1,68 @@
+"""Shutdown work every deployment's stop grace period must cover.
+
+The chart's ``terminationGracePeriodSeconds`` and both compose files'
+``stop_grace_period`` must outlast the Gateway's lifespan shutdown. Besides the
+memory queue flush, ``app.gateway.app.lifespan`` runs several *sequential*,
+individually bounded hooks around it, and ``langgraph_runtime`` drains in-flight
+runs on exit. These numbers are read from the Gateway rather than copied into
+the tests, so raising a drain budget or adding a bounded hook here fails the
+grace-period tests instead of silently outgrowing the deployments.
+"""
+
+from __future__ import annotations
+
+from app.gateway.deps import _RUN_DRAIN_TIMEOUT_SECONDS
+from deerflow.config.memory_config import MemoryConfig
+
+#: Teardown steps in ``app.gateway.app.lifespan`` that each wait up to
+#: ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS`` before proceeding. They run one after
+#: another, so a pathological shutdown can spend the full budget on every one.
+#: Keep this list in step with the ``asyncio.wait_for(..., timeout=
+#: _SHUTDOWN_HOOK_TIMEOUT_SECONDS)`` sites in that function.
+HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT: tuple[str, ...] = (
+    "startup trash sweep",
+    "notification delivery worker stop",
+    "channel service stop",
+    "browser session close",
+    "MCP session pool close",
+)
+
+
+def _app_constants() -> tuple[float, float]:
+    """Return ``(_SHUTDOWN_HOOK_TIMEOUT_SECONDS, _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS)``."""
+    # Importing the FastAPI module builds nothing, but it is heavy; keep it lazy.
+    from app.gateway.app import _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS, _SHUTDOWN_HOOK_TIMEOUT_SECONDS
+
+    return float(_SHUTDOWN_HOOK_TIMEOUT_SECONDS), float(_RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS)
+
+
+def bounded_hooks_seconds() -> float:
+    """Worst case for the sequential hooks bounded by ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS``."""
+    hook_timeout, _ = _app_constants()
+    return len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT) * hook_timeout
+
+
+def retrieval_warm_wait_seconds() -> float:
+    """Brief wait for the derived retrieval-index rebuild before the flush."""
+    _, retrieval_warm_timeout = _app_constants()
+    return retrieval_warm_timeout
+
+
+def run_drain_seconds() -> float:
+    """Bound on draining in-flight runs before the checkpointer is torn down."""
+    return float(_RUN_DRAIN_TIMEOUT_SECONDS)
+
+
+def memory_flush_seconds() -> float:
+    """Default ``memory.shutdown_flush_timeout_seconds``."""
+    return float(MemoryConfig.model_fields["shutdown_flush_timeout_seconds"].default)
+
+
+def lifespan_shutdown_seconds() -> float:
+    """Worst-case lifespan shutdown: every bounded hook + retrieval-warm wait + run drain + memory flush.
+
+    Unbounded steps (OIDC close, scheduler / task-service stops, backend close)
+    are expected to be quick and are not modeled; deployments keep slack on top
+    of this figure for them.
+    """
+    return bounded_hooks_seconds() + retrieval_warm_wait_seconds() + run_drain_seconds() + memory_flush_seconds()

backend/tests/test_compose_default_workers.py

@@ -16,9 +16,13 @@ import re
 from pathlib import Path
 
 import yaml
+from _gateway_shutdown_budget import HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT, lifespan_shutdown_seconds
 
 REPO_ROOT = Path(__file__).resolve().parents[2]
 COMPOSE_PATH = REPO_ROOT / "docker" / "docker-compose.yaml"
+DEV_COMPOSE_PATH = REPO_ROOT / "docker" / "docker-compose-dev.yaml"
+DEV_ENTRYPOINT_PATH = REPO_ROOT / "docker" / "dev-entrypoint.sh"
+GATEWAY_APP_PATH = REPO_ROOT / "backend" / "app" / "gateway" / "app.py"
 
 
 def _gateway_command() -> str:
@@ -43,3 +47,46 @@ def test_gateway_worker_count_remains_overridable():
     """The worker count must stay configurable, not hard-coded to 1."""
     command = _gateway_command()
     assert "${GATEWAY_WORKERS:-1}" in command, f"worker count must use ${{GATEWAY_WORKERS:-1}} so operators can override it; got: {command}"
+
+
+def _graceful_shutdown_bound(launch: str) -> int:
+    """Return uvicorn's ``--timeout-graceful-shutdown`` bound from a launch command or script."""
+    match = re.search(r"--timeout-graceful-shutdown (\d+)", launch)
+    assert match is not None, f"uvicorn launch must bound its graceful shutdown; got: {launch}"
+    return int(match.group(1))
+
+
+def _stop_grace_seconds(compose_path: Path) -> int:
+    compose = yaml.safe_load(compose_path.read_text(encoding="utf-8"))
+    grace = compose["services"]["gateway"]["stop_grace_period"]
+    match = re.fullmatch(r"(\d+)s", str(grace))
+    assert match is not None, f"stop_grace_period must be expressed in seconds, got {grace!r}"
+    return int(match.group(1))
+
+
+def test_gateway_bounds_uvicorn_graceful_shutdown():
+    """Open SSE connections must not hold lifespan shutdown (bounded hooks, run drain, memory flush) past the stop grace period."""
+    assert _graceful_shutdown_bound(_gateway_command()) <= 30
+
+
+def test_gateway_stop_grace_period_covers_the_shutdown_work():
+    """Docker's 10s default SIGKILLed the 30s memory queue flush on every restart."""
+    assert _stop_grace_seconds(COMPOSE_PATH) >= _graceful_shutdown_bound(_gateway_command()) + lifespan_shutdown_seconds()
+
+
+def test_dev_gateway_bounds_uvicorn_graceful_shutdown_and_covers_it():
+    """The dev stack launches uvicorn from dev-entrypoint.sh and is restarted far more often."""
+    bound = _graceful_shutdown_bound(DEV_ENTRYPOINT_PATH.read_text(encoding="utf-8"))
+    assert bound <= 30
+    assert _stop_grace_seconds(DEV_COMPOSE_PATH) >= bound + lifespan_shutdown_seconds()
+
+
+def test_shutdown_budget_models_every_bounded_lifespan_hook():
+    """Each ``wait_for(..., timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS)`` in the lifespan is a sequential 5s worst case the grace periods must cover."""
+    source = GATEWAY_APP_PATH.read_text(encoding="utf-8")
+    bounded_sites = source.count("timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS")
+    assert bounded_sites == len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT), (
+        f"app.gateway.app bounds {bounded_sites} teardown hook(s) with _SHUTDOWN_HOOK_TIMEOUT_SECONDS but "
+        f"_gateway_shutdown_budget models {len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT)}; update HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT "
+        "so the chart and compose grace periods keep covering the worst case"
+    )

backend/tests/test_helm_multi_replica_prereqs.py

@@ -0,0 +1,170 @@
+"""Regression tests for the Helm chart's multi-replica prerequisites.
+
+The chart used to ship ``gateway.replicas: 1`` with a README that cited the
+long-closed issue #3948, while its rendered config lacked the two settings a
+second Pod actually needs: run-ownership heartbeats (without a lease, a
+starting Pod writes every peer's live run off as an orphan) and
+database-backed run events. These tests pin the config block, the
+``DEER_FLOW_MULTI_INSTANCE`` declaration the Gateway's startup gate reads
+(derived from ``gateway.replicas`` or declared with ``gateway.multiInstance``
+for deployments scaled by kubectl/HPA), the shared ``AUTH_JWT_SECRET`` that is
+required once more than one instance runs, the ``existingAppSecret`` wiring,
+the rollout strategy, the PodDisruptionBudget, and a termination grace period
+that covers the shutdown work.
+
+The ``helm template`` tests skip when helm is not installed; CI's runner has it.
+"""
+
+from __future__ import annotations
+
+import re
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+import yaml
+from _gateway_shutdown_budget import lifespan_shutdown_seconds
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+CHART = REPO_ROOT / "deploy" / "helm" / "deer-flow"
+VALUES = CHART / "values.yaml"
+APP_SECRET_TEMPLATE = CHART / "templates" / "secret-app.yaml"
+README = CHART / "README.md"
+
+
+def _values() -> dict:
+    return yaml.safe_load(VALUES.read_text(encoding="utf-8"))
+
+
+def _rendered_config() -> dict:
+    return yaml.safe_load(_values()["config"])
+
+
+def _render_chart(*settings: str) -> list[dict]:
+    helm = shutil.which("helm")
+    if helm is None:
+        pytest.skip("helm is unavailable")
+    command = [helm, "template", "deer-flow", str(CHART)]
+    for setting in settings:
+        command.extend(["--set", setting])
+    rendered = subprocess.run(command, check=True, capture_output=True, text=True).stdout
+    return [document for document in yaml.safe_load_all(rendered) if isinstance(document, dict)]
+
+
+def _by_kind(documents: list[dict], kind: str, name_suffix: str) -> dict:
+    return next(document for document in documents if document.get("kind") == kind and document["metadata"]["name"].endswith(name_suffix))
+
+
+def _has_kind(documents: list[dict], kind: str) -> bool:
+    return any(document.get("kind") == kind for document in documents)
+
+
+def _container(documents: list[dict], deployment_suffix: str, container_name: str) -> dict:
+    containers = _by_kind(documents, "Deployment", deployment_suffix)["spec"]["template"]["spec"]["containers"]
+    return next(container for container in containers if container["name"] == container_name)
+
+
+def _env(container: dict) -> dict[str, dict]:
+    return {item["name"]: item for item in container["env"]}
+
+
+def _secret_ref(container: dict, env_name: str) -> dict:
+    return _env(container)[env_name]["valueFrom"]["secretKeyRef"]
+
+
+def test_default_config_enables_the_multi_replica_prerequisites() -> None:
+    config = _rendered_config()
+    assert config["database"]["backend"] == "postgres"
+    assert config["checkpointer"]["type"] == "postgres"
+    assert config["stream_bridge"]["type"] == "redis"
+    assert config["run_ownership"]["heartbeat_enabled"] is True, "without a lease every peer run is reclaimed as an orphan on Pod start"
+    assert config["run_events"]["backend"] == "db", "memory run events are process-local"
+
+
+def test_default_replica_count_stays_conservative_with_a_rollout_budget() -> None:
+    values = _values()
+    gateway = values["gateway"]
+    assert gateway["replicas"] == 1
+    assert gateway["multiInstance"] is None, "derived from replicas unless the operator declares it"
+    assert gateway["strategy"]["type"] == "RollingUpdate"
+    assert gateway["strategy"]["rollingUpdate"]["maxUnavailable"] == 0
+    assert gateway["strategy"]["rollingUpdate"]["maxSurge"] == 1
+    assert gateway["podDisruptionBudget"]["enabled"] is True
+    assert gateway["podDisruptionBudget"]["minAvailable"] == 1
+    assert values["existingAppSecret"] == ""
+
+
+def test_termination_grace_period_covers_the_shutdown_work() -> None:
+    gateway = _values()["gateway"]
+    required = gateway["preStopSleepSeconds"] + gateway["uvicornGracefulShutdownSeconds"] + lifespan_shutdown_seconds()
+    assert gateway["terminationGracePeriodSeconds"] > required, f"grace period must exceed {required}s of shutdown work"
+
+
+def test_app_secret_template_preserves_the_jwt_secret_across_upgrades() -> None:
+    """``helm template`` never sees ``lookup`` results, so pin the preservation in the source."""
+    template = APP_SECRET_TEMPLATE.read_text(encoding="utf-8")
+    assert 'index $prev.data "AUTH_JWT_SECRET"' in template, "must survive upgrades like the other app secrets"
+    assert "AUTH_JWT_SECRET: {{ $jwtSecret | quote }}" in template
+
+
+def test_readme_documents_the_multi_instance_knobs() -> None:
+    readme = README.read_text(encoding="utf-8")
+    assert re.search(r"do not raise\s+`gateway\.replicas` past 1", readme) is None
+    for needle in ("DEER_FLOW_MULTI_INSTANCE", "AUTH_JWT_SECRET", "gateway.multiInstance", "existingAppSecret", ".jwt_secret"):
+        assert needle in readme, needle
+
+
+@pytest.mark.parametrize("replicas", [1, 2, 3])
+def test_rendered_declaration_follows_the_replica_count(replicas: int) -> None:
+    values = _values()["gateway"]
+    documents = _render_chart(f"gateway.replicas={replicas}")
+    gateway = _container(documents, "-gateway", "gateway")
+    multi_instance = replicas > 1
+
+    assert _env(gateway)["DEER_FLOW_MULTI_INSTANCE"]["value"] == str(multi_instance).lower()
+    jwt = _secret_ref(gateway, "AUTH_JWT_SECRET")
+    assert jwt["name"].endswith("-app")
+    assert jwt["key"] == "AUTH_JWT_SECRET"
+    assert jwt["optional"] is (not multi_instance), "the shared key is required once Pods could race to write .jwt_secret"
+    assert f"--timeout-graceful-shutdown {values['uvicornGracefulShutdownSeconds']}" in " ".join(gateway["args"])
+
+    assert _has_kind(documents, "PodDisruptionBudget") is multi_instance
+    if multi_instance:
+        assert _by_kind(documents, "PodDisruptionBudget", "-gateway")["spec"]["minAvailable"] == values["podDisruptionBudget"]["minAvailable"]
+
+    deployment = _by_kind(documents, "Deployment", "-gateway")
+    assert deployment["spec"]["strategy"] == values["strategy"]
+    assert deployment["spec"]["template"]["spec"]["terminationGracePeriodSeconds"] == values["terminationGracePeriodSeconds"]
+
+
+def test_explicit_multi_instance_declaration_wins_over_the_replica_count() -> None:
+    """kubectl scale and HPAs change the count without a Helm upgrade, so the operator declares it."""
+    documents = _render_chart("gateway.replicas=1", "gateway.multiInstance=true")
+    gateway = _container(documents, "-gateway", "gateway")
+    assert _env(gateway)["DEER_FLOW_MULTI_INSTANCE"]["value"] == "true"
+    assert _secret_ref(gateway, "AUTH_JWT_SECRET")["optional"] is False
+    assert _has_kind(documents, "PodDisruptionBudget")
+
+
+def test_pdb_can_be_disabled_and_a_zero_graceful_shutdown_is_honored() -> None:
+    documents = _render_chart("gateway.replicas=2", "gateway.podDisruptionBudget.enabled=false", "gateway.uvicornGracefulShutdownSeconds=0")
+    assert not _has_kind(documents, "PodDisruptionBudget")
+    assert "--timeout-graceful-shutdown 0" in " ".join(_container(documents, "-gateway", "gateway")["args"])
+
+
+def test_existing_app_secret_is_honored_by_every_consumer() -> None:
+    """The helper used to ignore the value, so Pods referenced a Secret the chart no longer rendered."""
+    documents = _render_chart("existingAppSecret=my-app-secret", "gateway.replicas=2")
+    assert not any(document.get("kind") == "Secret" and document["metadata"]["name"].endswith("-app") for document in documents)
+    gateway = _container(documents, "-gateway", "gateway")
+    assert _secret_ref(gateway, "DEER_FLOW_INTERNAL_AUTH_TOKEN")["name"] == "my-app-secret"
+    assert _secret_ref(gateway, "AUTH_JWT_SECRET")["name"] == "my-app-secret"
+    frontend = _container(documents, "-frontend", "frontend")
+    assert _secret_ref(frontend, "BETTER_AUTH_SECRET")["name"] == "my-app-secret"
+
+
+def test_rendered_app_secret_carries_the_jwt_secret() -> None:
+    documents = _render_chart()
+    secret = _by_kind(documents, "Secret", "-app")
+    assert secret["stringData"]["AUTH_JWT_SECRET"]

deploy/helm/deer-flow/README.md

@@ -267,17 +267,57 @@ kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002
   `%40`). The chart uses an external `databaseUrl` verbatim and does not
   rewrite the DSN in a user-managed Secret.
 
-- **Graceful shutdown & memory drain.** The gateway pod sets `terminationGracePeriodSeconds` (default 45s, overridable via `gateway.terminationGracePeriodSeconds`) plus an optional `preStop` sleep (`gateway.preStopSleepSeconds`, default 5s). The grace period MUST exceed the Gateway's graceful-shutdown work — channel stop (~5s) plus the memory-queue drain (`memory.shutdown_flush_timeout_seconds`, default 30s) plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. **When you raise `memory.shutdown_flush_timeout_seconds`, raise `gateway.terminationGracePeriodSeconds` to match** (channel stop + drain + buffer).
-- **Gateway replicas.** Postgres + the Redis stream bridge together make the
-  gateway's *persisted* state (checkpointer + run/thread metadata) and *live
-  stream* path cross-pod-safe. The default is still 1 replica: **do not raise
-  `gateway.replicas` past 1 yet.** Run control — `create_or_reject` dedup,
-  `cancel`, and orphan reconciliation — is still worker-local (in-process
-  `asyncio.Lock` + in-memory `record.task`), tracked by [issue
-  #3948](https://github.com/bytedance/deer-flow/issues/3948). With >1 replica a
-  double-submit can create two runs on one thread (checkpoint corruption), a
-  cancel can land on a non-owner pod (409), and a crashed pod's runs stay
-  `pending`/`running` forever. Stay on 1 replica until that work lands.
+- **Graceful shutdown & memory drain.** The gateway pod sets `terminationGracePeriodSeconds` (default 90s, overridable via `gateway.terminationGracePeriodSeconds`) plus an optional `preStop` sleep (`gateway.preStopSleepSeconds`, default 5s), and bounds uvicorn's `--timeout-graceful-shutdown` (`gateway.uvicornGracefulShutdownSeconds`, default 10s) so an idle SSE connection cannot hold up lifespan shutdown indefinitely. The grace period MUST exceed the Gateway's graceful-shutdown work — the preStop sleep, the uvicorn timeout, and the lifespan's worst case: five hooks bounded at 5s each (startup trash sweep, notification delivery worker, channel service, browser sessions, MCP session pool), the 1s retrieval-warm wait, the in-flight run drain (5s) and the memory-queue drain (`memory.shutdown_flush_timeout_seconds`, default 30s), about 61s in total, plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. `backend/tests/_gateway_shutdown_budget.py` reads these bounds from the Gateway and pins the chart and compose budgets against them. **When you raise `memory.shutdown_flush_timeout_seconds`, raise `gateway.terminationGracePeriodSeconds` to match** (preStop + uvicorn timeout + ~31s of bounded hooks and drains + memory drain + buffer).
+- **Gateway replicas.** Run control is cross-pod-safe since the work tracked
+  by [issue #3948](https://github.com/bytedance/deer-flow/issues/3948) landed
+  (#4003, #4064, #4500): admission is a durable one-active-run-per-thread
+  constraint with `Idempotency-Key` reuse, a cancel that lands on a non-owner
+  Pod is recorded durably and executed by the owner on its next lease renewal,
+  and a crashed Pod's runs are reclaimed by a peer once their lease expires.
+  The chart's default `config` enables what that needs — Postgres for
+  `database` and `checkpointer`, `run_events.backend: db`,
+  `run_ownership.heartbeat_enabled: true`, and the Redis stream bridge
+  (sandbox ownership leases are inferred from it) — and exports
+  `DEER_FLOW_MULTI_INSTANCE=true` whenever `gateway.replicas > 1`, so the
+  Gateway's startup gate verifies those prerequisites instead of staying inert
+  (its worker count only sees one Pod). If you scale with `kubectl scale` or
+  an HPA instead of `gateway.replicas`, set `gateway.multiInstance: true`
+  first: those change the replica count without a Helm upgrade, so the
+  derived declaration, the PodDisruptionBudget and the required shared
+  `AUTH_JWT_SECRET` would otherwise keep their single-replica rendering.
+  Mind the window between setting it and the actual scale-up: the
+  PodDisruptionBudget (`minAvailable: 1`) is rendered immediately and blocks
+  every voluntary eviction and node drain while only one Pod exists, so set
+  `gateway.podDisruptionBudget.enabled: false` for that interim if you need
+  to drain nodes first. `gateway.replicas: 1` stays the
+  default because the following are still single-instance: **IM channels**
+  (every Pod would connect to every platform — Telegram polling conflicts and
+  Discord double-processes; keep 1 replica while channels are enabled), the
+  WeChat QR login, and browser tools. Before raising replicas, also set
+  `persistence.home.accessMode: ReadWriteMany` on multi-node clusters (thread
+  uploads, outputs, memory and `extensions_config.json` live on that volume)
+  and `agent_storage.backend: db` so custom agents are visible on every Pod.
+  A `PodDisruptionBudget` (`minAvailable: 1`) is rendered automatically for a
+  multi-instance gateway (same rule), and the rollout strategy is
+  surge-then-drain (`maxSurge: 1`, `maxUnavailable: 0`).
+- **App secret.** `<release>-app` holds `BETTER_AUTH_SECRET`,
+  `DEER_FLOW_INTERNAL_AUTH_TOKEN` and `AUTH_JWT_SECRET` (the session-cookie
+  signing key), each generated once and preserved across upgrades via
+  `lookup`. `existingAppSecret` points the gateway and frontend at a Secret
+  you manage instead (no `<release>-app` is generated); it must carry all
+  three keys. `AUTH_JWT_SECRET` is required whenever the gateway is
+  multi-instance — without it, concurrently booting Pods race to write their
+  own `.jwt_secret` on the home volume and sign sessions with different keys —
+  and only a single Pod may omit it and fall back to that file. **Upgrading a
+  release that predates `AUTH_JWT_SECRET`** generates a new key, so every
+  existing browser session is signed out once. To keep sessions, copy the key
+  the gateway has been using into the Secret before upgrading; `lookup` then
+  preserves it like the other app secrets:
+
+  ```bash
+  JWT=$(kubectl -n deer-flow exec deploy/deer-flow-gateway -- cat /app/backend/.deer-flow/.jwt_secret)
+  kubectl -n deer-flow patch secret deer-flow-app -p "{\"stringData\":{\"AUTH_JWT_SECRET\":\"$JWT\"}}"
+  ```
 - **Scheduled task recovery.** If a deployment explicitly enables
   `scheduler.multi_instance: true`, it must use shared Postgres,
   `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`.
@@ -285,8 +325,8 @@ kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002
   atomically takes over only expired leases, and fences stale post-launch
   bookkeeping. `max_concurrent_runs` is a shared global cap across Pods,
   including pre-launch dispatch reservations. Restart all Gateway Pods after
-  changing these startup-only settings. This does not remove the broader
-  Gateway replica limitations described above.
+  changing these startup-only settings. IM channels remain single-instance
+  regardless (see Gateway replicas above).
 - **Redis stream bridge.** A bundled single-instance redis StatefulSet
   (`redis.enabled: true`, `redis:7-alpine`) runs in the namespace and the
   gateway connects via the in-cluster Service. Per-run SSE events are stored in

deploy/helm/deer-flow/templates/NOTES.txt

@@ -69,7 +69,8 @@ Provisioner / sandbox notes:
 
 Generated secrets (persisted across upgrades):
 
-  • BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN are stored in the
+  • BETTER_AUTH_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and AUTH_JWT_SECRET (the
+    session-cookie signing key shared by every gateway Pod) are stored in the
     Secret `{{ include "deer-flow.appSecret" . }}`.
 
 Database ({{ if .Values.postgresql.enabled }}bundled postgres{{ else }}external postgres{{ end }}):

deploy/helm/deer-flow/templates/_helpers.tpl

@@ -70,9 +70,26 @@ imagePullSecrets:
 {{- else -}}{{- printf "%s-provider" (include "deer-flow.fullname" .) -}}{{- end -}}
 {{- end -}}
 
-{{/* Name of the Secret holding generated app secrets (auth token, better-auth). */}}
+{{/* Name of the Secret holding the app secrets (BETTER_AUTH_SECRET,
+     DEER_FLOW_INTERNAL_AUTH_TOKEN, AUTH_JWT_SECRET). `existingAppSecret` points
+     every consumer (gateway, frontend) at a user-managed Secret instead and
+     skips generating one; it must carry all three keys for more than one
+     gateway instance. */}}
 {{- define "deer-flow.appSecret" -}}
-{{- printf "%s-app" (include "deer-flow.fullname" .) -}}
+{{- if .Values.existingAppSecret -}}{{- .Values.existingAppSecret -}}
+{{- else -}}{{- printf "%s-app" (include "deer-flow.fullname" .) -}}{{- end -}}
+{{- end -}}
+
+{{/* "true" when more than one gateway instance shares the database. An explicit
+     boolean `gateway.multiInstance` wins -- set it before scaling with
+     `kubectl scale` or an HPA, which change the replica count without a Helm
+     upgrade -- otherwise it is derived from `gateway.replicas`. */}}
+{{- define "deer-flow.gatewayMultiInstance" -}}
+{{- if kindIs "invalid" .Values.gateway.multiInstance -}}
+{{- gt (int .Values.gateway.replicas) 1 -}}
+{{- else -}}
+{{- eq (toString .Values.gateway.multiInstance | lower) "true" -}}
+{{- end -}}
 {{- end -}}
 
 {{/* Name of the postgres StatefulSet/Service. */}}

deploy/helm/deer-flow/templates/gateway-deployment.yaml

@@ -8,6 +8,10 @@ metadata:
     app.kubernetes.io/component: gateway
 spec:
   replicas: {{ .Values.gateway.replicas }}
+  {{- with .Values.gateway.strategy }}
+  strategy:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   selector:
     matchLabels:
       {{- include "deer-flow.selectorLabels" . | nindent 6 }}
@@ -24,10 +28,14 @@ spec:
         checksum/extensions: {{ include "deer-flow.extensionsChecksum" . }}
     spec:
       {{- include "deer-flow.imagePullSecrets" . | nindent 6 }}
-      # MUST exceed channel stop (~5s) + memory shutdown_flush_timeout_seconds
-      # (default 30s) + buffer, or K8s SIGKILLs the memory drain mid-flight
-      # (values.yaml gateway.terminationGracePeriodSeconds). K8s defaults to 30s.
-      terminationGracePeriodSeconds: {{ .Values.gateway.terminationGracePeriodSeconds | default 45 }}
+      # MUST exceed preStop sleep + uvicorn --timeout-graceful-shutdown + the
+      # lifespan's worst case -- five hooks bounded at 5s each (trash sweep,
+      # notification worker, channel service, browser sessions, MCP session
+      # pool), the 1s retrieval-warm wait, the 5s run drain and the memory
+      # shutdown_flush_timeout_seconds (default 30s): ~61s -- plus buffer, or
+      # K8s SIGKILLs the memory drain mid-flight (values.yaml
+      # gateway.terminationGracePeriodSeconds). K8s defaults to 30s.
+      terminationGracePeriodSeconds: {{ .Values.gateway.terminationGracePeriodSeconds | default 90 }}
       securityContext:
         # Non-root: the gateway image has no USER directive and would otherwise
         # run as root. uid/gid 1000; fsGroup 1000 so the .deer-flow PVC is
@@ -83,7 +91,7 @@ spec:
           workingDir: /app
           command: ["sh", "-c"]
           args:
-            - cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers 1
+            - cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers 1 --timeout-graceful-shutdown {{ .Values.gateway.uvicornGracefulShutdownSeconds | int }}
           env:
             - name: CI
               value: "true"
@@ -118,6 +126,26 @@ spec:
                 secretKeyRef:
                   name: {{ include "deer-flow.appSecret" . }}
                   key: DEER_FLOW_INTERNAL_AUTH_TOKEN
+            # Session-cookie signing key shared by every Pod. Without it each
+            # Pod runs _load_or_create_secret(), which read-then-writes
+            # .jwt_secret on the home volume: concurrently booting replicas race
+            # and sign sessions with different keys. The key is therefore
+            # required whenever more than one instance runs; only a single Pod
+            # with a user-managed existingAppSecret that lacks it may fall back
+            # to the file.
+            - name: AUTH_JWT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "deer-flow.appSecret" . }}
+                  key: AUTH_JWT_SECRET
+                  optional: {{ ne (include "deer-flow.gatewayMultiInstance" .) "true" }}
+            # Tells the Gateway's startup gate that peers share the database;
+            # its GATEWAY_WORKERS count only sees this Pod. Derived from
+            # gateway.replicas unless gateway.multiInstance is set explicitly:
+            # kubectl scale and HPAs change the replica count without a Helm
+            # upgrade, so set multiInstance: true before scaling that way.
+            - name: DEER_FLOW_MULTI_INSTANCE
+              value: {{ include "deer-flow.gatewayMultiInstance" . | quote }}
             {{- /* PostgreSQL DSN — only mounted when postgres is configured
                   (bundled StatefulSet, external databaseUrl, or an existing
                   Secret). In sqlite/memory mode the env is omitted and

deploy/helm/deer-flow/templates/gateway-pdb.yaml

@@ -0,0 +1,24 @@
+{{- /*
+  PodDisruptionBudget for the gateway. Rendered only for a multi-instance
+  gateway (gateway.replicas > 1, or gateway.multiInstance: true for
+  deployments scaled with kubectl/HPA): with a single replica a budget of 1
+  would block every node drain, and with several it keeps at least one Pod
+  serving (and renewing its run leases) while nodes are drained.
+*/ -}}
+{{- $pdb := .Values.gateway.podDisruptionBudget | default dict -}}
+{{- if and $pdb.enabled (eq (include "deer-flow.gatewayMultiInstance" .) "true") }}
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ include "deer-flow.fullname" . }}-gateway
+  namespace: {{ include "deer-flow.namespace" . }}
+  labels:
+    {{- include "deer-flow.labels" . | nindent 4 }}
+    app.kubernetes.io/component: gateway
+spec:
+  minAvailable: {{ $pdb.minAvailable | default 1 | int }}
+  selector:
+    matchLabels:
+      {{- include "deer-flow.selectorLabels" . | nindent 6 }}
+      app.kubernetes.io/component: gateway
+{{- end }}

deploy/helm/deer-flow/templates/secret-app.yaml

@@ -2,12 +2,18 @@
 {{- $prev := lookup "v1" "Secret" (include "deer-flow.namespace" .) (include "deer-flow.appSecret" .) -}}
 {{- $betterAuth := "" -}}
 {{- $internalToken := "" -}}
+{{- $jwtSecret := "" -}}
 {{- if $prev -}}
 {{- $betterAuth = index $prev.data "BETTER_AUTH_SECRET" | default "" | b64dec -}}
 {{- $internalToken = index $prev.data "DEER_FLOW_INTERNAL_AUTH_TOKEN" | default "" | b64dec -}}
+{{- $jwtSecret = index $prev.data "AUTH_JWT_SECRET" | default "" | b64dec -}}
 {{- end -}}
 {{- if not $betterAuth -}}{{- $betterAuth = randAlphaNum 48 | lower -}}{{- end -}}
 {{- if not $internalToken -}}{{- $internalToken = randAlphaNum 40 -}}{{- end -}}
+{{- /* Session-cookie signing key. Generated once and preserved across upgrades
+       so sessions survive, and shared by every gateway Pod so a session signed
+       on one replica validates on another. */ -}}
+{{- if not $jwtSecret -}}{{- $jwtSecret = randAlphaNum 48 -}}{{- end -}}
 apiVersion: v1
 kind: Secret
 metadata:
@@ -19,4 +25,5 @@ type: Opaque
 stringData:
   BETTER_AUTH_SECRET: {{ $betterAuth | quote }}
   DEER_FLOW_INTERNAL_AUTH_TOKEN: {{ $internalToken | quote }}
+  AUTH_JWT_SECRET: {{ $jwtSecret | quote }}
 {{- end -}}

deploy/helm/deer-flow/values.yaml

@@ -25,18 +25,63 @@ image:
 
 # -- Gateway (backend) deployment.
 gateway:
-  replicas: 1               # Safe default. Postgres + the Redis stream bridge are
-                            # wired, but multi-replica needs issue #3948's run-control
-                            # work (cancel/dedup/reconcile). Scheduler multi_instance
-                             # recovery is safe only when its config prerequisites are met;
-                             # max_concurrent_runs is shared across Pods in that mode.
+  # Replica count. 1 is the conservative default; >1 is supported for the web
+  # and API path once the prerequisites below hold, all of which this chart
+  # sets in `config` by default: Postgres (`database` + `checkpointer`),
+  # `run_events.backend: db`, `run_ownership.heartbeat_enabled: true` and the
+  # Redis stream bridge (sandbox ownership leases are inferred from it). The
+  # chart also exports DEER_FLOW_MULTI_INSTANCE=true when replicas > 1 so the
+  # Gateway's startup gate enforces those prerequisites instead of staying
+  # inert (its worker count only sees one Pod). Still single-instance: IM
+  # channels (every Pod connects to every platform -- keep replicas at 1
+  # while channels are enabled), the WeChat QR login, and browser tools.
+  # Multi-node clusters also need `persistence.home.accessMode: ReadWriteMany`,
+  # and `agent_storage.backend: db` keeps custom agents visible on every Pod.
+  # Scheduler multi_instance recovery is safe only when its config
+  # prerequisites are met; max_concurrent_runs is shared across Pods then.
+  replicas: 1
+  # Explicit multi-instance declaration (boolean). Leave unset (null) to derive
+  # it from `replicas` above. Set `true` before scaling the gateway with
+  # `kubectl scale` or an HPA: those change the replica count without a Helm
+  # upgrade, so the derived DEER_FLOW_MULTI_INSTANCE, the PodDisruptionBudget
+  # and the required shared AUTH_JWT_SECRET would otherwise keep their
+  # single-replica rendering. Note the PodDisruptionBudget (minAvailable: 1)
+  # is rendered as soon as this is true: until the scale-up actually happens
+  # it blocks voluntary evictions and node drains of the single Pod. Set
+  # podDisruptionBudget.enabled: false for that interim if you need to drain.
+  multiInstance: null
+  # Rollout strategy. Surge-then-drain keeps one Pod serving while a peer
+  # starts; with run-ownership heartbeats the new Pod leaves the old Pod's live
+  # runs alone (their leases keep renewing) while the old Pod drains them.
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+      maxUnavailable: 0
+  # PodDisruptionBudget, rendered only for a multi-instance gateway (replicas
+  # > 1 or multiInstance: true; a budget of 1 on a single replica would block
+  # node drains). minAvailable falls back to 1 when unset; disable the budget
+  # rather than setting it to 0.
+  podDisruptionBudget:
+    enabled: true
+    minAvailable: 1
+  # uvicorn --timeout-graceful-shutdown (seconds): how long open HTTP
+  # connections (long SSE streams included) may hold up lifespan shutdown
+  # after SIGTERM before uvicorn closes them. Without a bound one idle SSE
+  # client can push the whole shutdown past the grace period into SIGKILL.
+  # 0 closes open connections immediately on SIGTERM.
+  uvicornGracefulShutdownSeconds: 10
   # Pod termination grace period (seconds). MUST exceed the Gateway's graceful
-  # shutdown work: channel stop (~5s) + memory queue drain
-  # (memory.shutdown_flush_timeout_seconds, default 30s) + buffer. K8s defaults
+  # shutdown work: preStopSleepSeconds + uvicornGracefulShutdownSeconds + the
+  # lifespan's worst case (five 5s-bounded hooks: trash sweep, notification
+  # worker, channel service, browser sessions, MCP session pool; the 1s
+  # retrieval-warm wait; the 5s run drain; the memory queue drain,
+  # memory.shutdown_flush_timeout_seconds, default 30s -- about 61s) + buffer.
+  # backend/tests/_gateway_shutdown_budget.py pins the arithmetic. K8s defaults
   # to 30s, which SIGKILLs the memory drain mid-flight and silently re-introduces
   # the memory loss that shutdown_flush is fixing. Raise this when you raise the
   # memory timeout. See gateway-deployment.yaml + README "Graceful shutdown".
-  terminationGracePeriodSeconds: 45
+  terminationGracePeriodSeconds: 90
   # preStop hook sleep (seconds) before SIGTERM, so the Service/ingress
   # deregisters the pod and in-flight HTTP requests drain before graceful
   # shutdown begins (which runs the memory queue drain). 0 disables the hook.
@@ -242,6 +287,11 @@ skills:
 secrets: {}
 # -- Use an existing Secret instead of creating one from `secrets` above.
 existingSecret: ""
+# -- Use an existing Secret for the app secrets instead of generating
+#    `<release>-app`. It must hold BETTER_AUTH_SECRET (frontend),
+#    DEER_FLOW_INTERNAL_AUTH_TOKEN and, for more than one gateway instance,
+#    AUTH_JWT_SECRET (the session-cookie signing key every Pod must share).
+existingAppSecret: ""
 
 # -- Ingress in front of nginx (port 2026). nginx preserves all internal routing.
 ingress:
@@ -319,6 +369,25 @@ config: |
   stream_bridge:
     type: redis
 
+  # Multi-replica prerequisites (harmless at 1 replica). Heartbeats give each
+  # run a lease, so a starting Pod only reclaims runs whose owner is really
+  # gone instead of writing every peer's live run off as an orphan; db run
+  # events share message history and delivery receipts across Pods. Sandbox
+  # ownership leases (#4206) are inferred from the Redis stream bridge above.
+  # Both are startup-only: restart every Gateway Pod together after changing
+  # them.
+  run_ownership:
+    heartbeat_enabled: true
+  run_events:
+    backend: db
+
+  # Custom agents and managed subagents default to per-Pod files under the
+  # home volume. Switch to the shared database when running more than one
+  # replica (import existing files once with
+  # `python backend/scripts/migrate_agents_to_db.py`):
+  # agent_storage:
+  #   backend: db
+
   memory:
     storage_path: memory.json
 

docker/dev-entrypoint.sh

@@ -162,8 +162,13 @@ fi
 
 # ── Hand off to uvicorn ─────────────────────────────────────────────────────
 
+# --timeout-graceful-shutdown bounds how long open SSE connections may delay
+# lifespan shutdown (memory flush, run drain) after SIGTERM; the compose
+# service's stop_grace_period (90s) covers this bound plus the lifespan's
+# worst-case drain work (~61s).
 PYTHONPATH=. exec uv run --no-sync uvicorn app.gateway.app:app \
     --host 0.0.0.0 --port 8001 \
+    --timeout-graceful-shutdown 10 \
     --reload \
     --reload-include='*.yaml' \
     --reload-include='.env' \

docker/docker-compose-dev.yaml

@@ -194,6 +194,12 @@ services:
     # this file readable and lets the script be linted (shellcheck-clean).
     # See PR #2767 / Issue #2754.
     command: ["sh", "/usr/local/bin/dev-entrypoint.sh"]
+    # Docker's 10s default SIGKILLed the 30s memory queue flush
+    # (memory.shutdown_flush_timeout_seconds) on every `make docker-restart`;
+    # dev-entrypoint.sh bounds uvicorn's graceful shutdown to 10s inside this.
+    # Must cover that bound plus the lifespan's worst case (~61s, see the
+    # gateway comment in docker-compose.yaml) with slack.
+    stop_grace_period: 90s
     volumes:
       # Mount the dev entrypoint as a read-only file so edits to the script
       # take effect on `make docker-restart` without requiring an image rebuild.

docker/docker-compose.yaml

@@ -109,13 +109,24 @@ services:
         NPM_REGISTRY: ${NPM_REGISTRY:-}
         LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65}
     container_name: deer-flow-gateway
-    # Gateway hosts the agent runtime with in-process RunManager + StreamBridge
-    # singletons -- run state lives in this worker's memory. Default to a single
-    # worker: the Redis stream bridge solves cross-worker SSE delivery and
-    # reconnect, but run cancel, request dedup, and per-worker IM channel
-    # services remain worker-local. Override GATEWAY_WORKERS only when the
-    # Redis stream bridge is enabled and those limitations are acceptable.
-    command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1}"
+    # Gateway hosts the agent runtime (RunManager + StreamBridge) in process.
+    # Default to a single worker: GATEWAY_WORKERS > 1 is supported only with
+    # Postgres (database.backend: postgres, run_events.backend: db,
+    # run_ownership.heartbeat_enabled: true) and the Redis stream bridge wired
+    # below -- the Gateway refuses to start otherwise. IM channels, the WeChat
+    # QR login and browser tools stay worker-local, so keep one worker while
+    # they are enabled. --timeout-graceful-shutdown bounds how long open SSE
+    # connections may delay lifespan shutdown after SIGTERM; stop_grace_period
+    # below must cover that bound plus the lifespan's worst case: five hooks
+    # bounded at 5s each (trash sweep, notification worker, channel service,
+    # browser sessions, MCP session pool), the 1s retrieval-warm wait, the 5s
+    # run drain and the 30s memory flush (memory.shutdown_flush_timeout_seconds)
+    # -- about 61s -- plus slack. backend/tests/_gateway_shutdown_budget.py
+    # pins the arithmetic.
+    command: sh -c "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --workers ${GATEWAY_WORKERS:-1} --timeout-graceful-shutdown 10"
+    # Docker's 10s default SIGKILLed the 30s memory queue flush
+    # (memory.shutdown_flush_timeout_seconds) on every restart.
+    stop_grace_period: 90s
     volumes:
       - ${DEER_FLOW_CONFIG_PATH}:/app/backend/config.yaml:ro
       # Writable on purpose: the Gateway edits this file at runtime (MCP

frontend/src/content/en/application/deployment-guide.mdx

@@ -145,6 +145,19 @@ run_events:
 
 Keep SQLite deployments to local, single-node, or short-lived evaluation environments.
 
+### Running more than one Gateway replica
+
+The Helm chart ships `gateway.replicas: 1`. Raising it is supported for the web and API path once every replica shares Postgres and Redis. The chart's default `config` already sets what the Gateway's startup gate checks:
+
+- `database.backend: postgres`, with the `checkpointer` section pointing at the same database
+- `run_events.backend: db`, so message history and delivery receipts are shared
+- `run_ownership.heartbeat_enabled: true`, so a starting Pod only reclaims runs whose owner is really gone
+- `stream_bridge.type: redis`, so SSE streams and reconnects work from any Pod (sandbox ownership leases are inferred from it)
+
+The chart exports `DEER_FLOW_MULTI_INSTANCE=true` when `gateway.replicas > 1`, and the Gateway refuses to start if any of these is missing. If you scale with `kubectl scale` or an HPA instead of `gateway.replicas`, set `gateway.multiInstance: true` first so the declaration, the PodDisruptionBudget and the required shared `AUTH_JWT_SECRET` do not keep their single-replica rendering (the PodDisruptionBudget then blocks node drains of the single Pod until the scale-up happens; disable it with `gateway.podDisruptionBudget.enabled: false` for that interim if needed). On multi-node clusters also set `persistence.home.accessMode: ReadWriteMany` (uploads, outputs, memory and `extensions_config.json` live on that volume) and `agent_storage.backend: db` so custom agents are visible on every Pod.
+
+Still single-instance: IM channels (every Pod would connect to every platform), the WeChat QR login, and browser tools. Keep one replica while those are enabled.
+
 ### K8s Provisioner setup
 
 The provisioner manages sandbox Pods in a Kubernetes cluster. It is included in `docker/docker-compose-dev.yaml`.

frontend/src/content/zh/application/deployment-guide.mdx

@@ -142,6 +142,19 @@ run_events:
 
 SQLite 部署应限制在本地、单节点或短期评估场景。
 
+### 运行多个 Gateway 副本
+
+Helm chart 默认 `gateway.replicas: 1`。当所有副本共享同一个 Postgres 与 Redis 时，Web 与 API 路径支持调高副本数。chart 默认的 `config` 已经设置了 Gateway 启动检查所要求的项：
+
+- `database.backend: postgres`，且 `checkpointer` 段指向同一个数据库
+- `run_events.backend: db`，消息历史与投递回执跨 Pod 共享
+- `run_ownership.heartbeat_enabled: true`，新启动的 Pod 只回收属主确实已死的 run
+- `stream_bridge.type: redis`，SSE 流与断线重连在任意 Pod 上都可用（沙箱所有权租约由此推断）
+
+当 `gateway.replicas > 1` 时，chart 会导出 `DEER_FLOW_MULTI_INSTANCE=true`，缺少上述任一项时 Gateway 会拒绝启动。如果你通过 `kubectl scale` 或 HPA 而不是 `gateway.replicas` 扩容，请先设置 `gateway.multiInstance: true`：它们在不执行 Helm upgrade 的情况下改变副本数，否则该声明、PodDisruptionBudget 和必需的共享 `AUTH_JWT_SECRET` 都会停留在单副本的渲染结果上（在真正扩容之前，PodDisruptionBudget 会阻止驱逐唯一的 Pod 和排空节点；如有需要，可在此期间设置 `gateway.podDisruptionBudget.enabled: false`）。多节点集群还需要把 `persistence.home.accessMode` 设为 `ReadWriteMany`（上传、产物、记忆与 `extensions_config.json` 都在该卷上），并设置 `agent_storage.backend: db` 让自定义 agent 在每个 Pod 上可见。
+
+仍然只能单实例运行的部分：IM 通道（每个 Pod 都会连接所有平台）、WeChat 扫码登录、浏览器工具。启用这些功能时请保持 1 个副本。
+
 ### K8s Provisioner 配置
 
 <Steps>