mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-28 14:10:37 +00:00
063383d995
* fix(deploy): wire the multi-replica prerequisites into the Helm chart and compose The chart shipped gateway.replicas: 1 with a README citing the long-closed issue #3948, while its rendered config lacked run_ownership heartbeats and db run events: raising replicas made every starting Pod write its peers' lease-less runs off as orphans, and the Gateway's startup gate could not notice because its worker count only sees one Pod. Enable heartbeats and db run events in the default config, export DEER_FLOW_MULTI_INSTANCE from the replica count for the startup gate, generate and inject a shared AUTH_JWT_SECRET (per-Pod .jwt_secret files logged users out across replicas), declare a surge-then-drain rollout strategy, render a PodDisruptionBudget when replicas > 1, and bound uvicorn's graceful shutdown inside a 90s termination grace period. docker-compose gets the same uvicorn bound and a 60s stop_grace_period so the 30s memory flush is no longer SIGKILLed. Docs and comments now list what actually remains single-instance: IM channels, WeChat QR login and browser tools. * fix(deploy): honor existingAppSecret, declare multi-instance explicitly, cover the dev stack's shutdown Follow-up to the multi-replica prerequisites wiring, addressing review: - `deer-flow.appSecret` returns `existingAppSecret` when it is set. The gateway and frontend kept referencing the generated `<release>-app` Secret, which the chart stopped rendering, so every Pod sat in CreateContainerConfigError. The value is now declared in values.yaml. - `gateway.multiInstance` (boolean, default null) overrides the replica-derived DEER_FLOW_MULTI_INSTANCE, PodDisruptionBudget and AUTH_JWT_SECRET requirement for deployments scaled with kubectl or an HPA, which change the replica count without a Helm upgrade. - AUTH_JWT_SECRET is required, not optional, whenever the gateway is multi-instance: concurrently booting Pods otherwise race to write different `.jwt_secret` files on the home volume and sign sessions with different keys. - `gateway.uvicornGracefulShutdownSeconds: 0` is honored instead of being swallowed by `default 10`. - docker-compose-dev.yaml and dev-entrypoint.sh get the same uvicorn graceful-shutdown bound and `stop_grace_period: 60s`. - The chart README documents the one-time sign-out when upgrading a release that predates AUTH_JWT_SECRET and how to seed the previous key into the Secret first. - Tests read the shutdown budget from the Gateway through a shared helper instead of hard-coding 5/5/30, the raw-template string matches give way to `helm template` renders, and new renders cover multiInstance, existingAppSecret, a disabled PDB and a zero graceful-shutdown bound. * docs(changelog): link the multi-replica deploy entry to #6347 * fix(deploy): budget every bounded teardown hook and flag the multiInstance PDB window Review follow-ups on #6347: - The shutdown budget modeled only channel stop + run drain + memory flush (40s). The lifespan also runs the startup trash sweep, the notification delivery worker stop, browser-session close and the MCP session-pool close, each bounded at _SHUTDOWN_HOOK_TIMEOUT_SECONDS, plus a 1s retrieval-warm wait: about 61s worst case. _gateway_shutdown_budget.py now reads all of them from the Gateway, a drift guard pins the hook count against app.py, and both compose files move stop_grace_period from 60s to 90s (Helm's 90s already had room: 5 + 10 + 61). Comments and the chart README cite the full arithmetic. - gateway.multiInstance: true renders the PodDisruptionBudget immediately, which blocks drains of the single Pod until the scale-up happens; the README, values comment and both deployment guides say so and name podDisruptionBudget.enabled: false as the interim escape hatch.
69 lines
2.9 KiB
Python
69 lines
2.9 KiB
Python
"""Shutdown work every deployment's stop grace period must cover.
|
|
|
|
The chart's ``terminationGracePeriodSeconds`` and both compose files'
|
|
``stop_grace_period`` must outlast the Gateway's lifespan shutdown. Besides the
|
|
memory queue flush, ``app.gateway.app.lifespan`` runs several *sequential*,
|
|
individually bounded hooks around it, and ``langgraph_runtime`` drains in-flight
|
|
runs on exit. These numbers are read from the Gateway rather than copied into
|
|
the tests, so raising a drain budget or adding a bounded hook here fails the
|
|
grace-period tests instead of silently outgrowing the deployments.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from app.gateway.deps import _RUN_DRAIN_TIMEOUT_SECONDS
|
|
from deerflow.config.memory_config import MemoryConfig
|
|
|
|
#: Teardown steps in ``app.gateway.app.lifespan`` that each wait up to
|
|
#: ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS`` before proceeding. They run one after
|
|
#: another, so a pathological shutdown can spend the full budget on every one.
|
|
#: Keep this list in step with the ``asyncio.wait_for(..., timeout=
|
|
#: _SHUTDOWN_HOOK_TIMEOUT_SECONDS)`` sites in that function.
|
|
HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT: tuple[str, ...] = (
|
|
"startup trash sweep",
|
|
"notification delivery worker stop",
|
|
"channel service stop",
|
|
"browser session close",
|
|
"MCP session pool close",
|
|
)
|
|
|
|
|
|
def _app_constants() -> tuple[float, float]:
|
|
"""Return ``(_SHUTDOWN_HOOK_TIMEOUT_SECONDS, _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS)``."""
|
|
# Importing the FastAPI module builds nothing, but it is heavy; keep it lazy.
|
|
from app.gateway.app import _RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS, _SHUTDOWN_HOOK_TIMEOUT_SECONDS
|
|
|
|
return float(_SHUTDOWN_HOOK_TIMEOUT_SECONDS), float(_RETRIEVAL_WARM_SHUTDOWN_TIMEOUT_SECONDS)
|
|
|
|
|
|
def bounded_hooks_seconds() -> float:
|
|
"""Worst case for the sequential hooks bounded by ``_SHUTDOWN_HOOK_TIMEOUT_SECONDS``."""
|
|
hook_timeout, _ = _app_constants()
|
|
return len(HOOKS_BOUNDED_BY_SHUTDOWN_HOOK_TIMEOUT) * hook_timeout
|
|
|
|
|
|
def retrieval_warm_wait_seconds() -> float:
|
|
"""Brief wait for the derived retrieval-index rebuild before the flush."""
|
|
_, retrieval_warm_timeout = _app_constants()
|
|
return retrieval_warm_timeout
|
|
|
|
|
|
def run_drain_seconds() -> float:
|
|
"""Bound on draining in-flight runs before the checkpointer is torn down."""
|
|
return float(_RUN_DRAIN_TIMEOUT_SECONDS)
|
|
|
|
|
|
def memory_flush_seconds() -> float:
|
|
"""Default ``memory.shutdown_flush_timeout_seconds``."""
|
|
return float(MemoryConfig.model_fields["shutdown_flush_timeout_seconds"].default)
|
|
|
|
|
|
def lifespan_shutdown_seconds() -> float:
|
|
"""Worst-case lifespan shutdown: every bounded hook + retrieval-warm wait + run drain + memory flush.
|
|
|
|
Unbounded steps (OIDC close, scheduler / task-service stops, backend close)
|
|
are expected to be quick and are not modeled; deployments keep slack on top
|
|
of this figure for them.
|
|
"""
|
|
return bounded_hooks_seconds() + retrieval_warm_wait_seconds() + run_drain_seconds() + memory_flush_seconds()
|