Nan Gao b99ac916b6
fix(auth): count login lockouts in the database across Gateway replicas (#6501)
* fix(auth): count login lockouts in the database across Gateway replicas

POST /api/v1/auth/login/local throttled failed logins per client IP in a
module-level dict, so with N Gateway replicas behind one load balancer an
attacker got N x auth.local.max_login_attempts guesses and a lockout on
one replica was invisible to the others.

Move the counter behind a LoginThrottleStore (deerflow.persistence.
login_throttle): MemoryLoginThrottleStore keeps the historical per-process
semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one
login_throttle row per IP through the application database, incrementing
and starting the lock in a single dialect-native upsert, applying check
decisions with compare-and-set predicates in a separate write
transaction, and sweeping served locks plus 24h-idle counters in bounded
batches. Both stores pin the existing contract: the lock starts when the
count reaches max_login_attempts, the committed duration is honored when
the policy shrinks mid-lock, a served sentence is never resurrected,
tightening keeps the count, success resets the IP, and the 429 status and
message are unchanged.

New startup-only selector auth.local.throttle_storage (auto | memory | db,
default auto = the database whenever database.backend is sqlite or
postgres). The Gateway installs the store once at startup after the
persistence engine; deps._validate_login_throttle_storage warns when a
multi-process deployment keeps memory. Migration 0034_login_throttle
creates the table; config_version 55 -> 56 in config.example.yaml and the
Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and
migrations AGENTS.md, CHANGELOG.

* docs(changelog): link the login throttle entry to #6501

* fix(auth): re-check expiry on the row the login-throttle sweep deletes

The bounded cleanup in SqlLoginThrottleStore.record_failure deleted
`WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On
PostgreSQL READ COMMITTED the expiry predicate applied only inside the
snapshot-bound subquery: when the DELETE had to wait on a row lock held
by a concurrent check() that was extending the same lock under a raised
lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the
updated tuple, IP membership still held for the old candidate, and the
now-active lock was deleted; the next attempt from that IP got a clean
record and reached password verification (review of #6501, P2).

Repeat the served/stale predicate on the DELETE target as well
(SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate
subquery, so the re-evaluation sees the updated row's expiry. The other
writes already carry their full compare-and-set predicate on the target
row: _discard and the check() UPDATE both use _matches (ip, fail_count,
locked_at, lock_duration_seconds), so a tuple changed while they waited
no longer matches and the stale decision is skipped. _discard keeps the
equality compare-and-set rather than adding an expiry condition: its
early-release branch deletes a lock whose committed sentence has not
elapsed under the lowered live policy, which an expiry predicate would
wrongly block, and the equality predicate already pins every input of
the decision. reset stays unconditional by design.

Tests: a dialect-compile test asserts the DELETE's own WHERE carries the
expiry predicate for postgresql and sqlite (red on the previous HEAD), and
a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the
update-versus-sweep interleaving with two engines, waiting on
pg_stat_activity for the sweep to block before the extender commits.
Docs: sql.py concurrency contract and the migrations guide's 0034 bullet.

* fix(auth): keep an active login lock's clock and fail closed on explicit db

Address the "changes requested" review on #6501.

Lock clock (confirmed defect): record_failure re-stamped locked_at = now
for every failure on an already-locked row, so a sustained attacker's
sentence slid forward with each attempt. Both stores now apply one rule
in one atomic decision: new_count = fail_count + 1; below max_attempts the
IP counts with no lock (a failure after the operator raised the threshold
clears an existing lock); at or over it, a failure during an *active* lock
keeps locked_at and the committed lock_duration unchanged, while a
never-locked or served row starts anew with now / lockout_seconds (so the
failure that reaches a raised threshold starts a fresh lock rather than
keeping a stale locked_at). The SQL upsert expresses it as nested case()
on the row's own values; the memory store mirrors it. The sweeps in both
stores now leave the row being recorded to that decision instead of
deleting a served lock or stale counter first, which split the contract
(SQL would have produced a fresh (1, NULL, NULL) row). base.py states the
rule.

Explicit db downgrade: resolve_login_throttle_store raises SystemExit —
the failure type of the other startup validations in app.gateway.deps —
when throttle_storage='db' resolves to the shared table but no engine is
initialised, so langgraph_runtime refuses to start instead of silently
running every replica on per-process counters; the lazy bare-app path
applies the same rule and installs nothing. 'auto' keeps warning and
falling back. 'db' on a memory database still degrades with a warning
(there is no shared table to insist on).

Non-blocking items: _key() logs a warning when a client key is truncated
to 255 characters (never the key itself); the sql.py docstring covers
wall-clock locked_at across replicas (run NTP; skew shifts evaluation by
that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were
chosen; a store-level blocking-I/O anchor covers the SQL reset path; the
_validate_login_throttle_storage warning was verified to render
'throttle_storage=memory' for the StrEnum (the gate test now passes the
enum member and asserts that text). reset stays unconditional: a peer
failure that commits after a reset leaves fail_count=1, one genuine
failure after the cleared history, which a new test documents.

Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md,
AUTH_DESIGN.md and the throttle_storage field description.

* fix(auth): re-evaluate a login-throttle check whose compare-and-set missed

SqlLoginThrottleStore.check applied its decision (commit the live
duration, or clear a served / released lock) with a compare-and-set
predicate on the snapshot it had read, but ignored the affected-row
count. When a peer changed the row between the read and the write — an
admitted wrong password incrementing fail_count on another replica, or a
successful login clearing the row — the UPDATE or DELETE matched zero
rows, yet check reported the decision as applied: it could return an
extended 60s sentence while the row kept its 1s duration, so the next
check cleared the lock and let password verification through; a live
duration decrease could be dropped the same way; and a row a peer had
just cleared was still reported as locked (#6501 review, P2).

Every compare-and-set write in check now checks result.rowcount. On a
miss the row is re-read and re-evaluated from the top, up to
CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision
on the new snapshot and never clobbering a reset or replacement: a row
that is gone is reported as allowed, a fresh peer lock is reported as
locked, and an incremented count gets the extension or decrease
committed on it. Exhausting the bound fails closed: nothing is written
and the answer is the last snapshot's lock under the live policy, logged
at debug. The read stays in its own short transaction and each write in
a separate one.

The evaluation is now one shared pure function, evaluate_check in
base.py (CheckDecision), used by both stores so the contract cannot
diverge. The memory store keeps a single pass: its compare-and-set
cannot miss because there is no await between the read and the write on
the loop thread, which its docstring now states.

Tests (red first on df26c2acf) use a session proxy that runs the peer
mutation on a second engine right after the snapshot read ends its
transaction: the reviewer's raised-duration interleaving (59.5s reported
and committed, the increment survives, a later check still locked), the
mirrored decrease, early release on the fresh row, a reset landing
between read and write (0.0, no row recreated), and the bound-exhausted
fail-closed case.

* docs(changelog): resolve merge markers left in the Unreleased section

* test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test

Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting
0034 is the single head; this branch chains 0035_login_throttle after it, so
the test now pins the current head and walks the chain like its siblings.

* perf(auth): one session per login-throttle check, one transaction per failure

Two non-blocking performance suggestions from the #6501 review.

Single session per check. The router probed the row with store.get()
and then called store.check(), which re-read it: a recorded IP cost two
connection-pool checkouts and two SELECTs per request on the
unauthenticated login path, letting a lockout flood compete with
legitimate traffic for Postgres connections. LoginThrottleStore.check
now takes the policy as an async callable (LoginThrottlePolicy) and
resolves it lazily inside the store: the probe, the policy resolution
and the compare-and-set loop share one session, a clean IP returns 0.0
without ever calling the policy (the config.yaml read is still skipped
for the common case), and a recorded IP resolves it exactly once, also
across compare-and-set retries. The decision is made on a snapshot read
after the resolution: the policy read is a yield point, and a decision
that needs no write has no compare-and-set to catch a row a peer changed
meanwhile — so one session, but still a read after the yield. The
memory store takes the same signature and re-reads its record after the
await for the same reason. The router passes its asyncio.to_thread
policy read as the callable and drops the separate probe; get stays on
the protocol for tests and anchors. No short-TTL verdict cache: it would
mask a successful login's reset for the TTL window, and the single
session already halves the hot path.

Sweep in the upsert's transaction, plus an index. record_failure ran the
housekeeping DELETE in its own transaction and the upsert in a second
one, two commits (two WAL flushes on PostgreSQL) per failed login; the
sweep now rides in the upsert's transaction with one commit, the keep
exclusion already protecting the row being recorded. The new
ix_login_throttle_updated_at (ORM __table_args__ and the unreleased
0035 migration, created after the table and dropped before it, each
guarded separately so a table left by an interrupted attempt still gets
its index) serves the sweep's stale-counter predicate; the served-lock
predicate scans a table the sweep itself keeps small.

Tests (red first on a137fbc47): a clean IP never resolves the policy and
a recorded IP resolves it exactly once (both stores), the SQL store opens
exactly one session per check and commits once per record_failure
(counting wrappers), the migration test asserts the index after upgrade,
after completing a partial upgrade that lacked it, and gone after
downgrade while compare_metadata stays empty; the router-level
"mutated during policy resolution" tests now inject their mutation
through the policy callable.

* docs(changelog): note config_version 57 in the login throttle entry
2026-10-08 22:35:19 +08:00
..

DeerFlow Helm Chart

Deploys the full DeerFlow stack to Kubernetes: gateway (backend + embedded LangGraph runtime), frontend (Next.js), nginx (internal reverse proxy preserving the compose routing), and the provisioner (K8s-native sandbox that spawns code-execution Pods on demand).

This chart translates the production docker/docker-compose.yaml into native Kubernetes resources. No existing repo files are modified.

Prerequisites

  • A Kubernetes cluster (Docker Desktop K8s, OrbStack, kind, k3d, or a real cluster).
  • kubectl + helm 3.8+ installed (OCI registry support stabilized in 3.8; earlier 3.x needs HELM_EXPERIMENTAL_OCI=1).
  • The three DeerFlow images — either the published ones (see "Install the published chart" below) or built locally (see step 1).
  • An Ingress controller (e.g. ingress-nginx) if you enable ingress.

Install the published chart (GHCR)

The chart and all three images are published to GHCR on every v* release tag (see .github/workflows/container.yaml and chart.yaml). Skip the build step and install directly:

helm install deer-flow oci://ghcr.io/<owner>/charts/deer-flow \
  --version <version> \
  -n deer-flow --create-namespace \
  -f my-values.yaml

where <owner> is the GitHub owner the chart is published from and <version> matches the release tag without the leading v (tag v0.1.0 → --version 0.1.0).

Note: the helm chart is new in 2.1.0 - no chart was published before it. It publishes to oci://ghcr.io/<owner>/charts/deer-flow (the charts/ prefix keeps it distinct from the deer-flow-{backend,frontend,provisioner} image packages).

Point the chart at the published images:

image:
  registry: ghcr.io/<owner>     # owner prefix; images are <owner>/deer-flow-<name>
  tag: "<version>"              # match the release tag (sans leading `v`)
  pullSecrets:
    - { name: regcred }         # only if the GHCR package is private

The chart's gatewayImage / frontendImage / provisionerImage defaults already match the published image names (deer-flow-backend, deer-flow-frontend, deer-flow-provisioner), so only registry and tag are required. New GHCR packages default to private — flip the package to public in its GHCR settings page for unauthenticated pulls, otherwise create a pull secret (step 1) and reference it via image.pullSecrets.

The OCI chart and the images are versioned independently of the chart's appVersion; always set image.tag to the release that matches your chart --version unless you have a reason to pin differently.

1. Build & push images (custom builds only)

Skip this section if you're using the published chart above. To build the images yourself from the existing Dockerfiles:

REGISTRY=ghcr.io/yourorg
TAG=latest

# backend - build with the `postgres` extra so multi-replica deploys can use
# shared Postgres (matches the published image)
docker build -t $REGISTRY/deer-flow-backend:$TAG --build-arg UV_EXTRAS=postgres -f backend/Dockerfile .
# frontend
docker build -t $REGISTRY/deer-flow-frontend:$TAG -f frontend/Dockerfile .
# provisioner
docker build -t $REGISTRY/deer-flow-provisioner:$TAG -f docker/provisioner/Dockerfile docker/provisioner

docker push $REGISTRY/deer-flow-backend:$TAG
docker push $REGISTRY/deer-flow-frontend:$TAG
docker push $REGISTRY/deer-flow-provisioner:$TAG

These names match the chart's gatewayImage / frontendImage / provisionerImage defaults, so only image.registry and image.tag need to point at them.

If your registry needs auth, create a pull secret:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=youruser \
  --docker-password=yourtoken \
  -n deer-flow

2. Configure values

Copy and edit values.yaml → my-values.yaml. At minimum set:

image:
  registry: ghcr.io/yourorg
  tag: latest
  pullSecrets:
    - { name: regcred }

ingress:
  enabled: true
  className: nginx
  host: deer-flow.example.com
  tls:
    enabled: true
    secretName: deer-flow-tls

secrets:
  OPENAI_API_KEY: sk-...
  # add channel tokens, search keys, etc. as needed

The default ingress annotations permit a 100 MiB local .skill archive plus multipart framing, stream request bodies without ingress buffering, and allow up to 600 seconds for a response, which the API requests that wait on a model call or a whole run all need — skill install and custom-skill edits (each file is scanned by an LLM), /api/threads/{id}/compact, /api/input-polish, and /api/runs/wait. If you replace ingress.annotations, preserve equivalent size, streaming, and response-timeout settings for your ingress controller, or local skill uploads may fail before DeerFlow completes the installation and those requests may time out while Gateway is still working — for /api/runs/wait the disconnect also cancels the run.

Provide your model config under config (keep secrets as $VAR references — they resolve from the secrets map):

config: |
  config_version: 57
  models:
    - name: gpt-4
      use: langchain_openai:ChatOpenAI
      model: gpt-4
      api_key: $OPENAI_API_KEY
      request_timeout: 600.0
  sandbox:
    use: deerflow.community.aio_sandbox:AioSandboxProvider
    provisioner_url: http://provisioner:8002
    provisioner_api_key: $PROVISIONER_API_KEY   # injected from the app Secret
  database:
    backend: postgres
    postgres_url: $DATABASE_URL
    pool_recycle: 300
    command_timeout: 30
  checkpointer:
    type: postgres
    connection_string: $DATABASE_URL
  stream_bridge:
    type: redis   # cross-pod SSE; URL from DEER_FLOW_STREAM_BRIDGE_REDIS_URL
  knowledge_base:
    enabled: true
    scope_selection_enabled: false
    # Provider connection/retrieval settings belong on the knowledge_search tool.
  # Tools MUST be listed explicitly - the agent gets none otherwise
  # (BUILTIN_TOOLS only adds present_file + ask_clarification). The chart
  # default in values.yaml enables the sandbox tools + web tools (web_search,
  # web_fetch, image_search - no API key); when you override `config:`, copy
  # them in. Full list in values.yaml / config.example.yaml. The web tools need
  # outbound egress from the gateway pod.
  tool_groups:
    - name: web
    - name: file:read
    - name: file:write
    - name: bash
    - name: knowledge
  tools:
    - name: web_search
      group: web
      use: deerflow.community.ddg_search.tools:web_search_tool
      max_results: 5
    - name: web_fetch
      group: web
      use: deerflow.community.jina_ai.tools:web_fetch_tool
      timeout: 10
    - name: image_search
      group: web
      use: deerflow.community.image_search.tools:image_search_tool
      max_results: 5
    - name: knowledge_search
      group: knowledge
      use: deerflow.community.ragflow.tools:knowledge_search_tool
      base_url: http://ragflow:9380
      api_key: $RAGFLOW_API_KEY
    - name: list_knowledge_bases
      group: knowledge
      use: deerflow.community.ragflow.tools:list_knowledge_bases_tool
    - name: bash
      group: bash
      use: deerflow.sandbox.tools:bash_tool
    # also: ls, read_file, glob, grep, write_file, str_replace (see values.yaml)

$DATABASE_URL is injected from the postgres Secret (see below). The checkpointer: section is required for multi-replica operation — the LangGraph Store (cross-thread memory + thread list) reads it and does not fall back to database:. stream_bridge.type: redis is the default and routes live SSE events through the bundled redis StatefulSet (or redis.external). Because config: is a single override blob, a partial config: replaces the chart default entirely - keep the tools:/tool_groups: block (or the agent will have no tools) and the sandbox:/database:/checkpointer:/stream_bridge: sections shown above.

extensionsConfig is an initial seed, not a live read-only mount. An init container copies it into /app/backend/.deer-flow/extensions-config/extensions_config.json, where the Gateway can persist MCP and skill-state API updates. With persistence.home.enabled: true, the runtime file is kept on the home PVC and is not overwritten by later Helm upgrades; delete that runtime file before a pod restart only when you intentionally want a changed extensionsConfig seed to replace it. With persistence disabled, the writable copy uses emptyDir and is reseeded whenever the Pod is replaced.

3. Install (from a local chart checkout)

For a custom build or local development, install from the chart directory:

helm install deer-flow deploy/helm/deer-flow \
  -n deer-flow --create-namespace \
  -f my-values.yaml

4. Verify

kubectl -n deer-flow get pods
kubectl -n deer-flow port-forward svc/nginx 2026:2026
curl http://localhost:2026/health          # gateway liveness via nginx
curl http://localhost:2026/health/ready    # readiness: database, checkpointer, stream_bridge, provisioner

/health/ready is what the gateway readinessProbe hits. It answers 503 while Postgres or the Redis stream bridge is unreachable (stream_bridge: unreachable), so those pods leave the Service instead of accepting runs they cannot stream. The provisioner field reports the provisioner's own /health but never changes the status code: every gateway pod shares that one provisioner.

Hit the Ingress host (map it in /etc/hosts for local clusters) to load the UI.

Provisioner sanity check (/health is unauthenticated; /api/* requires the PROVISIONER_API_KEY the chart generates into the app Secret, so a 401 from the gateway's sandbox calls means the two Pods disagree on that key):

kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002/health

Architecture notes

  • PostgreSQL is the default database. A bundled single-instance postgres StatefulSet (postgresql.enabled: true) runs in the namespace and the gateway connects via the in-cluster Service. The DSN is auto-generated into a Secret (key database-url) and injected as DATABASE_URL; config.yaml references it as $DATABASE_URL in database.postgres_url. Schema is bootstrapped automatically on gateway startup (alembic create_all + stamp head). For real HA, disable the bundled instance and point at a managed DB with a full DSN (the chart wraps it into the Secret) or with a Secret you manage (key database-url):

    postgresql:
      enabled: false
      external:
        databaseUrl: postgresql://deerflow:changeme@mydb.example.com:5432/deerflow
        # or: existingSecret: my-deerflow-db   # key `database-url`
    

    URL-encode special characters in the DSN password (for example, @ as %40). The chart uses an external databaseUrl verbatim and does not rewrite the DSN in a user-managed Secret.

  • Graceful shutdown & memory drain. The gateway pod sets terminationGracePeriodSeconds (default 90s, overridable via gateway.terminationGracePeriodSeconds) plus an optional preStop sleep (gateway.preStopSleepSeconds, default 5s), and bounds uvicorn's --timeout-graceful-shutdown (gateway.uvicornGracefulShutdownSeconds, default 10s) so an idle SSE connection cannot hold up lifespan shutdown indefinitely. The grace period MUST exceed the Gateway's graceful-shutdown work — the preStop sleep, the uvicorn timeout, and the lifespan's worst case: seven hooks bounded at 5s each (startup trash sweep, notification delivery worker, channel service, scheduled task service, subagent batch service, browser sessions, MCP session pool), the 1s retrieval-warm wait, the in-flight run drain (5s) and the memory-queue drain (memory.shutdown_flush_timeout_seconds, default 30s), about 71s in total, plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. backend/tests/_gateway_shutdown_budget.py reads these bounds from the Gateway and pins the chart and compose budgets against them. When you raise memory.shutdown_flush_timeout_seconds, raise gateway.terminationGracePeriodSeconds to match (preStop + uvicorn timeout + ~41s of bounded hooks and drains + memory drain + buffer).

  • Gateway replicas. Run control is cross-pod-safe since the work tracked by issue #3948 landed (#4003, #4064, #4500): admission is a durable one-active-run-per-thread constraint with Idempotency-Key reuse, a cancel that lands on a non-owner Pod is recorded durably and executed by the owner on its next lease renewal, and a crashed Pod's runs are reclaimed by a peer once their lease expires. The chart's default config enables what that needs — Postgres for database and checkpointer, run_events.backend: db, run_ownership.heartbeat_enabled: true, and the Redis stream bridge (sandbox ownership leases are inferred from it) — and exports DEER_FLOW_MULTI_INSTANCE=true whenever gateway.replicas > 1, so the Gateway's startup gate verifies those prerequisites instead of staying inert (its worker count only sees one Pod). If you scale with kubectl scale or an HPA instead of gateway.replicas, set gateway.multiInstance: true first: those change the replica count without a Helm upgrade, so the derived declaration, the PodDisruptionBudget and the required shared AUTH_JWT_SECRET would otherwise keep their single-replica rendering. Mind the window between setting it and the actual scale-up: the PodDisruptionBudget (minAvailable: 1) is rendered immediately and blocks every voluntary eviction and node drain while only one Pod exists, so set gateway.podDisruptionBudget.enabled: false for that interim if you need to drain nodes first. gateway.replicas: 1 stays the default because the following are still single-instance: IM channels (every Pod would connect to every platform — Telegram polling conflicts and Discord double-processes; keep 1 replica while channels are enabled), the WeChat QR login, and browser tools. Before raising replicas, also set persistence.home.accessMode: ReadWriteMany on multi-node clusters (thread uploads, outputs, memory and extensions_config.json live on that volume) and agent_storage.backend: db so custom agents are visible on every Pod. Memory itself is multi-instance safe on that volume (per-user file locks, journaled writes, and a Pod re-syncs a user's search index on its next search after a peer writes), but DeerMem's derived SQLite FTS5 index is not: SQLite WAL is unsupported on network filesystems, every Pod start would rebuild a shared index under its peers, and one Pod's corruption recovery would delete it from under them. The default config therefore sets memory.backend_config.retrieval_index_path: /var/lib/deerflow/memory-index, a Pod-local emptyDir the gateway Deployment mounts; keep that line when you override config: — a multi-instance gateway that leaves the index under the memory root logs a warning at startup. The index is rebuilt from the Markdown facts on every Pod start, so losing the emptyDir loses nothing. A PodDisruptionBudget (minAvailable: 1) is rendered automatically for a multi-instance gateway (same rule), and the rollout strategy is surge-then-drain (maxSurge: 1, maxUnavailable: 0).

  • App secret. <release>-app holds BETTER_AUTH_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN, AUTH_JWT_SECRET (the session-cookie signing key) and PROVISIONER_API_KEY (the key the gateway presents to the sandbox provisioner), each generated once and preserved across upgrades via lookup. existingAppSecret points the gateway, frontend and provisioner at a Secret you manage instead (no <release>-app is generated); it must carry the first three keys and, while provisioner.enabled is true, also PROVISIONER_API_KEY: the provisioner's verify_api_key middleware answers 401 to every /api/* request while its key is empty or differs from the one the gateway sends, so a deployment without it cannot create a single sandbox. The chart injects the same Secret value into both Pods and the default config references it as sandbox.provisioner_api_key: $PROVISIONER_API_KEY (keep that line when you override config:; the harness refuses to start when a referenced variable is unset). With provisioner.enabled: false the gateway takes PROVISIONER_API_KEY from secrets or existingSecret when you supply one, so an external provisioner keeps its key (the chart emits no env entry, which would win over envFrom); otherwise the gateway's start command defaults the variable to an empty string, so the default config still loads even when a user-managed provider Secret holds only model keys. AUTH_JWT_SECRET is required whenever the gateway is multi-instance — without it, concurrently booting Pods race to write their own .jwt_secret on the home volume and sign sessions with different keys — and only a single Pod may omit it and fall back to that file. Upgrading a release that predates AUTH_JWT_SECRET generates a new key, so every existing browser session is signed out once. To keep sessions, copy the key the gateway has been using into the Secret before upgrading; lookup then preserves it like the other app secrets:

    JWT=$(kubectl -n deer-flow exec deploy/deer-flow-gateway -- cat /app/backend/.deer-flow/.jwt_secret)
    kubectl -n deer-flow patch secret deer-flow-app -p "{\"stringData\":{\"AUTH_JWT_SECRET\":\"$JWT\"}}"
    
  • Scheduled task recovery. If a deployment explicitly enables scheduler.multi_instance: true, it must use shared Postgres, run_ownership.heartbeat_enabled: true, and run_events.backend: db. Scheduler startup then preserves live scheduled runs owned by another Pod, atomically takes over only expired leases, and fences stale post-launch bookkeeping. max_concurrent_runs is a shared global cap across Pods, including pre-launch dispatch reservations. Restart all Gateway Pods after changing these startup-only settings. IM channels remain single-instance regardless (see Gateway replicas above).

  • Redis stream bridge. A bundled single-instance redis StatefulSet (redis.enabled: true, redis:7-alpine) runs in the namespace and the gateway connects via the in-cluster Service. Per-run SSE events are stored in Redis Streams (PR #3191) so a client connected to any gateway pod receives live events and reconnect resumes from Last-Event-ID. The URL is auto-generated into a Secret (key redis-url) and injected as DEER_FLOW_STREAM_BRIDGE_REDIS_URL; config.yaml sets stream_bridge.type: redis by default. No-auth by default (ClusterIP isolation, matching compose); set redis.auth.password to enable AUTH. For a managed Redis, disable the bundled instance and point at it via redis.external. The gateway readiness probe pings this Redis on every check and reports the pod unready while it is unreachable, since without the bridge no run can publish, stream or be cancelled from a peer pod.

  • Persistence. A PVC (<release>-home) backs /app/backend/.deer-flow (sqlite DB, memory, custom agents, per-thread user-data). The gateway mounts it with subPath: deer-flow so the layout matches the provisioner's PVC user-data mode. Default ReadWriteOnce; use ReadWriteMany (NFS) on multi-node clusters so sandbox Pods on other nodes can mount it.

  • Provisioner RBAC. The provisioner gets a ServiceAccount with a namespaced Role (get/list/watch/create/delete on pods + services) and a narrow ClusterRole (namespace get/create). It uses in-cluster service-account creds — no kubeconfig mount. The unused update/patch/pods-exec/events verbs were dropped (audited against docker/provisioner/app.py).

  • Skills. Disabled by default (emptyDir at /app/skills). Populate via skills.existingClaim or skills.configMap, or bake skills into a custom gateway image.

Security

Enforced posture

All workloads run as non-root with all Linux capabilities dropped. No container escalates privileges or runs as uid 0.

workload runAsUser fsGroup writable-path handling
gateway 1000 1000 .deer-flow PVC group-writable via fsGroup; PYTHONDONTWRITEBYTECODE=1 suppresses .pyc writes; UV_CACHE_DIR=/tmp
frontend 1000 (node) 1000 emptyDir at /app/frontend/.next/cache (root-owned in the image)
nginx 101 (nginx) 101 command writes the rendered config to /tmp/nginx.conf and loads nginx -c /tmp/nginx.conf (since /etc/nginx is root-owned); emptyDir at /var/cache/nginx
provisioner 1000 — no PVC; PYTHONDONTWRITEBYTECODE=1
postgres 999 (postgres) 999 official postgres:16 entrypoint detects non-root and skips the chown/gosu dance; data PVC group-writable via fsGroup
redis 999 (redis) 999 official redis:7-alpine entrypoint detects non-root and skips the gosu dance; data PVC group-writable via fsGroup

Every container sets:

  • runAsNonRoot: true
  • allowPrivilegeEscalation: false
  • capabilities.drop: ["ALL"]
  • seccompProfile: { type: RuntimeDefault }

All listening ports are >1024 (8001 / 3000 / 2026 / 8002 / 5432), so no NET_BIND_SERVICE capability is required.

ConfigMap rollout. ConfigMaps mount via subPath, which does not receive in-place updates — a helm upgrade that changes only a ConfigMap would leave pods on stale config. Each pod template carries a checksum/* annotation (SHA256 of the rendered ConfigMap): checksum/config + checksum/extensions on the gateway, checksum/nginx on nginx. Any content change alters the pod spec and triggers a rolling restart.

Resource defaults. Every workload ships with modest requests+limits in values.yaml; override per workload (gateway.resources, frontend.resources, nginx.resources, provisioner.resources, postgresql.primary.resources, redis.primary.resources).

Not yet enforced (deferred hardening)

These are intentionally not set in this chart revision. Each can be added per-workload with testing:

  • readOnlyRootFilesystem: true — makes the container's root filesystem immutable so a compromised process can't persist changes to the image. Not enabled because it requires auditing every runtime write path and mounting an emptyDir over each. Known paths:
    • gateway / frontend / nginx / provisioner: /tmp (uv cache, python tempfiles, the nginx config + pid, node temp) — one emptyDir at /tmp each.
    • postgres: /tmp and /var/run/postgresql (the Unix-socket dir). The first four are mechanical. postgres is the hard case — the official image writes its socket to /var/run/postgresql and isn't designed for a read-only root, so it may need socket-path redirection (PGHOST/unix_socket_directories). Optionally, add USER directives to the backend/Dockerfile, frontend/Dockerfile, and docker/provisioner/Dockerfile so the images are non-root by default (defense in depth — the chart already forces the uid via securityContext, so this is not required). A cluster enforcing the restricted Pod Security Admission standard would require this setting.
  • Provisioner RBAC narrowing. The Role grants get/list/watch/create/delete on pods and services in the namespace (update/patch/pods-exec/events were dropped as unused). These verbs still apply to all Pods in the namespace, not just sandbox Pods — RBAC can't scope by label, so the remaining options are a dedicated sandbox namespace or admission control (OPA/Kyverno).
  • startupProbe. Workloads have readiness + liveness probes but no startup probe. The gateway's livenessProbe.initialDelaySeconds: 30 covers slow starts today; a startupProbe would let it take arbitrarily long to initialize without risking a liveness kill during a cold start (e.g. slow model config load).

None of these affect correctness of the current deployment.

Migrating an existing volume to non-root

fsGroup does not apply to subPath mounts, and it changes group ownership but not file mode — so a PVC written by an earlier root run (e.g. a cluster that ran the gateway as root before enabling this hardening, or a backup restore of root-owned files) will keep files like .jwt_secret at 0600 root:root. The non-root gateway (uid 1000) then can't read them and crashes on the first auth request with RuntimeError: Failed to read JWT secret from .../​.jwt_secret.

Fresh installs are unaffected — uid 1000 creates every file as 1000:1000.

To fix an existing root-written PVC, run a one-shot root pod that chowns the volume to the gateway uid (1000), then restart the gateway:

cat <<'EOF' | kubectl apply -n deer-flow -f -
apiVersion: v1
kind: Pod
metadata: { name: fix-home-perms, namespace: deer-flow }
spec:
  restartPolicy: Never
  containers:
    - name: chown
      image: busybox:1.36
      command: ["sh", "-c"]
      args: ["chown -R 1000:1000 /home-pvc/deer-flow && chmod -R g+rwX /home-pvc/deer-flow"]
      volumeMounts:
        - { name: home, mountPath: /home-pvc }
  volumes:
    - name: home
      persistentVolumeClaim: { claimName: deer-flow-deer-flow-home }
EOF
kubectl -n deer-flow wait --for=condition=Ready pod/fix-home-perms --timeout=30s
kubectl -n deer-flow delete pod fix-home-perms
kubectl -n deer-flow rollout restart deploy/deer-flow-deer-flow-gateway

(On a single-node cluster the fix pod can mount the RWO PVC concurrently with the gateway; on multi-node, scale the gateway to 0 first.) A durable alternative — an opt-in root volumePermissions initContainer that chowns on every start (the Bitnami pattern) — is not yet wired into this chart; it would introduce a root container, so it's left as an operator decision for now.

Sandbox Service type

The provisioner exposes each sandbox Pod behind a per-sandbox Service whose type is controlled by provisioner.sandboxServiceType (default ClusterIP).

ClusterIP (default, recommended). The provisioner returns a cluster-DNS URL - http://sandbox-<id>-svc.<namespace>.svc.cluster.local:8080 - so the gateway reaches its sandbox entirely inside the cluster network. No node IP, no high port, and the code-execution surface is not bound on every node's interfaces. This is correct for the chart, where the gateway always runs in-cluster.

NodePort (Docker-Compose/hybrid escape hatch). Set provisioner.sandboxServiceType: NodePort only when the gateway is not in K8s (e.g. the compose dev path, where the gateway is a container reaching sandbox Pods on the host's Docker Desktop K8s). The provisioner then returns http://{NODE_HOST}:{NodePort}. NODE_HOST defaults to the provisioner pod's node IP via the downward API (status.hostIP); because a NodePort is exposed on every node, the gateway can reach <node-IP>:<NodePort> on most clusters without configuration. Override provisioner.nodeHost only if your CNI or network policy blocks pod->node-IP traffic:

kubectl get nodes -o wide    # use INTERNAL-IP or EXTERNAL-IP
provisioner:
  sandboxServiceType: NodePort
  nodeHost: 192.168.x.x

On multi-node clusters, also switch persistence.home.accessMode to ReadWriteMany (this is orthogonal to the Service type - it governs whether a sandbox Pod can be scheduled on a node other than the gateway's).

Sandbox lark-cli runtime (optional)

The Lark/Feishu lark-cli integration needs a lark-cli binary inside the sandbox. For remote/Kubernetes (provisioner) deployments the sandbox-side path comes from an optional runtime image instead of an install-time GitHub download. The chart exposes the same two knobs the Compose stack reads on the provisioner:

provisioner:
  # Pattern A - an init container copies the binaries into a shared emptyDir.
  larkCliInitImage: deer-flow/lark-cli-init:v1.0.65
  # Pattern B - a shim init container + broker sidecar owns the credentials, so
  # the plaintext config/data dirs are never mounted into the sandbox.
  # Supersedes larkCliInitImage when both are set.
  larkCliBrokerImage: deer-flow/lark-cli-broker:v1.0.65

Both default to empty, which leaves the feature off (legacy behavior) and makes an in-sandbox lark-cli call fail with exit 127 (command not found). When set, they render LARK_CLI_INIT_IMAGE / LARK_CLI_BROKER_IMAGE on the provisioner Deployment - the names docker/docker-compose.yaml uses - and the variable is omitted entirely while empty. Point them at a tag that exists in a registry your nodes can pull from (mirror the registry prefix if you do not use Docker Hub). The images are built from docker/lark-cli-init and docker/lark-cli-broker; see those READMEs and the root README's Lark section for the build/publish flow and the credential model. Broker mode is the safer choice on a shared cluster: the app secret and OAuth tokens stay in the sidecar instead of the sandbox container.

An image here does not authenticate anyone by itself. The Gateway only asks the provisioner to attach the runtime once the Lark integration pack is installed for the user, so the sandbox gets the binary but the per-user credentials still follow the normal install/authorize flow. The Lark integration status reports sandbox_runtime_mode / sandbox_runtime_ready so the Settings UI surfaces a missing runtime instead of a later command not found.

Lint / dry-run

helm lint deploy/helm/deer-flow
helm template deer-flow deploy/helm/deer-flow -n deer-flow -f my-values.yaml | \
  kubectl apply --dry-run=client -f -

Uninstall

helm uninstall deer-flow -n deer-flow
# the PVC is NOT deleted by default — remove it manually if desired:
kubectl -n deer-flow delete pvc -l app.kubernetes.io/instance=deer-flow