Nan Gao d18d2937e7
feat(channels): add channels.leader_lease and gate an explicitly disabled lease under multi-process signals (#6746)
* feat(channels): add channels.leader_lease and gate it under multi-process signals

IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.

Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.

ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.

* docs(changelog): link #6746

* fix(channels): stop recommending the unimplemented leader lease as a remedy

Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.

The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.

* fix(channels): warn on an enabled unknown channel key; reword the changelog entry

Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.

The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.

* fix(dev): disable UI channel providers in multi-instance overlay

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-12 23:05:16 +08:00
..

DeerFlow Helm Chart

Deploys the full DeerFlow stack to Kubernetes: gateway (backend + embedded LangGraph runtime), frontend (Next.js), nginx (internal reverse proxy preserving the compose routing), and the provisioner (K8s-native sandbox that spawns code-execution Pods on demand).

This chart translates the production docker/docker-compose.yaml into native Kubernetes resources. No existing repo files are modified.

Prerequisites

  • A Kubernetes cluster (Docker Desktop K8s, OrbStack, kind, k3d, or a real cluster).
  • kubectl + helm 3.8+ installed (OCI registry support stabilized in 3.8; earlier 3.x needs HELM_EXPERIMENTAL_OCI=1).
  • The three DeerFlow images — either the published ones (see "Install the published chart" below) or built locally (see step 1).
  • An Ingress controller (e.g. ingress-nginx) if you enable ingress.

Install the published chart (GHCR)

The chart and all three images are published to GHCR on every v* release tag (see .github/workflows/container.yaml and chart.yaml). Skip the build step and install directly:

helm install deer-flow oci://ghcr.io/<owner>/charts/deer-flow \
  --version <version> \
  -n deer-flow --create-namespace \
  -f my-values.yaml

where <owner> is the GitHub owner the chart is published from and <version> matches the release tag without the leading v (tag v0.1.0 → --version 0.1.0).

Note: the helm chart is new in 2.1.0 - no chart was published before it. It publishes to oci://ghcr.io/<owner>/charts/deer-flow (the charts/ prefix keeps it distinct from the deer-flow-{backend,frontend,provisioner} image packages).

Point the chart at the published images:

image:
  registry: ghcr.io/<owner>     # owner prefix; images are <owner>/deer-flow-<name>
  tag: "<version>"              # match the release tag (sans leading `v`)
  pullSecrets:
    - { name: regcred }         # only if the GHCR package is private

The chart's gatewayImage / frontendImage / provisionerImage defaults already match the published image names (deer-flow-backend, deer-flow-frontend, deer-flow-provisioner), so only registry and tag are required. New GHCR packages default to private — flip the package to public in its GHCR settings page for unauthenticated pulls, otherwise create a pull secret (step 1) and reference it via image.pullSecrets.

The OCI chart and the images are versioned independently of the chart's appVersion; always set image.tag to the release that matches your chart --version unless you have a reason to pin differently.

1. Build & push images (custom builds only)

Skip this section if you're using the published chart above. To build the images yourself from the existing Dockerfiles:

REGISTRY=ghcr.io/yourorg
TAG=latest

# backend - build with the `postgres` extra so multi-replica deploys can use
# shared Postgres (matches the published image)
docker build -t $REGISTRY/deer-flow-backend:$TAG --build-arg UV_EXTRAS=postgres -f backend/Dockerfile .
# frontend
docker build -t $REGISTRY/deer-flow-frontend:$TAG -f frontend/Dockerfile .
# provisioner
docker build -t $REGISTRY/deer-flow-provisioner:$TAG -f docker/provisioner/Dockerfile docker/provisioner

docker push $REGISTRY/deer-flow-backend:$TAG
docker push $REGISTRY/deer-flow-frontend:$TAG
docker push $REGISTRY/deer-flow-provisioner:$TAG

These names match the chart's gatewayImage / frontendImage / provisionerImage defaults, so only image.registry and image.tag need to point at them.

If your registry needs auth, create a pull secret:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=youruser \
  --docker-password=yourtoken \
  -n deer-flow

2. Configure values

Copy and edit values.yaml → my-values.yaml. At minimum set:

image:
  registry: ghcr.io/yourorg
  tag: latest
  pullSecrets:
    - { name: regcred }

ingress:
  enabled: true
  className: nginx
  host: deer-flow.example.com
  tls:
    enabled: true
    secretName: deer-flow-tls

secrets:
  OPENAI_API_KEY: sk-...
  # add channel tokens, search keys, etc. as needed

The default ingress annotations permit a 100 MiB local .skill archive plus multipart framing, stream request bodies without ingress buffering, and allow up to 600 seconds for a response, which the API requests that wait on a model call or a whole run all need — skill install and custom-skill edits (each file is scanned by an LLM), /api/threads/{id}/compact, /api/input-polish, and /api/runs/wait. If you replace ingress.annotations, preserve equivalent size, streaming, and response-timeout settings for your ingress controller, or local skill uploads may fail before DeerFlow completes the installation and those requests may time out while Gateway is still working — for /api/runs/wait the disconnect also cancels the run.

Provide your model config under config (keep secrets as $VAR references — they resolve from the secrets map):

config: |
  config_version: 60
  models:
    - name: gpt-4
      use: langchain_openai:ChatOpenAI
      model: gpt-4
      api_key: $OPENAI_API_KEY
      request_timeout: 600.0
  sandbox:
    use: deerflow.community.aio_sandbox:AioSandboxProvider
    provisioner_url: http://provisioner:8002
    provisioner_api_key: $PROVISIONER_API_KEY   # injected from the app Secret
  database:
    backend: postgres
    postgres_url: $DATABASE_URL
    pool_recycle: 300
    command_timeout: 30
  checkpointer:
    type: postgres
    connection_string: $DATABASE_URL
  stream_bridge:
    type: redis   # cross-pod SSE; URL from DEER_FLOW_STREAM_BRIDGE_REDIS_URL
  knowledge_base:
    enabled: true
    scope_selection_enabled: false
    # Provider connection/retrieval settings belong on the knowledge_search tool.
  # Tools MUST be listed explicitly - the agent gets none otherwise
  # (BUILTIN_TOOLS only adds present_file + ask_clarification). The chart
  # default in values.yaml enables the sandbox tools + web tools (web_search,
  # web_fetch, image_search - no API key); when you override `config:`, copy
  # them in. Full list in values.yaml / config.example.yaml. The web tools need
  # outbound egress from the gateway pod.
  tool_groups:
    - name: web
    - name: file:read
    - name: file:write
    - name: bash
    - name: knowledge
  tools:
    - name: web_search
      group: web
      use: deerflow.community.ddg_search.tools:web_search_tool
      max_results: 5
    - name: web_fetch
      group: web
      use: deerflow.community.jina_ai.tools:web_fetch_tool
      timeout: 10
    - name: image_search
      group: web
      use: deerflow.community.image_search.tools:image_search_tool
      max_results: 5
    - name: knowledge_search
      group: knowledge
      use: deerflow.community.ragflow.tools:knowledge_search_tool
      base_url: http://ragflow:9380
      api_key: $RAGFLOW_API_KEY
    - name: list_knowledge_bases
      group: knowledge
      use: deerflow.community.ragflow.tools:list_knowledge_bases_tool
    - name: bash
      group: bash
      use: deerflow.sandbox.tools:bash_tool
    # also: ls, read_file, glob, grep, write_file, str_replace (see values.yaml)

$DATABASE_URL is injected from the postgres Secret (see below). The checkpointer: section is required for multi-replica operation — the LangGraph Store (cross-thread memory + thread list) reads it and does not fall back to database:. stream_bridge.type: redis is the default and routes live SSE events through the bundled redis StatefulSet (or redis.external). Because config: is a single override blob, a partial config: replaces the chart default entirely - keep the tools:/tool_groups: block (or the agent will have no tools) and the sandbox:/database:/checkpointer:/stream_bridge: sections shown above.

extensionsConfig is an initial seed, not a live read-only mount. An init container copies it into /app/backend/.deer-flow/extensions-config/extensions_config.json, where the Gateway can persist MCP and skill-state API updates. With persistence.home.enabled: true, the runtime file is kept on the home PVC and is not overwritten by later Helm upgrades; delete that runtime file before a pod restart only when you intentionally want a changed extensionsConfig seed to replace it. With persistence disabled, the writable copy uses emptyDir and is reseeded whenever the Pod is replaced.

3. Install (from a local chart checkout)

For a custom build or local development, install from the chart directory:

helm install deer-flow deploy/helm/deer-flow \
  -n deer-flow --create-namespace \
  -f my-values.yaml

4. Verify

kubectl -n deer-flow get pods
kubectl -n deer-flow port-forward svc/nginx 2026:2026
curl http://localhost:2026/health          # gateway liveness via nginx
curl http://localhost:2026/health/ready    # readiness: database, checkpointer, stream_bridge, provisioner

/health/ready is what the gateway readinessProbe hits. It answers 503 while Postgres or the Redis stream bridge is unreachable (stream_bridge: unreachable), so those pods leave the Service instead of accepting runs they cannot stream. The provisioner field reports the provisioner's own /health but never changes the status code: every gateway pod reaches the same provisioner Service, whichever replica answers.

Hit the Ingress host (map it in /etc/hosts for local clusters) to load the UI.

Provisioner sanity check (/health is unauthenticated; /api/* requires the PROVISIONER_API_KEY the chart generates into the app Secret, so a 401 from the gateway's sandbox calls means the two Pods disagree on that key):

kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002/health

Architecture notes

  • PostgreSQL is the default database. A bundled single-instance postgres StatefulSet (postgresql.enabled: true) runs in the namespace and the gateway connects via the in-cluster Service. The DSN is auto-generated into a Secret (key database-url) and injected as DATABASE_URL; config.yaml references it as $DATABASE_URL in database.postgres_url. Schema is bootstrapped automatically on gateway startup (alembic create_all + stamp head). For real HA, disable the bundled instance and point at a managed DB with a full DSN (the chart wraps it into the Secret) or with a Secret you manage (key database-url):

    postgresql:
      enabled: false
      external:
        databaseUrl: postgresql://deerflow:changeme@mydb.example.com:5432/deerflow
        # or: existingSecret: my-deerflow-db   # key `database-url`
    

    URL-encode special characters in the DSN password (for example, @ as %40). The chart uses an external databaseUrl verbatim and does not rewrite the DSN in a user-managed Secret.

  • Graceful shutdown & memory drain. The gateway pod sets terminationGracePeriodSeconds (default 90s, overridable via gateway.terminationGracePeriodSeconds) plus an optional preStop sleep (gateway.preStopSleepSeconds, default 5s), and bounds uvicorn's --timeout-graceful-shutdown (gateway.uvicornGracefulShutdownSeconds, default 10s) so an idle SSE connection cannot hold up lifespan shutdown indefinitely. The grace period MUST exceed the Gateway's graceful-shutdown work — the preStop sleep, the uvicorn timeout, and the lifespan's worst case: seven hooks bounded at 5s each (startup trash sweep, notification delivery worker, channel service, scheduled task service, subagent batch service, browser sessions, MCP session pool), the 1s retrieval-warm wait, the in-flight run drain (5s) and the memory-queue drain (memory.shutdown_flush_timeout_seconds, default 30s), about 71s in total, plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. backend/tests/_gateway_shutdown_budget.py reads these bounds from the Gateway and pins the chart and compose budgets against them. When you raise memory.shutdown_flush_timeout_seconds, raise gateway.terminationGracePeriodSeconds to match (preStop + uvicorn timeout + ~41s of bounded hooks and drains + memory drain + buffer).

  • Gateway replicas. Run control is cross-pod-safe since the work tracked by issue #3948 landed (#4003, #4064, #4500): admission is a durable one-active-run-per-thread constraint with Idempotency-Key reuse, a cancel that lands on a non-owner Pod is recorded durably and executed by the owner on its next lease renewal, and a crashed Pod's runs are reclaimed by a peer once their lease expires. The chart's default config enables what that needs — Postgres for database and checkpointer, run_events.backend: db, run_ownership.heartbeat_enabled: true, and the Redis stream bridge (sandbox ownership leases are inferred from it) — and exports DEER_FLOW_MULTI_INSTANCE=true whenever gateway.replicas > 1, so the Gateway's startup gate verifies those prerequisites instead of staying inert (its worker count only sees one Pod). If you scale with kubectl scale or an HPA instead of gateway.replicas, set gateway.multiInstance: true first: those change the replica count without a Helm upgrade, so the derived declaration, the PodDisruptionBudget and the required shared AUTH_JWT_SECRET would otherwise keep their single-replica rendering. Mind the window between setting it and the actual scale-up: the PodDisruptionBudget (minAvailable: 1) is rendered immediately and blocks every voluntary eviction and node drain while only one Pod exists, so set gateway.podDisruptionBudget.enabled: false for that interim if you need to drain nodes first. gateway.replicas: 1 stays the default because the following are still single-instance: IM channels (every Pod would connect to every platform — Telegram polling conflicts and Discord double-processes; keep 1 replica while channels are enabled). channels.leader_lease in config (default auto) is reserved for the per-provider leader lease that will give one Pod each platform connection; until that ships the setting only drives the startup gate, which refuses enabled: false while channels are enabled, and auto/true do not yet make several replicas safe. Also single-instance: the WeChat QR login and browser tools. Before raising replicas, also set persistence.home.accessMode: ReadWriteMany on multi-node clusters (thread uploads, outputs, memory and extensions_config.json live on that volume) and agent_storage.backend: db so custom agents are visible on every Pod. Memory itself is multi-instance safe on that volume (per-user file locks, journaled writes, and a Pod re-syncs a user's search index on its next search after a peer writes), but DeerMem's derived SQLite FTS5 index is not: SQLite WAL is unsupported on network filesystems, every Pod start would rebuild a shared index under its peers, and one Pod's corruption recovery would delete it from under them. The default config therefore sets memory.backend_config.retrieval_index_path: /var/lib/deerflow/memory-index, a Pod-local emptyDir the gateway Deployment mounts; keep that line when you override config: — a multi-instance gateway that leaves the index under the memory root logs a warning at startup. The index is rebuilt from the Markdown facts on every Pod start, so losing the emptyDir loses nothing. A PodDisruptionBudget (minAvailable: 1; an integer or a percentage string such as "50%", while 0 fails the render -- disable the budget instead) is rendered automatically for a multi-instance gateway (same rule), and the rollout strategy is surge-then-drain (maxSurge: 1, maxUnavailable: 0). The sandbox provisioner is not on the single-instance list: it scales independently (next item).

  • Provisioner replicas. provisioner.replicas (default 1) scales the sandbox provisioner. It keeps no state of its own between requests: the sandbox Pods and Services it creates carry the labels (app=deer-flow-sandbox, sandbox-id) that are its only registry; create, discover, list and destroy all read them back from the API server; create tolerates the 409 AlreadyExists a concurrent creator on another replica produces (several gateway Pods already run that race against one provisioner) and destroy tolerates 404; and NodePorts are allocated by the API server, not the provisioner. The provisioner Service therefore spreads the gateway's calls over any number of replicas without affinity. At 1 replica it is the last single point of failure in a multi-replica gateway deployment: while its Pod restarts or its node drains no sandbox can be created or discovered (running sandboxes are unaffected; the gateway talks to them directly). A PodDisruptionBudget (provisioner.podDisruptionBudget, minAvailable: 1) is rendered only while provisioner.replicas > 1 -- on a single replica it would block every node drain -- and the rollout strategy is surge-then-drain (maxSurge: 1, maxUnavailable: 0), both mirroring the gateway. There is no multiInstance switch for the provisioner because nothing else in the chart renders differently per replica count; scaling it with kubectl scale only leaves it without the budget.

  • App secret. <release>-app holds BETTER_AUTH_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN, AUTH_JWT_SECRET (the session-cookie signing key) and PROVISIONER_API_KEY (the key the gateway presents to the sandbox provisioner), each generated once and preserved across upgrades via lookup. existingAppSecret points the gateway, frontend and provisioner at a Secret you manage instead (no <release>-app is generated); it must carry the first three keys and, while provisioner.enabled is true, also PROVISIONER_API_KEY: the provisioner's verify_api_key middleware answers 401 to every /api/* request while its key is empty or differs from the one the gateway sends, so a deployment without it cannot create a single sandbox. The chart injects the same Secret value into both Pods and the default config references it as sandbox.provisioner_api_key: $PROVISIONER_API_KEY (keep that line when you override config:; the harness refuses to start when a referenced variable is unset). With provisioner.enabled: false the gateway takes PROVISIONER_API_KEY from secrets or existingSecret when you supply one, so an external provisioner keeps its key (the chart emits no env entry, which would win over envFrom); otherwise the gateway's start command defaults the variable to an empty string, so the default config still loads even when a user-managed provider Secret holds only model keys. AUTH_JWT_SECRET is required whenever the gateway is multi-instance — without it, concurrently booting Pods race to write their own .jwt_secret on the home volume and sign sessions with different keys — and only a single Pod may omit it and fall back to that file. Upgrading a release that predates AUTH_JWT_SECRET generates a new key, so every existing browser session is signed out once. To keep sessions, copy the key the gateway has been using into the Secret before upgrading; lookup then preserves it like the other app secrets:

    JWT=$(kubectl -n deer-flow exec deploy/deer-flow-gateway -- cat /app/backend/.deer-flow/.jwt_secret)
    kubectl -n deer-flow patch secret deer-flow-app -p "{\"stringData\":{\"AUTH_JWT_SECRET\":\"$JWT\"}}"
    
  • Credentials key. The app Secret also holds DEER_FLOW_CREDENTIALS_KEY, the Fernet key the gateway encrypts stored credentials with (per-connection IM channel credentials today). The chart generates it once (32 random bytes, urlsafe base64) and preserves it across upgrades via lookup; every gateway Pod reads the same value. Back it up: deleting the Secret or the release generates a new key, and credentials stored under the old one become unreadable (they are treated as missing, never as an error). To rotate, prepend a new key — new,old — and keep the old one listed until stored values have been rewritten. An existingAppSecret should carry it; the env entry is optional, so without it a single Pod falls back to an auto-generated .credentials_key on the home volume, while a multi-instance gateway with channel_connections.enabled refuses to start. If you add the key to an existingAppSecret later, copy that file's value so stored credentials stay readable:

    KEY=$(kubectl -n deer-flow exec deploy/deer-flow-gateway -- cat /app/backend/.deer-flow/.credentials_key)
    kubectl -n deer-flow patch secret my-app-secret -p "{\"stringData\":{\"DEER_FLOW_CREDENTIALS_KEY\":\"$KEY\"}}"
    
  • Scheduled task recovery. If a deployment explicitly enables scheduler.multi_instance: true, it must use shared Postgres, run_ownership.heartbeat_enabled: true, and run_events.backend: db. Scheduler startup then preserves live scheduled runs owned by another Pod, atomically takes over only expired leases, and fences stale post-launch bookkeeping. max_concurrent_runs is a shared global cap across Pods, including pre-launch dispatch reservations. Restart all Gateway Pods after changing these startup-only settings. IM channels remain single-instance regardless (see Gateway replicas above).

  • Redis stream bridge. A bundled single-instance redis StatefulSet (redis.enabled: true, redis:7-alpine) runs in the namespace and the gateway connects via the in-cluster Service. Per-run SSE events are stored in Redis Streams (PR #3191) so a client connected to any gateway pod receives live events and reconnect resumes from Last-Event-ID. The URL is auto-generated into a Secret (key redis-url) and injected as DEER_FLOW_STREAM_BRIDGE_REDIS_URL; config.yaml sets stream_bridge.type: redis by default. No-auth by default (ClusterIP isolation, matching compose); set redis.auth.password to enable AUTH. For a managed Redis, disable the bundled instance and point at it via redis.external. The gateway readiness probe pings this Redis on every check and reports the pod unready while it is unreachable, since without the bridge no run can publish, stream or be cancelled from a peer pod.

  • Persistence. A PVC (<release>-home) backs /app/backend/.deer-flow (sqlite DB, memory, custom agents, per-thread user-data). The gateway mounts it with subPath: deer-flow so the layout matches the provisioner's PVC user-data mode. Default ReadWriteOnce; use ReadWriteMany (NFS) on multi-node clusters so sandbox Pods on other nodes can mount it.

  • Provisioner RBAC. The provisioner gets a ServiceAccount with a namespaced Role (get/list/watch/create/delete on pods + services) and a narrow ClusterRole (namespace get/create). It uses in-cluster service-account creds — no kubeconfig mount. The unused update/patch/pods-exec/events verbs were dropped (audited against docker/provisioner/app.py).

  • Skills. Disabled by default (emptyDir at /app/skills). Populate via skills.existingClaim or skills.configMap, or bake skills into a custom gateway image.

Security

Enforced posture

All workloads run as non-root with all Linux capabilities dropped. No container escalates privileges or runs as uid 0.

workload runAsUser fsGroup writable-path handling
gateway 1000 1000 .deer-flow PVC group-writable via fsGroup; PYTHONDONTWRITEBYTECODE=1 suppresses .pyc writes; UV_CACHE_DIR=/tmp
frontend 1000 (node) 1000 emptyDir at /app/frontend/.next/cache (root-owned in the image)
nginx 101 (nginx) 101 command writes the rendered config to /tmp/nginx.conf and loads nginx -c /tmp/nginx.conf (since /etc/nginx is root-owned); emptyDir at /var/cache/nginx
provisioner 1000 — no PVC; PYTHONDONTWRITEBYTECODE=1
postgres 999 (postgres) 999 official postgres:16 entrypoint detects non-root and skips the chown/gosu dance; data PVC group-writable via fsGroup
redis 999 (redis) 999 official redis:7-alpine entrypoint detects non-root and skips the gosu dance; data PVC group-writable via fsGroup

Every container sets:

  • runAsNonRoot: true
  • allowPrivilegeEscalation: false
  • capabilities.drop: ["ALL"]
  • seccompProfile: { type: RuntimeDefault }

All listening ports are >1024 (8001 / 3000 / 2026 / 8002 / 5432), so no NET_BIND_SERVICE capability is required.

ConfigMap rollout. ConfigMaps mount via subPath, which does not receive in-place updates — a helm upgrade that changes only a ConfigMap would leave pods on stale config. Each pod template carries a checksum/* annotation (SHA256 of the rendered ConfigMap): checksum/config + checksum/extensions on the gateway, checksum/nginx on nginx. Any content change alters the pod spec and triggers a rolling restart.

Resource defaults. Every workload ships with modest requests+limits in values.yaml; override per workload (gateway.resources, frontend.resources, nginx.resources, provisioner.resources, postgresql.primary.resources, redis.primary.resources).

Not yet enforced (deferred hardening)

These are intentionally not set in this chart revision. Each can be added per-workload with testing:

  • readOnlyRootFilesystem: true — makes the container's root filesystem immutable so a compromised process can't persist changes to the image. Not enabled because it requires auditing every runtime write path and mounting an emptyDir over each. Known paths:
    • gateway / frontend / nginx / provisioner: /tmp (uv cache, python tempfiles, the nginx config + pid, node temp) — one emptyDir at /tmp each.
    • postgres: /tmp and /var/run/postgresql (the Unix-socket dir). The first four are mechanical. postgres is the hard case — the official image writes its socket to /var/run/postgresql and isn't designed for a read-only root, so it may need socket-path redirection (PGHOST/unix_socket_directories). Optionally, add USER directives to the backend/Dockerfile, frontend/Dockerfile, and docker/provisioner/Dockerfile so the images are non-root by default (defense in depth — the chart already forces the uid via securityContext, so this is not required). A cluster enforcing the restricted Pod Security Admission standard would require this setting.
  • Provisioner RBAC narrowing. The Role grants get/list/watch/create/delete on pods and services in the namespace (update/patch/pods-exec/events were dropped as unused). These verbs still apply to all Pods in the namespace, not just sandbox Pods — RBAC can't scope by label, so the remaining options are a dedicated sandbox namespace or admission control (OPA/Kyverno).
  • startupProbe. Workloads have readiness + liveness probes but no startup probe. The gateway's livenessProbe.initialDelaySeconds: 30 covers slow starts today; a startupProbe would let it take arbitrarily long to initialize without risking a liveness kill during a cold start (e.g. slow model config load).

None of these affect correctness of the current deployment.

Migrating an existing volume to non-root

fsGroup does not apply to subPath mounts, and it changes group ownership but not file mode — so a PVC written by an earlier root run (e.g. a cluster that ran the gateway as root before enabling this hardening, or a backup restore of root-owned files) will keep files like .jwt_secret at 0600 root:root. The non-root gateway (uid 1000) then can't read them and crashes on the first auth request with RuntimeError: Failed to read or persist the JWT secret at .../​.jwt_secret.

Fresh installs are unaffected — uid 1000 creates every file as 1000:1000.

To fix an existing root-written PVC, run a one-shot root pod that chowns the volume to the gateway uid (1000), then restart the gateway:

cat <<'EOF' | kubectl apply -n deer-flow -f -
apiVersion: v1
kind: Pod
metadata: { name: fix-home-perms, namespace: deer-flow }
spec:
  restartPolicy: Never
  containers:
    - name: chown
      image: busybox:1.36
      command: ["sh", "-c"]
      args: ["chown -R 1000:1000 /home-pvc/deer-flow && chmod -R g+rwX /home-pvc/deer-flow"]
      volumeMounts:
        - { name: home, mountPath: /home-pvc }
  volumes:
    - name: home
      persistentVolumeClaim: { claimName: deer-flow-deer-flow-home }
EOF
kubectl -n deer-flow wait --for=condition=Ready pod/fix-home-perms --timeout=30s
kubectl -n deer-flow delete pod fix-home-perms
kubectl -n deer-flow rollout restart deploy/deer-flow-deer-flow-gateway

(On a single-node cluster the fix pod can mount the RWO PVC concurrently with the gateway; on multi-node, scale the gateway to 0 first.) A durable alternative — an opt-in root volumePermissions initContainer that chowns on every start (the Bitnami pattern) — is not yet wired into this chart; it would introduce a root container, so it's left as an operator decision for now.

Sandbox Service type

The provisioner exposes each sandbox Pod behind a per-sandbox Service whose type is controlled by provisioner.sandboxServiceType (default ClusterIP).

ClusterIP (default, recommended). The provisioner returns a cluster-DNS URL - http://sandbox-<id>-svc.<namespace>.svc.cluster.local:8080 - so the gateway reaches its sandbox entirely inside the cluster network. No node IP, no high port, and the code-execution surface is not bound on every node's interfaces. This is correct for the chart, where the gateway always runs in-cluster.

NodePort (Docker-Compose/hybrid escape hatch). Set provisioner.sandboxServiceType: NodePort only when the gateway is not in K8s (e.g. the compose dev path, where the gateway is a container reaching sandbox Pods on the host's Docker Desktop K8s). The provisioner then returns http://{NODE_HOST}:{NodePort}. NODE_HOST defaults to the provisioner pod's node IP via the downward API (status.hostIP); because a NodePort is exposed on every node, the gateway can reach <node-IP>:<NodePort> on most clusters without configuration. Override provisioner.nodeHost only if your CNI or network policy blocks pod->node-IP traffic:

kubectl get nodes -o wide    # use INTERNAL-IP or EXTERNAL-IP
provisioner:
  sandboxServiceType: NodePort
  nodeHost: 192.168.x.x

On multi-node clusters, also switch persistence.home.accessMode to ReadWriteMany (this is orthogonal to the Service type - it governs whether a sandbox Pod can be scheduled on a node other than the gateway's).

Sandbox lark-cli runtime (optional)

The Lark/Feishu lark-cli integration needs a lark-cli binary inside the sandbox. For remote/Kubernetes (provisioner) deployments the sandbox-side path comes from an optional runtime image instead of an install-time GitHub download. The chart exposes the same two knobs the Compose stack reads on the provisioner:

provisioner:
  # Pattern A - an init container copies the binaries into a shared emptyDir.
  larkCliInitImage: deer-flow/lark-cli-init:v1.0.65
  # Pattern B - a shim init container + broker sidecar owns the credentials, so
  # the plaintext config/data dirs are never mounted into the sandbox.
  # Supersedes larkCliInitImage when both are set.
  larkCliBrokerImage: deer-flow/lark-cli-broker:v1.0.65

Both default to empty, which leaves the feature off (legacy behavior) and makes an in-sandbox lark-cli call fail with exit 127 (command not found). When set, they render LARK_CLI_INIT_IMAGE / LARK_CLI_BROKER_IMAGE on the provisioner Deployment - the names docker/docker-compose.yaml uses - and the variable is omitted entirely while empty. Point them at a tag that exists in a registry your nodes can pull from (mirror the registry prefix if you do not use Docker Hub). The images are built from docker/lark-cli-init and docker/lark-cli-broker; see those READMEs and the root README's Lark section for the build/publish flow and the credential model. Broker mode is the safer choice on a shared cluster: the app secret and OAuth tokens stay in the sidecar instead of the sandbox container.

An image here does not authenticate anyone by itself. The Gateway only asks the provisioner to attach the runtime once the Lark integration pack is installed for the user, so the sandbox gets the binary but the per-user credentials still follow the normal install/authorize flow. The Lark integration status reports sandbox_runtime_mode / sandbox_runtime_ready so the Settings UI surfaces a missing runtime instead of a later command not found.

Lint / dry-run

helm lint deploy/helm/deer-flow
helm template deer-flow deploy/helm/deer-flow -n deer-flow -f my-values.yaml | \
  kubectl apply --dry-run=client -f -

Uninstall

helm uninstall deer-flow -n deer-flow
# the PVC is NOT deleted by default — remove it manually if desired:
kubectl -n deer-flow delete pvc -l app.kubernetes.io/instance=deer-flow