* feat(memory): template externalization — externalized prompts + signal patterns Externalize two categories of hardcoded content in the DeerMem memory backend so operators can customise them without touching Python code. Prompt externalization (plugin 06): - 4 memory extraction prompts moved from Python string constants to YAML files under core/prompts/ (consolidation / fact_extraction / memory_update.chat / staleness_review). - load_prompt(name) and load_prompt_messages(name, variables) loaders. - memory_update uses the chat format (system / user message split) for prompt-caching-friendly system prefixes; other prompts stay text. - The extraction callback + per-agent prompt directories + Jinja2 dependency are intentionally not included (minimal surface). - Compatible with the upstream prompt additions from #4143 (expected_valid_days, staleFactsToExtend, KEEP/REMOVE/EXTEND in staleness review). Signal-pattern externalization (plugin 07, regex only): - Correction and reinforcement detection patterns externalised to core/message_patterns/{correction,reinforcement}.yaml. - load_patterns(name, patterns_dir) loader with bundled defaults. - detect_correction / detect_reinforcement accept a keyword-only patterns= parameter; signatures are backward-compatible. - patterns_dir config field added to DeerMemConfig. - Importance scorer (importance.py, build_importance_scorer, _prepare_update changes) is NOT included — deferred to a later PR. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): fail loudly on invalid yaml in prompt/pattern loaders Replace silent error handling in load_prompt / load_prompt_messages / load_patterns so malformed yaml or missing required keys raise ValueError with the file path rather than a raw YAMLError traceback or silent empty-string / empty-list return. Changes: - load_prompt: YAMLError -> ValueError(path); missing/empty 'template' key -> ValueError - load_prompt_messages: YAMLError -> ValueError(path); missing/empty 'messages' key -> ValueError; KeyError from .format (placeholder mismatch) -> ValueError - load_patterns: YAMLError -> ValueError (was silent warn+return []). OSError still degrades (permission, not format). Not-a-list yaml -> ValueError (was silent warn+return []). Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): wire prompts_dir through updater, fail-loud patterns_dir, bump config_version Addresses PR feedback on the template-externalization PR: P1 — Wire prompts_dir into the DeerMem update path: - Add prompts_dir field to DeerMemConfig (default None = bundled defaults). - Thread it through DeerMem -> MemoryUpdater.__init__(prompts_dir=). - _build_staleness_section / _build_consolidation_section accept prompts_dir= keyword and call load_prompt() instead of relying on module-level shim constants. - _prepare_update_prompt passes prompts_dir to load_prompt_messages and to both section builders. - Add _PROMPT_CACHE to load_prompt so repeated lookups (once per memory-update cycle) do not re-read yaml from disk. P2 — Explicit patterns_dir must find its files: - When patterns_dir is explicitly set and a YAML file is missing, load_patterns() raises FileNotFoundError instead of silently caching an empty list. Bundled defaults (patterns_dir=None) still log a WARNING and return [] for a missing bundled file (packaging bug). P3 — Bump config_version and sync Helm: - config.example.yaml: 26 -> 27 - deploy/helm/deer-flow/values.yaml: 26 -> 27 - deploy/helm/deer-flow/README.md: 26 -> 27 - scripts/check_config_version.sh confirms parity. Tests: 224 passed (218 memory + 6 config_version). Lint: ruff check + ruff format --check pass. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): cache load_prompt_messages, validate format fields, fail-loud load_patterns Review feedback from willem-bd: Cache for load_prompt_messages: Add _CHAT_TEMPLATE_CACHE + _render_messages() helper so the parsed chat templates are cached per (name, agent, prompts_dir). On cache hit only .format() rendering runs; the yaml file is read once per key. Validate format field in both loaders: load_prompt rejects format='chat' (redirects to load_prompt_messages); load_prompt_messages rejects format='text' (redirects to load_prompt). Prevents operators from loading a chat yaml via the text loader (or vice versa) without a clear error. Load_patterns fail-loud for explicit directories: - OSError (permission, etc.) now raises for explicit patterns_dir instead of silently disabling detection. - Malformed entries (missing/empty pattern key, wrong type) are skipped with a WARNING instead of silent skip. - Unknown flag names are warned instead of silently dropped. - re.error from compile() raises ValueError with file path and entry index instead of a bare re.error traceback. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): thread agent_name through section builders, validate explicit prompts at construction, bump config_version to 28 P1 — Thread agent_name through staleness/consolidation section builders: _build_staleness_section and _build_consolidation_section now accept agent_name= and pass it to load_prompt(), so per-agent prompt overrides work for section templates too (not just memory_update). Previously only prompts_dir was threaded; agent_name was ignored for section builders. P1 — Validate explicit prompts at DeerMem construction: When DeerMemConfig.prompts_dir is explicitly set, DeerMem.__init__ now pre-loads all four prompt templates (staleness_review, consolidation, fact_extraction, memory_update with dummy variables) at construction time. A missing file, malformed YAML, or invalid placeholder raises immediately at startup instead of being caught by the updater's generic error handler and silently dropped as a failed update. P2 — Advance config_version to 28: Upstream main already consumed the previous 26→27 bump with a different schema change. Bump config.example.yaml, Helm values.yaml, and Helm README.md to 28 to version this PR's patterns_dir and prompts_dir additions. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): render text templates at construction, propagate PromptConfigurationError Keep prompt-configuration failures out of the recoverable update catch: - Define PromptConfigurationError(ValueError) in prompt.py. Raised by load_prompt, load_prompt_messages, and _render_messages for bad yaml, missing keys, and invalid placeholders. - _do_update_memory_sync_impl, update_memory's executor path, and _process_queue all re-raise (PromptConfigurationError, FileNotFoundError, OSError) before the generic except Exception, so a bad explicit prompt is never silently returned as False. - DeerMem.__init__ prompt validation now renders text templates with dummy variables (.format(stale_facts="") etc.) so an unknown placeholder in staleness_review.yaml, consolidation.yaml, or fact_extraction.yaml is caught at construction. Co-Authored-By: Claude <noreply@anthropic.com> * fix(memory): drop re-raise guards, soften validation comment, exclude dormant fact_extraction - Remove (PromptConfigurationError, FileNotFoundError, OSError) re-raise guards from _do_update_memory_sync_impl, update_memory executor path, and _process_queue. The existing except Exception: logger.exception(...) already logs prompt-config errors at ERROR with full traceback; the re-raise was killing the entire batch and only surfacing via stderr. Per-agent override errors now log at ERROR per-item without aborting co-tenant updates. - Soften the construction validation comment to clarify it only covers global templates. Per-agent overrides are validated lazily at first use and logged at ERROR by the updater's exception handler. - Drop fact_extraction from construction validation (dormant prompt with no runtime caller; the shim + yaml remain for backward compat). Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
DeerFlow Helm Chart
Deploys the full DeerFlow stack to Kubernetes: gateway (backend + embedded LangGraph runtime), frontend (Next.js), nginx (internal reverse proxy preserving the compose routing), and the provisioner (K8s-native sandbox that spawns code-execution Pods on demand).
This chart translates the production docker/docker-compose.yaml into native
Kubernetes resources. No existing repo files are modified.
Prerequisites
- A Kubernetes cluster (Docker Desktop K8s, OrbStack, kind, k3d, or a real cluster).
kubectl+helm3.8+ installed (OCI registry support stabilized in 3.8; earlier 3.x needsHELM_EXPERIMENTAL_OCI=1).- The three DeerFlow images — either the published ones (see "Install the published chart" below) or built locally (see step 1).
- An Ingress controller (e.g. ingress-nginx) if you enable
ingress.
Install the published chart (GHCR)
The chart and all three images are published to GHCR on every v* release tag
(see .github/workflows/container.yaml and chart.yaml). Skip the build step
and install directly:
helm install deer-flow oci://ghcr.io/<owner>/charts/deer-flow \
--version <version> \
-n deer-flow --create-namespace \
-f my-values.yaml
where <owner> is the GitHub owner the chart is published from and <version>
matches the release tag without the leading v (tag v0.1.0 → --version 0.1.0).
Note: the helm chart is new in 2.1.0 - no chart was published before it. It publishes to
oci://ghcr.io/<owner>/charts/deer-flow(thecharts/prefix keeps it distinct from thedeer-flow-{backend,frontend,provisioner}image packages).
Point the chart at the published images:
image:
registry: ghcr.io/<owner> # owner prefix; images are <owner>/deer-flow-<name>
tag: "<version>" # match the release tag (sans leading `v`)
pullSecrets:
- { name: regcred } # only if the GHCR package is private
The chart's gatewayImage / frontendImage / provisionerImage defaults
already match the published image names (deer-flow-backend,
deer-flow-frontend, deer-flow-provisioner), so only registry and tag
are required. New GHCR packages default to private — flip the package to
public in its GHCR settings page for unauthenticated pulls, otherwise create a
pull secret (step 1) and reference it via image.pullSecrets.
The OCI chart and the images are versioned independently of the chart's
appVersion; always setimage.tagto the release that matches your chart--versionunless you have a reason to pin differently.
1. Build & push images (custom builds only)
Skip this section if you're using the published chart above. To build the images yourself from the existing Dockerfiles:
REGISTRY=ghcr.io/yourorg
TAG=latest
# backend - build with the `postgres` extra so multi-replica deploys can use
# shared Postgres (matches the published image)
docker build -t $REGISTRY/deer-flow-backend:$TAG --build-arg UV_EXTRAS=postgres -f backend/Dockerfile .
# frontend
docker build -t $REGISTRY/deer-flow-frontend:$TAG -f frontend/Dockerfile .
# provisioner
docker build -t $REGISTRY/deer-flow-provisioner:$TAG -f docker/provisioner/Dockerfile docker/provisioner
docker push $REGISTRY/deer-flow-backend:$TAG
docker push $REGISTRY/deer-flow-frontend:$TAG
docker push $REGISTRY/deer-flow-provisioner:$TAG
These names match the chart's gatewayImage / frontendImage /
provisionerImage defaults, so only image.registry and image.tag need to
point at them.
If your registry needs auth, create a pull secret:
kubectl create secret docker-registry regcred \
--docker-server=ghcr.io \
--docker-username=youruser \
--docker-password=yourtoken \
-n deer-flow
2. Configure values
Copy and edit values.yaml → my-values.yaml. At minimum set:
image:
registry: ghcr.io/yourorg
tag: latest
pullSecrets:
- { name: regcred }
ingress:
enabled: true
className: nginx
host: deer-flow.example.com
tls:
enabled: true
secretName: deer-flow-tls
secrets:
OPENAI_API_KEY: sk-...
# add channel tokens, search keys, etc. as needed
Provide your model config under config (keep secrets as $VAR references —
they resolve from the secrets map):
config: |
config_version: 28
models:
- name: gpt-4
use: langchain_openai:ChatOpenAI
model: gpt-4
api_key: $OPENAI_API_KEY
request_timeout: 600.0
sandbox:
use: deerflow.community.aio_sandbox:AioSandboxProvider
provisioner_url: http://provisioner:8002
database:
backend: postgres
postgres_url: $DATABASE_URL
checkpointer:
type: postgres
connection_string: $DATABASE_URL
stream_bridge:
type: redis # cross-pod SSE; URL from DEER_FLOW_STREAM_BRIDGE_REDIS_URL
# Tools MUST be listed explicitly - the agent gets none otherwise
# (BUILTIN_TOOLS only adds present_file + ask_clarification). The chart
# default in values.yaml enables the sandbox tools + web tools (web_search,
# web_fetch, image_search - no API key); when you override `config:`, copy
# them in. Full list in values.yaml / config.example.yaml. The web tools need
# outbound egress from the gateway pod.
tool_groups:
- name: web
- name: file:read
- name: file:write
- name: bash
tools:
- name: web_search
group: web
use: deerflow.community.ddg_search.tools:web_search_tool
max_results: 5
- name: web_fetch
group: web
use: deerflow.community.jina_ai.tools:web_fetch_tool
timeout: 10
- name: image_search
group: web
use: deerflow.community.image_search.tools:image_search_tool
max_results: 5
- name: bash
group: bash
use: deerflow.sandbox.tools:bash_tool
# also: ls, read_file, glob, grep, write_file, str_replace (see values.yaml)
$DATABASE_URL is injected from the postgres Secret (see below). The
checkpointer: section is required for multi-replica operation — the LangGraph
Store (cross-thread memory + thread list) reads it and does not fall back to
database:. stream_bridge.type: redis is the default and routes live SSE
events through the bundled redis StatefulSet (or redis.external).
Because config: is a single override blob, a partial config: replaces the
chart default entirely - keep the tools:/tool_groups: block (or the agent
will have no tools) and the sandbox:/database:/checkpointer:/stream_bridge:
sections shown above.
3. Install (from a local chart checkout)
For a custom build or local development, install from the chart directory:
helm install deer-flow deploy/helm/deer-flow \
-n deer-flow --create-namespace \
-f my-values.yaml
4. Verify
kubectl -n deer-flow get pods
kubectl -n deer-flow port-forward svc/nginx 2026:2026
curl http://localhost:2026/health # gateway health via nginx
Hit the Ingress host (map it in /etc/hosts for local clusters) to load the UI.
Provisioner sanity check:
kubectl -n deer-flow exec deploy/deer-flow-provisioner -- curl -s localhost:8002/health
Architecture notes
- PostgreSQL is the default database. A bundled single-instance postgres
StatefulSet (
postgresql.enabled: true) runs in the namespace and the gateway connects via the in-cluster Service. The DSN is auto-generated into a Secret (keydatabase-url) and injected asDATABASE_URL;config.yamlreferences it as$DATABASE_URLindatabase.postgres_url. Schema is bootstrapped automatically on gateway startup (alembiccreate_all+stamp head). For real HA, disable the bundled instance and point at a managed DB:postgresql: enabled: false external: host: mydb.example.com # or set databaseUrl / existingSecret port: 5432 database: deerflow username: deerflow password: changeme - Graceful shutdown & memory drain. The gateway pod sets
terminationGracePeriodSeconds(default 45s, overridable viagateway.terminationGracePeriodSeconds) plus an optionalpreStopsleep (gateway.preStopSleepSeconds, default 5s). The grace period MUST exceed the Gateway's graceful-shutdown work — channel stop (~5s) plus the memory-queue drain (memory.shutdown_flush_timeout_seconds, default 30s) plus a buffer — because the drain runs on a daemon thread and K8s SIGKILLs anything still running at the end of the grace window. K8s defaults to 30s, which SIGKILLs the drain mid-flight and silently re-introduces the memory loss the drain is fixing. When you raisememory.shutdown_flush_timeout_seconds, raisegateway.terminationGracePeriodSecondsto match (channel stop + drain + buffer). - Gateway replicas. Postgres + the Redis stream bridge together make the
gateway's persisted state (checkpointer + run/thread metadata) and live
stream path cross-pod-safe. The default is still 1 replica: do not raise
gateway.replicaspast 1 yet. Run control —create_or_rejectdedup,cancel, and orphan reconciliation — is still worker-local (in-processasyncio.Lock+ in-memoryrecord.task), tracked by issue #3948. With >1 replica a double-submit can create two runs on one thread (checkpoint corruption), a cancel can land on a non-owner pod (409), and a crashed pod's runs staypending/runningforever. Stay on 1 replica until that work lands. - Redis stream bridge. A bundled single-instance redis StatefulSet
(
redis.enabled: true,redis:7-alpine) runs in the namespace and the gateway connects via the in-cluster Service. Per-run SSE events are stored in Redis Streams (PR #3191) so a client connected to any gateway pod receives live events and reconnect resumes fromLast-Event-ID. The URL is auto-generated into a Secret (keyredis-url) and injected asDEER_FLOW_STREAM_BRIDGE_REDIS_URL;config.yamlsetsstream_bridge.type: redisby default. No-auth by default (ClusterIP isolation, matching compose); setredis.auth.passwordto enable AUTH. For a managed Redis, disable the bundled instance and point at it viaredis.external. - Persistence. A PVC (
<release>-home) backs/app/backend/.deer-flow(sqlite DB, memory, custom agents, per-thread user-data). The gateway mounts it withsubPath: deer-flowso the layout matches the provisioner's PVC user-data mode. DefaultReadWriteOnce; useReadWriteMany(NFS) on multi-node clusters so sandbox Pods on other nodes can mount it. - Provisioner RBAC. The provisioner gets a ServiceAccount with a namespaced
Role (get/list/watch/create/delete on pods + services) and a narrow ClusterRole
(namespace get/create). It uses in-cluster service-account creds — no
kubeconfig mount. The unused update/patch/pods-exec/events verbs were dropped
(audited against
docker/provisioner/app.py). - Skills. Disabled by default (emptyDir at
/app/skills). Populate viaskills.existingClaimorskills.configMap, or bake skills into a custom gateway image.
Security
Enforced posture
All workloads run as non-root with all Linux capabilities dropped. No container escalates privileges or runs as uid 0.
| workload | runAsUser | fsGroup | writable-path handling |
|---|---|---|---|
| gateway | 1000 | 1000 | .deer-flow PVC group-writable via fsGroup; PYTHONDONTWRITEBYTECODE=1 suppresses .pyc writes; UV_CACHE_DIR=/tmp |
| frontend | 1000 (node) |
1000 | emptyDir at /app/frontend/.next/cache (root-owned in the image) |
| nginx | 101 (nginx) |
101 | command writes the rendered config to /tmp/nginx.conf and loads nginx -c /tmp/nginx.conf (since /etc/nginx is root-owned); emptyDir at /var/cache/nginx |
| provisioner | 1000 | — | no PVC; PYTHONDONTWRITEBYTECODE=1 |
| postgres | 999 (postgres) |
999 | official postgres:16 entrypoint detects non-root and skips the chown/gosu dance; data PVC group-writable via fsGroup |
| redis | 999 (redis) |
999 | official redis:7-alpine entrypoint detects non-root and skips the gosu dance; data PVC group-writable via fsGroup |
Every container sets:
runAsNonRoot: trueallowPrivilegeEscalation: falsecapabilities.drop: ["ALL"]seccompProfile: { type: RuntimeDefault }
All listening ports are >1024 (8001 / 3000 / 2026 / 8002 / 5432), so no
NET_BIND_SERVICE capability is required.
ConfigMap rollout. ConfigMaps mount via subPath, which does not receive
in-place updates — a helm upgrade that changes only a ConfigMap would leave
pods on stale config. Each pod template carries a checksum/* annotation (SHA256
of the rendered ConfigMap): checksum/config + checksum/extensions on the
gateway, checksum/nginx on nginx. Any content change alters the pod spec and
triggers a rolling restart.
Resource defaults. Every workload ships with modest requests+limits in
values.yaml; override per workload (gateway.resources, frontend.resources,
nginx.resources, provisioner.resources, postgresql.primary.resources,
redis.primary.resources).
Not yet enforced (deferred hardening)
These are intentionally not set in this chart revision. Each can be added per-workload with testing:
readOnlyRootFilesystem: true— makes the container's root filesystem immutable so a compromised process can't persist changes to the image. Not enabled because it requires auditing every runtime write path and mounting anemptyDirover each. Known paths:- gateway / frontend / nginx / provisioner:
/tmp(uv cache, python tempfiles, the nginx config + pid, node temp) — oneemptyDirat/tmpeach. - postgres:
/tmpand/var/run/postgresql(the Unix-socket dir). The first four are mechanical. postgres is the hard case — the official image writes its socket to/var/run/postgresqland isn't designed for a read-only root, so it may need socket-path redirection (PGHOST/unix_socket_directories). Optionally, addUSERdirectives to thebackend/Dockerfile,frontend/Dockerfile, anddocker/provisioner/Dockerfileso the images are non-root by default (defense in depth — the chart already forces the uid viasecurityContext, so this is not required). A cluster enforcing therestrictedPod Security Admission standard would require this setting.
- gateway / frontend / nginx / provisioner:
- Provisioner RBAC narrowing. The Role grants get/list/watch/create/delete on pods and services in the namespace (update/patch/pods-exec/events were dropped as unused). These verbs still apply to all Pods in the namespace, not just sandbox Pods — RBAC can't scope by label, so the remaining options are a dedicated sandbox namespace or admission control (OPA/Kyverno).
startupProbe. Workloads have readiness + liveness probes but no startup probe. The gateway'slivenessProbe.initialDelaySeconds: 30covers slow starts today; astartupProbewould let it take arbitrarily long to initialize without risking a liveness kill during a cold start (e.g. slow model config load).
None of these affect correctness of the current deployment.
Migrating an existing volume to non-root
fsGroup does not apply to subPath mounts, and it changes group ownership
but not file mode — so a PVC written by an earlier root run (e.g. a cluster
that ran the gateway as root before enabling this hardening, or a backup restore
of root-owned files) will keep files like .jwt_secret at 0600 root:root. The
non-root gateway (uid 1000) then can't read them and crashes on the first auth
request with RuntimeError: Failed to read JWT secret from .../.jwt_secret.
Fresh installs are unaffected — uid 1000 creates every file as 1000:1000.
To fix an existing root-written PVC, run a one-shot root pod that chowns the volume to the gateway uid (1000), then restart the gateway:
cat <<'EOF' | kubectl apply -n deer-flow -f -
apiVersion: v1
kind: Pod
metadata: { name: fix-home-perms, namespace: deer-flow }
spec:
restartPolicy: Never
containers:
- name: chown
image: busybox:1.36
command: ["sh", "-c"]
args: ["chown -R 1000:1000 /home-pvc/deer-flow && chmod -R g+rwX /home-pvc/deer-flow"]
volumeMounts:
- { name: home, mountPath: /home-pvc }
volumes:
- name: home
persistentVolumeClaim: { claimName: deer-flow-deer-flow-home }
EOF
kubectl -n deer-flow wait --for=condition=Ready pod/fix-home-perms --timeout=30s
kubectl -n deer-flow delete pod fix-home-perms
kubectl -n deer-flow rollout restart deploy/deer-flow-deer-flow-gateway
(On a single-node cluster the fix pod can mount the RWO PVC concurrently with the
gateway; on multi-node, scale the gateway to 0 first.) A durable alternative —
an opt-in root volumePermissions initContainer that chowns on every start (the
Bitnami pattern) — is not yet wired into this chart; it would introduce a root
container, so it's left as an operator decision for now.
Sandbox Service type
The provisioner exposes each sandbox Pod behind a per-sandbox Service whose type
is controlled by provisioner.sandboxServiceType (default ClusterIP).
ClusterIP (default, recommended). The provisioner returns a cluster-DNS
URL - http://sandbox-<id>-svc.<namespace>.svc.cluster.local:8080 - so the
gateway reaches its sandbox entirely inside the cluster network. No node IP, no
high port, and the code-execution surface is not bound on every node's
interfaces. This is correct for the chart, where the gateway always runs
in-cluster.
NodePort (Docker-Compose/hybrid escape hatch). Set
provisioner.sandboxServiceType: NodePort only when the gateway is not in K8s
(e.g. the compose dev path, where the gateway is a container reaching sandbox
Pods on the host's Docker Desktop K8s). The provisioner then returns
http://{NODE_HOST}:{NodePort}. NODE_HOST defaults to the provisioner pod's
node IP via the downward API
(status.hostIP); because a NodePort is exposed on every node, the gateway can
reach <node-IP>:<NodePort> on most clusters without configuration. Override
provisioner.nodeHost only if your CNI or network policy blocks pod->node-IP
traffic:
kubectl get nodes -o wide # use INTERNAL-IP or EXTERNAL-IP
provisioner:
sandboxServiceType: NodePort
nodeHost: 192.168.x.x
On multi-node clusters, also switch persistence.home.accessMode to
ReadWriteMany (this is orthogonal to the Service type - it governs whether a
sandbox Pod can be scheduled on a node other than the gateway's).
Lint / dry-run
helm lint deploy/helm/deer-flow
helm template deer-flow deploy/helm/deer-flow -n deer-flow -f my-values.yaml | \
kubectl apply --dry-run=client -f -
Uninstall
helm uninstall deer-flow -n deer-flow
# the PVC is NOT deleted by default — remove it manually if desired:
kubectl -n deer-flow delete pvc -l app.kubernetes.io/instance=deer-flow