* feat(memory): template externalization — externalized prompts + signal patterns
Externalize two categories of hardcoded content in the DeerMem memory
backend so operators can customise them without touching Python code.
Prompt externalization (plugin 06):
- 4 memory extraction prompts moved from Python string constants to YAML
files under core/prompts/ (consolidation / fact_extraction /
memory_update.chat / staleness_review).
- load_prompt(name) and load_prompt_messages(name, variables) loaders.
- memory_update uses the chat format (system / user message split) for
prompt-caching-friendly system prefixes; other prompts stay text.
- The extraction callback + per-agent prompt directories + Jinja2
dependency are intentionally not included (minimal surface).
- Compatible with the upstream prompt additions from #4143
(expected_valid_days, staleFactsToExtend, KEEP/REMOVE/EXTEND in
staleness review).
Signal-pattern externalization (plugin 07, regex only):
- Correction and reinforcement detection patterns externalised to
core/message_patterns/{correction,reinforcement}.yaml.
- load_patterns(name, patterns_dir) loader with bundled defaults.
- detect_correction / detect_reinforcement accept a keyword-only
patterns= parameter; signatures are backward-compatible.
- patterns_dir config field added to DeerMemConfig.
- Importance scorer (importance.py, build_importance_scorer,
_prepare_update changes) is NOT included — deferred to a later PR.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): fail loudly on invalid yaml in prompt/pattern loaders
Replace silent error handling in load_prompt / load_prompt_messages /
load_patterns so malformed yaml or missing required keys raise ValueError
with the file path rather than a raw YAMLError traceback or silent
empty-string / empty-list return.
Changes:
- load_prompt: YAMLError -> ValueError(path); missing/empty 'template'
key -> ValueError
- load_prompt_messages: YAMLError -> ValueError(path); missing/empty
'messages' key -> ValueError; KeyError from .format (placeholder
mismatch) -> ValueError
- load_patterns: YAMLError -> ValueError (was silent warn+return []).
OSError still degrades (permission, not format). Not-a-list yaml
-> ValueError (was silent warn+return []).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): wire prompts_dir through updater, fail-loud patterns_dir, bump config_version
Addresses PR feedback on the template-externalization PR:
P1 — Wire prompts_dir into the DeerMem update path:
- Add prompts_dir field to DeerMemConfig (default None = bundled defaults).
- Thread it through DeerMem -> MemoryUpdater.__init__(prompts_dir=).
- _build_staleness_section / _build_consolidation_section accept
prompts_dir= keyword and call load_prompt() instead of relying on
module-level shim constants.
- _prepare_update_prompt passes prompts_dir to load_prompt_messages
and to both section builders.
- Add _PROMPT_CACHE to load_prompt so repeated lookups (once per
memory-update cycle) do not re-read yaml from disk.
P2 — Explicit patterns_dir must find its files:
- When patterns_dir is explicitly set and a YAML file is missing,
load_patterns() raises FileNotFoundError instead of silently caching
an empty list. Bundled defaults (patterns_dir=None) still log a
WARNING and return [] for a missing bundled file (packaging bug).
P3 — Bump config_version and sync Helm:
- config.example.yaml: 26 -> 27
- deploy/helm/deer-flow/values.yaml: 26 -> 27
- deploy/helm/deer-flow/README.md: 26 -> 27
- scripts/check_config_version.sh confirms parity.
Tests: 224 passed (218 memory + 6 config_version).
Lint: ruff check + ruff format --check pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): cache load_prompt_messages, validate format fields, fail-loud load_patterns
Review feedback from willem-bd:
Cache for load_prompt_messages:
Add _CHAT_TEMPLATE_CACHE + _render_messages() helper so the parsed chat
templates are cached per (name, agent, prompts_dir). On cache hit only
.format() rendering runs; the yaml file is read once per key.
Validate format field in both loaders:
load_prompt rejects format='chat' (redirects to load_prompt_messages);
load_prompt_messages rejects format='text' (redirects to load_prompt).
Prevents operators from loading a chat yaml via the text loader (or vice
versa) without a clear error.
Load_patterns fail-loud for explicit directories:
- OSError (permission, etc.) now raises for explicit patterns_dir
instead of silently disabling detection.
- Malformed entries (missing/empty pattern key, wrong type) are skipped
with a WARNING instead of silent skip.
- Unknown flag names are warned instead of silently dropped.
- re.error from compile() raises ValueError with file path and entry
index instead of a bare re.error traceback.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): thread agent_name through section builders, validate explicit prompts at construction, bump config_version to 28
P1 — Thread agent_name through staleness/consolidation section builders:
_build_staleness_section and _build_consolidation_section now accept
agent_name= and pass it to load_prompt(), so per-agent prompt overrides
work for section templates too (not just memory_update). Previously
only prompts_dir was threaded; agent_name was ignored for section
builders.
P1 — Validate explicit prompts at DeerMem construction:
When DeerMemConfig.prompts_dir is explicitly set, DeerMem.__init__ now
pre-loads all four prompt templates (staleness_review, consolidation,
fact_extraction, memory_update with dummy variables) at construction
time. A missing file, malformed YAML, or invalid placeholder raises
immediately at startup instead of being caught by the updater's
generic error handler and silently dropped as a failed update.
P2 — Advance config_version to 28:
Upstream main already consumed the previous 26→27 bump with a
different schema change. Bump config.example.yaml, Helm values.yaml,
and Helm README.md to 28 to version this PR's patterns_dir and
prompts_dir additions.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): render text templates at construction, propagate PromptConfigurationError
Keep prompt-configuration failures out of the recoverable update catch:
- Define PromptConfigurationError(ValueError) in prompt.py. Raised by
load_prompt, load_prompt_messages, and _render_messages for bad yaml,
missing keys, and invalid placeholders.
- _do_update_memory_sync_impl, update_memory's executor path, and
_process_queue all re-raise (PromptConfigurationError, FileNotFoundError,
OSError) before the generic except Exception, so a bad explicit prompt
is never silently returned as False.
- DeerMem.__init__ prompt validation now renders text templates with
dummy variables (.format(stale_facts="") etc.) so an unknown
placeholder in staleness_review.yaml, consolidation.yaml, or
fact_extraction.yaml is caught at construction.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): drop re-raise guards, soften validation comment, exclude dormant fact_extraction
- Remove (PromptConfigurationError, FileNotFoundError, OSError) re-raise
guards from _do_update_memory_sync_impl, update_memory executor path,
and _process_queue. The existing except Exception: logger.exception(...)
already logs prompt-config errors at ERROR with full traceback; the
re-raise was killing the entire batch and only surfacing via stderr.
Per-agent override errors now log at ERROR per-item without aborting
co-tenant updates.
- Soften the construction validation comment to clarify it only covers
global templates. Per-agent overrides are validated lazily at first use
and logged at ERROR by the updater's exception handler.
- Drop fact_extraction from construction validation (dormant prompt with
no runtime caller; the shim + yaml remain for backward compat).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
de024646a7