lllyfff de024646a7
feat(memory): memory template externalization (#4286)
* feat(memory): template externalization — externalized prompts + signal patterns

Externalize two categories of hardcoded content in the DeerMem memory
backend so operators can customise them without touching Python code.

Prompt externalization (plugin 06):
- 4 memory extraction prompts moved from Python string constants to YAML
  files under core/prompts/ (consolidation / fact_extraction /
  memory_update.chat / staleness_review).
- load_prompt(name) and load_prompt_messages(name, variables) loaders.
- memory_update uses the chat format (system / user message split) for
  prompt-caching-friendly system prefixes; other prompts stay text.
- The extraction callback + per-agent prompt directories + Jinja2
  dependency are intentionally not included (minimal surface).
- Compatible with the upstream prompt additions from #4143
  (expected_valid_days, staleFactsToExtend, KEEP/REMOVE/EXTEND in
  staleness review).

Signal-pattern externalization (plugin 07, regex only):
- Correction and reinforcement detection patterns externalised to
  core/message_patterns/{correction,reinforcement}.yaml.
- load_patterns(name, patterns_dir) loader with bundled defaults.
- detect_correction / detect_reinforcement accept a keyword-only
  patterns= parameter; signatures are backward-compatible.
- patterns_dir config field added to DeerMemConfig.
- Importance scorer (importance.py, build_importance_scorer,
  _prepare_update changes) is NOT included — deferred to a later PR.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): fail loudly on invalid yaml in prompt/pattern loaders

Replace silent error handling in load_prompt / load_prompt_messages /
load_patterns so malformed yaml or missing required keys raise ValueError
with the file path rather than a raw YAMLError traceback or silent
empty-string / empty-list return.

Changes:
- load_prompt: YAMLError -> ValueError(path); missing/empty 'template'
  key -> ValueError
- load_prompt_messages: YAMLError -> ValueError(path); missing/empty
  'messages' key -> ValueError; KeyError from .format (placeholder
  mismatch) -> ValueError
- load_patterns: YAMLError -> ValueError (was silent warn+return []).
  OSError still degrades (permission, not format). Not-a-list yaml
  -> ValueError (was silent warn+return []).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): wire prompts_dir through updater, fail-loud patterns_dir, bump config_version

Addresses PR feedback on the template-externalization PR:

P1 — Wire prompts_dir into the DeerMem update path:
- Add prompts_dir field to DeerMemConfig (default None = bundled defaults).
- Thread it through DeerMem -> MemoryUpdater.__init__(prompts_dir=).
- _build_staleness_section / _build_consolidation_section accept
  prompts_dir= keyword and call load_prompt() instead of relying on
  module-level shim constants.
- _prepare_update_prompt passes prompts_dir to load_prompt_messages
  and to both section builders.
- Add _PROMPT_CACHE to load_prompt so repeated lookups (once per
  memory-update cycle) do not re-read yaml from disk.

P2 — Explicit patterns_dir must find its files:
- When patterns_dir is explicitly set and a YAML file is missing,
  load_patterns() raises FileNotFoundError instead of silently caching
  an empty list. Bundled defaults (patterns_dir=None) still log a
  WARNING and return [] for a missing bundled file (packaging bug).

P3 — Bump config_version and sync Helm:
- config.example.yaml: 26 -> 27
- deploy/helm/deer-flow/values.yaml: 26 -> 27
- deploy/helm/deer-flow/README.md: 26 -> 27
- scripts/check_config_version.sh confirms parity.

Tests: 224 passed (218 memory + 6 config_version).
Lint: ruff check + ruff format --check pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): cache load_prompt_messages, validate format fields, fail-loud load_patterns

Review feedback from willem-bd:

Cache for load_prompt_messages:
  Add _CHAT_TEMPLATE_CACHE + _render_messages() helper so the parsed chat
  templates are cached per (name, agent, prompts_dir). On cache hit only
  .format() rendering runs; the yaml file is read once per key.

Validate format field in both loaders:
  load_prompt rejects format='chat' (redirects to load_prompt_messages);
  load_prompt_messages rejects format='text' (redirects to load_prompt).
  Prevents operators from loading a chat yaml via the text loader (or vice
  versa) without a clear error.

Load_patterns fail-loud for explicit directories:
  - OSError (permission, etc.) now raises for explicit patterns_dir
    instead of silently disabling detection.
  - Malformed entries (missing/empty pattern key, wrong type) are skipped
    with a WARNING instead of silent skip.
  - Unknown flag names are warned instead of silently dropped.
  - re.error from compile() raises ValueError with file path and entry
    index instead of a bare re.error traceback.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): thread agent_name through section builders, validate explicit prompts at construction, bump config_version to 28

P1 — Thread agent_name through staleness/consolidation section builders:
  _build_staleness_section and _build_consolidation_section now accept
  agent_name= and pass it to load_prompt(), so per-agent prompt overrides
  work for section templates too (not just memory_update). Previously
  only prompts_dir was threaded; agent_name was ignored for section
  builders.

P1 — Validate explicit prompts at DeerMem construction:
  When DeerMemConfig.prompts_dir is explicitly set, DeerMem.__init__ now
  pre-loads all four prompt templates (staleness_review, consolidation,
  fact_extraction, memory_update with dummy variables) at construction
  time. A missing file, malformed YAML, or invalid placeholder raises
  immediately at startup instead of being caught by the updater's
  generic error handler and silently dropped as a failed update.

P2 — Advance config_version to 28:
  Upstream main already consumed the previous 26→27 bump with a
  different schema change. Bump config.example.yaml, Helm values.yaml,
  and Helm README.md to 28 to version this PR's patterns_dir and
  prompts_dir additions.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): render text templates at construction, propagate PromptConfigurationError

Keep prompt-configuration failures out of the recoverable update catch:

- Define PromptConfigurationError(ValueError) in prompt.py. Raised by
  load_prompt, load_prompt_messages, and _render_messages for bad yaml,
  missing keys, and invalid placeholders.

- _do_update_memory_sync_impl, update_memory's executor path, and
  _process_queue all re-raise (PromptConfigurationError, FileNotFoundError,
  OSError) before the generic except Exception, so a bad explicit prompt
  is never silently returned as False.

- DeerMem.__init__ prompt validation now renders text templates with
  dummy variables (.format(stale_facts="") etc.) so an unknown
  placeholder in staleness_review.yaml, consolidation.yaml, or
  fact_extraction.yaml is caught at construction.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): drop re-raise guards, soften validation comment, exclude dormant fact_extraction

- Remove (PromptConfigurationError, FileNotFoundError, OSError) re-raise
  guards from _do_update_memory_sync_impl, update_memory executor path,
  and _process_queue. The existing except Exception: logger.exception(...)
  already logs prompt-config errors at ERROR with full traceback; the
  re-raise was killing the entire batch and only surfacing via stderr.
  Per-agent override errors now log at ERROR per-item without aborting
  co-tenant updates.

- Soften the construction validation comment to clarify it only covers
  global templates. Per-agent overrides are validated lazily at first use
  and logged at ERROR by the updater's exception handler.

- Drop fact_extraction from construction validation (dormant prompt with
  no runtime caller; the shim + yaml remain for backward compat).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-20 07:24:34 +08:00
..

Memory Backends

Each subfolder under agents/memory/backends/ is a pluggable memory backend. Swap the active one by changing one line in config.yaml - no deer-flow core changes required.

  • deermem/ - the default backend (deer-flow's own: structured facts + JSON storage).
  • noop/ - an empty backend and the template to copy when adding a new one.

This guide tells you which files to touch when you change, swap, or add a memory system. Paths are relative to backend/ unless noted.


Table of Contents

Add a New Backend

Copy noop/ to backends/<yourname>/ and edit three files in this folder plus two outside it.

File What to change
backends/<yourname>/config.py Declare your config fields + from_backend_config (parse backend_config; read storage_path from it - do not import deer-flow path helpers)
backends/<yourname>/<yourname>_manager.py Rename the class; parse config in __init__; implement the 9 ABC methods; optionally implement fact CRUD (see Backend Contract)
backends/<yourname>/__init__.py MANAGER_CLASS = YourManager (relative import)
config.yaml (repo root, parent of backend/) memory.manager_class: <yourname> + your knobs under memory.backend_config
packages/harness/pyproject.toml Only if the backend needs external libs: declare the dependency; add [tool.uv.sources] for vendored source. Otherwise uv sync purges it (see Common Pitfalls)

See the docstring at the top of noop/noop_manager.py for the full 6-step walkthrough.

Switch the Active Backend

Edit config.yaml (repo root) only:

memory:
  manager_class: <name>        # deermem / noop / <yourname>
  backend_config: { ... }      # that backend's private config

Then restart deer-flow - the memory manager is a process-level singleton; a running process does not hot-reload config or backend code.

Backend Contract

1. The 9 ABC methods

Implement every method on MemoryManager in packages/harness/deerflow/agents/memory/manager.py. Signatures must match (parameter names, keyword-only args). noop is the empty-implementation reference.

2. Return shape (critical, easy to get wrong)

get_memory / export_memory / clear_memory / import_memory return a dict that the gateway casts to the DeerMem shape (MemoryResponse: version / lastUpdated / user / history / facts[]). Your backend must return a dict this shape accepts, or:

  • the data is silently dropped (pydantic ignores unknown fields);
  • the frontend gets empty defaults and lastUpdated="" crashes the date formatter.

A non-DeerMem backend maps its native records (e.g. {"results": [...]}) into this shape via a small adapter helper.

3. Optional capabilities (DeerMem-internal, not on the ABC)

The gateway probes these with hasattr(manager, "<name>") and returns 501 when absent:

  • create_fact / delete_fact / update_fact - the frontend's add/delete/edit-fact buttons. Signatures are in the commented block at the bottom of noop/noop_manager.py.
  • reload_memory - the frontend's reload button (delegate to get_memory if your backend has no cache).
  • warm - one-time warm-up at gateway startup (skipped if absent).

Implement the ones you support; leave the rest as 501.

4. Portability (the golden rule)

Important

A backend talks to the host through exactly two channels: (1) the ABC method arguments (manager.py), and (2) the backend_config dict. The only from deerflow import allowed anywhere in your backend folder is the ABC contract line in <name>_manager.py:

from deerflow.agents.memory.manager import MemoryManager

Change that one line (and only that line) to port the backend to another agent. Do not import deer-flow path helpers, config singletons, or models - get storage_path and everything else from backend_config.

5. What the host injects into backend_config

The factory (manager.py::get_memory_manager) injects these for every backend:

  • storage_path (str) - a writable state dir (the host's runtime_home by default, or whatever config.yaml sets). Use this as your storage root.
  • tracing_callback (Callable | None) - trace your LLM calls (langfuse). Ignore if you don't trace.
  • should_keep_hidden_message (Callable | None) - filter hide_from_ui messages. Ignore if not relevant.
  • Plus whatever the user puts under config.yaml::memory.backend_config (your backend's own knobs).

Do Not Modify

These are backend-agnostic. Don't touch them when swapping backends (unless you're changing the shared contract, which affects every backend):

File Role
packages/harness/deerflow/agents/memory/manager.py ABC + factory + scanner
packages/harness/deerflow/agents/middlewares/memory_middleware.py after_agent -> manager.add
packages/harness/deerflow/agents/memory/summarization_hook.py summarization -> manager.add_nowait
packages/harness/deerflow/agents/lead_agent/prompt.py _get_memory_context -> manager.get_context
app/gateway/routers/memory.py HTTP endpoints -> manager.* (hasattr-probed)
packages/harness/deerflow/config/memory_config.py shared 4 fields (enabled / injection_enabled / manager_class / backend_config)
frontend/src/components/workspace/settings/memory-settings-page.tsx frontend memory page (assumes DeerMem shape)

Note

The gateway and frontend are currently hard-coded to the DeerMem shape - that's why backends must return DeerMem-shape data (contract #2). Making them fully backend-agnostic is a larger refactor; see E:\deerflow\memory\plugin\00-插件兼容性矩阵.md.

Common Pitfalls

Lessons from integrating external backends:

  1. External deps must be declared in pyproject.toml. A bare uv pip install is purged on the next uv sync / langgraph dev. Declare the dep (and [tool.uv.sources] for vendored source).
  2. Return the DeerMem shape. Otherwise the frontend crashes with Invalid time value and your data is silently dropped. Build a small adapter helper to map your native records into it.
  3. Fact CRUD returns 501 if not implemented. The frontend's delete-fact button reports Operation 'delete fact' not supported. Implement delete_fact (and friends) to fix it.
  4. Don't import runtime_home. Read storage_path from backend_config. (The noop template shows the correct pattern; importing deer-flow path helpers breaks portability - contract #4.)
  5. Restart deer-flow after changes. The manager is a process-level singleton; a running process does not hot-reload config or backend code.
  6. Cap get_context length yourself. The host applies no token budget; the backend must truncate (DeerMem has max_injection_tokens; noop does not).

Reference

  • Template: noop/ - empty implementation with full docstrings; copy and go.
  • Design proposal: E:\deerflow\memory\记忆系统方案.md.
  • Plugin plans + compatibility matrix: E:\deerflow\memory\plugin\ (00-插件兼容性矩阵.md is the spine; defines the 9 shared contracts S1-S9).