mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 21:50:36 +00:00
docs(memory): correct backend contract details in the MemoryManager guide (#5943)
Follow-up to #5929. A config or custom backend written from a few statements on the page fails: - OpenViking's failure_policy.read takes fail_open or raise, and the backend supports middleware mode only. - from_config is abstract, so it belongs in the mandatory set. - Tool mode also needs get_memory() and fact CRUD, because it skips the per-turn add() writes. - Management methods return the DeerMem shape, and create_fact returns a (memory_data, fact_id) tuple. - staleness_review_enabled is on by default, and the governance flags live under memory.backend_config. The zh page gets the same fixes plus a typo (回记 -> 召回). Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
This commit is contained in:
parent
d700ad9edf
commit
457ac8cabd
@ -47,7 +47,7 @@ At the core of the memory feature is a **backend-neutral `MemoryManager`** (defi
|
||||
The design goal is a single sentence: **swapping the memory backend requires zero changes anywhere else in DeerFlow**.
|
||||
|
||||
- **The contract says what, not how.** `get_context()` only has to return injection-ready text — whether that comes from a local file load or a remote retrieval call is the backend's own business. The contract does not even assume memory is a set of "facts".
|
||||
- **Tiered by implementation cost.** Only two methods are mandatory (write via `add`, read-and-inject via `get_context`); every other management operation (search, clear, import, fact CRUD, etc.) ships with a default "unsupported" implementation, so a backend exposes exactly as much capability as it implements.
|
||||
- **Tiered by implementation cost.** Only three members are mandatory: the `from_config` classmethod (assembly), `add` (write) and `get_context` (read-and-inject). Search, clear, import and fact CRUD ship with a default "unsupported" implementation, and the buffered-write helpers (`add_nowait`, `shutdown_flush`, `cancel_by_agent`) default to simple pass-throughs, so a backend exposes exactly as much capability as it implements.
|
||||
- **Fail loud, never silently.** Memory is persistent data. A misconfigured backend or a corrupted file raises directly (e.g. `ValueError` at build time, HTTP 409 on write conflicts) instead of quietly falling back to another store — which would silently write your data to the wrong place.
|
||||
- **Host and backend decoupled.** Host capabilities (Langfuse spans, the default LLM, extraction metrics) are injected as hooks that backends consume as needed, so the backend package itself never depends on a DeerFlow-specific concept and can be shipped independently.
|
||||
|
||||
@ -55,7 +55,7 @@ The `MemoryManager` contract at a glance:
|
||||
|
||||
| Tier | Methods | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Mandatory | `add` / `get_context` | Queue conversations for write (debounced) / return injection-ready memory text |
|
||||
| Mandatory | `from_config` / `add` / `get_context` | Build the instance from `backend_config` and host hooks / queue conversations for write (debounced) / return injection-ready memory text |
|
||||
| Management ops | `search`, `get_memory`, `clear_memory`, `import_memory`, `create_fact` / `update_fact` / `delete_fact`, `add_nowait`, `cancel_by_agent`, `shutdown_flush` | Search & inspect, clear & import, fact CRUD for tool mode, emergency flush before summarization, cancel pending extraction, bounded drain on graceful shutdown |
|
||||
| Optional hooks | `warm`, `reload_memory`, `on_pre_compress` / `on_turn_start`, plus async variants | Startup warm-up, cache reload, future extension points |
|
||||
|
||||
@ -92,7 +92,7 @@ deermem is the default backend and implements the full contract:
|
||||
- The factory `get_memory_manager()` reads `manager_class: deermem` and passes `backend_config` plus a set of **host hooks** (Langfuse tracing callbacks, the default extraction LLM, hidden-message filtering, extraction metrics) to `DeerMem.from_config()`, which assembles the instance.
|
||||
- Storage is bucketed per `(user, agent)`: `memory.json` holds only shared summaries and revision metadata, every fact is its own Markdown file (with YAML front matter), and a SQLite FTS5 full-text index backs `memory_search`.
|
||||
- Writes are queued by `MemoryMiddleware` calling `add()` after each turn (debounced and batched); summarization triggers an `add_nowait()` emergency flush so compacted content is never lost.
|
||||
- On top of that sit opt-in fact-governance capabilities: write-side near-duplicate merging (`fact_dedup_enabled`), staleness review (`staleness_review_enabled`), consolidation (`consolidation_enabled`), capacity eviction policy (`fact_eviction_policy: hybrid-v1`), and relevance-aware retrieval (`retrieval_relevance_enabled`).
|
||||
- On top of that sit fact-governance capabilities, all set under `memory.backend_config`: staleness review (`staleness_review_enabled`, on by default), plus opt-in write-side near-duplicate merging (`fact_dedup_enabled`), consolidation (`consolidation_enabled`), a capacity eviction policy (`fact_eviction_policy: hybrid-v1`; the default is `confidence`), and relevance-aware retrieval (`retrieval_relevance_enabled`).
|
||||
|
||||
### How openviking is integrated
|
||||
|
||||
@ -103,13 +103,13 @@ memory:
|
||||
enabled: true
|
||||
injection_enabled: true
|
||||
manager_class: openviking
|
||||
mode: middleware # middleware mode is recommended
|
||||
mode: middleware # required: openviking supports middleware mode only
|
||||
backend_config:
|
||||
base_url: http://openviking:1933
|
||||
owner_user_id: default # use default when auth is disabled
|
||||
api_key_env: OPENVIKING_API_KEY
|
||||
failure_policy:
|
||||
read: fail_open # read failure returns empty results (fail_closed aborts)
|
||||
read: fail_open # read failure returns empty results (raise aborts the turn)
|
||||
write: log_and_drop
|
||||
retrieval:
|
||||
top_k: 8
|
||||
@ -165,13 +165,16 @@ Only `from_config` + `add` + `get_context` are mandatory. Everything else is opt
|
||||
|
||||
| Capability you want | Method to override | Notes |
|
||||
| --- | --- | --- |
|
||||
| Tool mode (`memory_search` etc.) | `search()`, and set `supports_search = True` | Flag/implementation consistency is enforced at instantiation |
|
||||
| Inspect / clear / import memory | `get_memory()` / `clear_memory()` / `import_memory()` | Unimplemented ops raise "not supported" |
|
||||
| Fact-level CRUD | `create_fact()` / `update_fact()` / `delete_fact()` | Backs `memory_add` / `memory_update` / `memory_delete` in tool mode |
|
||||
| Tool mode (`memory_search` etc.) | `search()` with `supports_search = True`, plus `get_memory()` and the fact CRUD methods below | Flag/implementation consistency is enforced at instantiation. Tool mode does not install `MemoryMiddleware`, so the per-turn `add()` writes stop unless you set `requires_passive_writes_in_tool_mode = True` (the `add_nowait()` flush before summarization still runs, and by default it calls `add()`); `memory_add` reads `get_memory()` before calling `create_fact()` |
|
||||
| Inspect / clear / import memory | `get_memory()` / `clear_memory()` / `import_memory()` | Unimplemented ops raise "not supported". Return the DeerMem document shape, which the Gateway casts to `MemoryResponse` (`version`, `lastUpdated`, `user`, `history`, and `facts` whose entries carry `id` and `content`) |
|
||||
| Fact-level CRUD | `create_fact()` / `update_fact()` / `delete_fact()` | Backs `memory_add` / `memory_update` / `memory_delete` in tool mode. `create_fact` returns `(memory_data, fact_id)`; raise `KeyError` for an unknown fact id (HTTP 404) |
|
||||
| Buffered writes | `add_nowait()` / `shutdown_flush()` / `cancel_by_agent()` | The defaults only call `add()`, report the flush as done, and cancel nothing; override them if your `add()` buffers |
|
||||
| Startup warm-up (indexing, encoders, …) | `warm()`, returning `True` / `False` / `None` | `None` means nothing to warm; the log says "skipped" honestly |
|
||||
| Releasing connections and other resources | `close()` | Called on graceful shutdown |
|
||||
| Cache reload | `reload_memory()` | For caching backends after out-of-band edits |
|
||||
|
||||
`backend/packages/harness/deerflow/agents/memory/backends/README.md` explains the DeerMem return shape and common pitfalls, and `backends/noop/` is a working template to copy.
|
||||
|
||||
### Step 3: Declare failure and conflict semantics
|
||||
|
||||
- Read-failure policy is declared via `read_failures_are_fatal_for_config()`: permissive by default (failures return empty text); setting `backend_config.failure_policy.read: fail_closed` makes callers abort instead of degrading.
|
||||
|
||||
@ -46,7 +46,7 @@ import { Callout } from "nextra/components";
|
||||
设计目标只有一个:**换记忆后端时,DeerFlow 其余代码零改动**。
|
||||
|
||||
- **契约只管“做什么”,不管“怎么做”**。`get_context()` 只要求返回一段可注入提示的文本——是本地文件加载还是远程检索,由后端自行决定;契约甚至不假设记忆一定是“事实”的集合。
|
||||
- **按实现成本分层**。只有两个方法是必须实现的(写入 `add`、读取注入 `get_context`);其余管理操作(搜索、清空、导入、事实增删改等)都带默认的“不支持”实现,后端实现多少能力就暴露多少能力。
|
||||
- **按实现成本分层**。必须实现的只有三个:类方法 `from_config`(组装实例)、`add`(写入)和 `get_context`(读取注入);搜索、清空、导入和事实增删改默认“不支持”,缓冲写入相关的 `add_nowait` / `shutdown_flush` / `cancel_by_agent` 默认直接透传,后端实现多少能力就暴露多少能力。
|
||||
- **失败要响亮,不要静默**。记忆是持久化数据,配错后端、写坏文件时系统会直接报错(如构建时抛出 `ValueError`、写冲突返回 HTTP 409),绝不悄悄回退到另一个存储——那等于把你的数据写到了错误的地方。
|
||||
- **宿主与后端解耦**。追踪(Langfuse span)、默认 LLM、提取指标等宿主能力以“钩子”形式注入后端,后端按需消费,因此后端代码包本身不依赖任何 DeerFlow 特有概念,可以独立发布。
|
||||
|
||||
@ -54,7 +54,7 @@ import { Callout } from "nextra/components";
|
||||
|
||||
| 分层 | 方法 | 作用 |
|
||||
| --- | --- | --- |
|
||||
| 必须实现 | `add` / `get_context` | 写入对话(防抖排队)/ 返回注入就绪的记忆文本 |
|
||||
| 必须实现 | `from_config` / `add` / `get_context` | 根据 `backend_config` 和宿主钩子组装实例 / 写入对话(防抖排队)/ 返回注入就绪的记忆文本 |
|
||||
| 管理操作 | `search`、`get_memory`、`clear_memory`、`import_memory`、`create_fact` / `update_fact` / `delete_fact`、`add_nowait`、`cancel_by_agent`、`shutdown_flush` | 搜索与查看、清空与导入、工具模式的事实增删改、摘要压缩前的紧急冲刷、取消挂起的提取、优雅停机排水 |
|
||||
| 可选钩子 | `warm`、`reload_memory`、`on_pre_compress` / `on_turn_start` 及 async 变体 | 启动预热、缓存重载、面向未来的扩展点 |
|
||||
|
||||
@ -75,7 +75,7 @@ memory:
|
||||
| `manager_class` | 类型 | 定位 |
|
||||
| --- | --- | --- |
|
||||
| `deermem` | 本地(默认) | 自带的完整记忆实现:Markdown 事实存储、摘要、全文检索与事实生命周期管理 |
|
||||
| `openviking` | 远程 | 接入独立的 OpenViking 记忆服务器,通过官方 `langchain-openviking` 适配器回记 |
|
||||
| `openviking` | 远程 | 接入独立的 OpenViking 记忆服务器,通过官方 `langchain-openviking` 适配器召回 |
|
||||
| `mem0` / `honcho` | 远程 | 接入 Mem0 / Honcho 记忆产品的适配器 |
|
||||
| `noop` | 本地 | 空实现:保留记忆调用点但什么都不做 |
|
||||
|
||||
@ -91,7 +91,7 @@ deermem 是默认后端,完整实现了契约的全部能力:
|
||||
- 工厂 `get_memory_manager()` 读取 `manager_class: deermem`,把 `backend_config` 连同一组**宿主钩子**(Langfuse 追踪回调、默认提取 LLM、隐藏消息过滤、提取指标回调)一起传给 `DeerMem.from_config()` 装配实例。
|
||||
- 存储按 `(用户, Agent)` 双维度分桶:`memory.json` 只存共享摘要与修订元数据,每条事实是独立的 Markdown 文件(带 YAML front matter),并维护 SQLite FTS5 全文索引供 `memory_search` 使用。
|
||||
- 写入侧由 `MemoryMiddleware` 在每轮结束后调用 `add()` 排队(防抖批量处理),摘要压缩前通过 `add_nowait()` 紧急冲刷,避免被压缩掉的内容丢失。
|
||||
- 在此之上提供可选的事实治理能力(按需开启):写侧近重复去重(`fact_dedup_enabled`)、陈旧性复核(`staleness_review_enabled`)、合并巩固(`consolidation_enabled`)、容量淘汰策略(`fact_eviction_policy: hybrid-v1`)以及相关性感知检索(`retrieval_relevance_enabled`)。
|
||||
- 在此之上提供事实治理能力,全部配置在 `memory.backend_config` 下:陈旧性复核(`staleness_review_enabled`,默认开启),以及按需开启的写侧近重复去重(`fact_dedup_enabled`)、合并巩固(`consolidation_enabled`)、容量淘汰策略(`fact_eviction_policy: hybrid-v1`,默认为 `confidence`)和相关性感知检索(`retrieval_relevance_enabled`)。
|
||||
|
||||
### openviking 是怎么接入的
|
||||
|
||||
@ -102,13 +102,13 @@ memory:
|
||||
enabled: true
|
||||
injection_enabled: true
|
||||
manager_class: openviking
|
||||
mode: middleware # 建议保持 middleware 模式
|
||||
mode: middleware # 必须:openviking 只支持 middleware 模式
|
||||
backend_config:
|
||||
base_url: http://openviking:1933
|
||||
owner_user_id: default # 未开启认证时用 default
|
||||
api_key_env: OPENVIKING_API_KEY
|
||||
failure_policy:
|
||||
read: fail_open # 读取失败时返回空结果(fail_closed 则中断)
|
||||
read: fail_open # 读取失败时返回空结果(raise 则中断本轮)
|
||||
write: log_and_drop
|
||||
retrieval:
|
||||
top_k: 8
|
||||
@ -163,13 +163,16 @@ class MyMemoryManager(MemoryManager):
|
||||
|
||||
| 想支持的能力 | 覆盖的方法 | 备注 |
|
||||
| --- | --- | --- |
|
||||
| 工具模式(`memory_search` 等) | `search()`,并设 `supports_search = True` | 声明与实现的一致性在实例化时强制校验 |
|
||||
| 记忆的查看 / 清空 / 导入 | `get_memory()` / `clear_memory()` / `import_memory()` | 不实现则调用时报“不支持” |
|
||||
| 事实级增删改 | `create_fact()` / `update_fact()` / `delete_fact()` | 工具模式的 `memory_add` / `memory_update` / `memory_delete` 走这里 |
|
||||
| 工具模式(`memory_search` 等) | `search()` 并设 `supports_search = True`,另需 `get_memory()` 和下方的事实增删改方法 | 声明与实现的一致性在实例化时强制校验。工具模式不安装 `MemoryMiddleware`,除非设 `requires_passive_writes_in_tool_mode = True`,否则每轮结束后的 `add()` 写入不会发生(摘要压缩前的 `add_nowait()` 冲刷仍会执行,其默认实现会调用 `add()`);`memory_add` 会先读 `get_memory()` 再调 `create_fact()` |
|
||||
| 记忆的查看 / 清空 / 导入 | `get_memory()` / `clear_memory()` / `import_memory()` | 不实现则调用时报“不支持”。返回值须是 DeerMem 的文档结构,Gateway 会把它转成 `MemoryResponse`(`version`、`lastUpdated`、`user`、`history`,以及每条都带 `id` 和 `content` 的 `facts`) |
|
||||
| 事实级增删改 | `create_fact()` / `update_fact()` / `delete_fact()` | 工具模式的 `memory_add` / `memory_update` / `memory_delete` 走这里。`create_fact` 返回 `(memory_data, fact_id)`;事实 id 不存在时抛 `KeyError`(HTTP 404) |
|
||||
| 缓冲写入 | `add_nowait()` / `shutdown_flush()` / `cancel_by_agent()` | 默认实现只会调用 `add()`、直接报告冲刷完成、什么也不取消;如果你的 `add()` 带缓冲,需要自己覆盖 |
|
||||
| 启动预热(建索引、加载编码器等) | `warm()`,返回 `True` / `False` / `None` | `None` 表示无事可做,日志会如实记录“跳过” |
|
||||
| 释放连接等外部资源 | `close()` | 优雅停机时调用 |
|
||||
| 缓存重载 | `reload_memory()` | 供带缓存的后端在带外修改后刷新 |
|
||||
|
||||
DeerMem 的返回结构和常见坑见 `backend/packages/harness/deerflow/agents/memory/backends/README.md`,`backends/noop/` 是可以直接复制的模板。
|
||||
|
||||
### 第 3 步:声明失败与冲突语义
|
||||
|
||||
- 读取失败的处理策略通过 `read_failures_are_fatal_for_config()` 声明:默认宽容(失败返回空文本),配置 `backend_config.failure_policy.read: fail_closed` 则让调用方中止而不是降级。
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user