mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-08-01 10:56:02 +00:00
fix(memory): prevent task-scoped data from entering long-term memory (#4604)
* fix(memory): gate long-term updates by scope * docs(memory): note custom prompts_dir migration; fix stale accept-filter comment * fix(memory): harden scope-gate review paths
This commit is contained in:
parent
2a143dced6
commit
cccda35cc5
@ -1014,6 +1014,8 @@ requires an explicit local-development opt-in. See the
|
||||
|
||||
Memory updates now skip duplicate fact entries at apply time, so repeated preferences and context do not accumulate endlessly across sessions.
|
||||
|
||||
In the default DeerMem `middleware` mode, automatic extraction now classifies every proposed fact by scope, durability, and authority before a deterministic write gate accepts it. Only durable, descriptive user-level facts are stored; current-thread or project constraints and one-time action permissions stay in conversation state. User-global summaries require both user scope and descriptive authority, contradiction removals are scope-gated, and a replacement-dependent removal is applied only when its replacement actually survives validation and storage. These classification labels are extraction-only metadata, add no extra LLM call, and are not written into the fact files. The explicit CRUD tools in `memory.mode: tool` remain a separate, model-directed path. Deployments that override the bundled DeerMem prompts via `memory.backend_config.prompts_dir` must add the new classification fields to their custom templates (the `memory_update` fact/summary/removal formats and the `consolidation` consolidated-fact schema): the write gate fails closed, so an un-migrated template stops every extraction-driven fact, summary, and removal write, surfacing only through the `rejected_by_scope_gate` metrics and the high-rejection-rate warning.
|
||||
|
||||
File-backed memory now separates global user context from agent facts. Each user has one `memory.json` containing only the project-independent `user` and `history` summaries; every fact is a canonical Markdown file below `agents/{agent_name}/facts/`. Existing lead-agent middleware, API, Settings, import/export, and embedded-client calls that omit `agent_name` resolve inside DeerMem to the reserved `__default__` bucket. That bucket is outside the valid custom-agent name grammar, so a real custom agent named `lead-agent` has a separate fact repository and deleting a custom agent cannot delete a memory-only directory without `config.yaml`. Public agent identifiers are case-insensitive and canonicalized to lowercase. Runtime/API readers still receive a compatibility `facts` array for the selected/default agent, so the frontend does not read agent facts from `memory.json`; structured Markdown `source` metadata is projected to the historical string field at the MemoryManager boundary. An unscoped Clear All first migrates facts from unread legacy per-agent JSON without adopting its soon-to-be-cleared summaries, then removes shared summaries and facts from every agent bucket while preserving agent configuration files, so a later read cannot resurrect skipped legacy facts; an explicitly agent-scoped clear removes only that agent's facts. On first normal read, old facts embedded in the user JSON are migrated automatically to `__default__`; facts written to the earlier implicit `lead-agent` bucket are also moved when that directory is not a real custom agent. Migration and normal writes notify the configured retrieval adapter only after durable storage locks are released. DeerMem uses a scope-aware SQLite FTS5/BM25 adapter by default, stores only rebuildable derived index data under `.retrieval/`, and rebuilds it in the background during Gateway startup or lazily on the first scoped search. A corrupt derived index is recreated automatically. Set `memory.backend_config.retrieval_adapter` to an empty string to disable it and use the local substring fallback. Chinese tokenization is optional; install the backend `memory-zh` extra (`uv sync --extra memory-zh`) for jieba-assisted sub-phrase search. Journaled writes, a shared user lock, and optimistic user-memory revisions prevent silent lost updates.
|
||||
|
||||
Memory injection follows the configured operation mode. In `middleware` mode, DeerMem injects the user-global summaries and the selected agent's facts. In `tool` mode, the automatic `<memory>` block contains only the global `user` and `history` summaries; agent facts are retrieved explicitly through `memory_search`, avoiding duplicate automatic and tool-returned fact context. Setting `memory.injection_enabled: false` still disables the entire block in either mode.
|
||||
|
||||
@ -651,6 +651,8 @@ DeerFlow 不只是“会说它能做”,它是真的有一台自己的“电
|
||||
|
||||
跨 session 使用时,DeerFlow 会逐步积累关于你的持久 memory,包括你的个人偏好、知识背景,以及长期沉淀下来的工作习惯。你用得越多,它越了解你的写作风格、技术栈和重复出现的工作流。memory 保存在本地,控制权也始终在你手里。
|
||||
|
||||
默认 DeerMem `middleware` 模式会先判断候选信息的作用域、持久性和授权属性,再由确定性写入门决定是否保存。只有稳定、描述性的用户级事实能进入长期 memory;当前对话或项目的约束、一次性操作授权仍留在对话状态中。用户全局 summary 必须同时具有用户级作用域和描述性授权属性,基于矛盾的删除也会经过作用域保护;如果删除依赖一条替代事实,只有替代事实真正通过校验并保留下来后才执行删除。这些分类字段只用于本次抽取,不写入 fact 文件,也不增加 LLM 调用次数。`memory.mode: tool` 的显式 CRUD 仍是独立的模型直写路径。如果通过 `memory.backend_config.prompts_dir` 覆盖了内置抽取模板,必须同步在自定义模板中加入新的分类字段(`memory_update` 的 fact/summary/removal 格式与 `consolidation` 的合并 fact 结构):写入门是 fail closed 的,未迁移的旧模板会导致所有抽取驱动的 fact、summary 与删除写入停止,只能通过 `rejected_by_scope_gate` 指标和高拒绝率告警发现。
|
||||
|
||||
## 推荐模型
|
||||
|
||||
DeerFlow 对模型没有强绑定,只要实现了 OpenAI 兼容 API 的 LLM,理论上都可以接入。不过在下面这些能力上表现更强的模型,通常会更适合 DeerFlow:
|
||||
|
||||
@ -935,6 +935,7 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_
|
||||
runtime.
|
||||
- `memory.mode: tool` skips `MemoryMiddleware` and registers `memory_search`, `memory_add`, `memory_update`, and `memory_delete` on the agent. The model decides when to search, add, update, or delete facts; this is opt-in/experimental and should not be described as better than middleware mode without eval evidence.
|
||||
- Both modes share `FileMemoryStorage`, per-user/per-agent isolation, manual CRUD primitives, and the updater backend. Injection is mode-aware: middleware mode injects global `user`/`history` summaries plus the selected agent's facts, while tool mode injects only the global summaries and leaves every agent fact behind `memory_search` to avoid duplicating automatically injected and retrieval-returned context. `memory.injection_enabled: false` suppresses the complete block in either mode.
|
||||
- Middleware extraction classifies proposed facts with extraction-only `scope`/`durability`/`authority` labels. `_apply_updates` accepts only `user` + `durable` + `descriptive` new/consolidated facts, accepts only wholly user-scoped summary prose with `authority=descriptive`, and rejects missing labels per item without aborting unrelated updates. Contradiction removals use object entries with `id`, `scope`, `reason`, and optional zero-based `replacementFactIndex`; task/project removals fail closed, and a paired removal runs only when the referenced replacement survives the scope/confidence gates, deduplication, and max-fact trim under another fact ID. The labels are not persisted, so no storage migration is required. Staleness removals retain their independent candidate/cap guardrails, while tool-mode CRUD remains outside this extraction gate. Custom `memory.backend_config.prompts_dir` templates (including per-agent overrides) must carry the same classification fields; an un-migrated template makes the fail-closed gate reject every extraction-driven write, observable only through `rejected_by_scope_gate` and the >60% fact-rejection warning.
|
||||
- Middleware mode queue debounces (30s default), batches updates, and commits global summaries plus the selected/default agent's fact delta through a user-level lock, optimistic user-memory revisions, per-fact revisions, and a recoverable target-file journal. Only explicitly marked point operations may rebase a stale shared revision, and only while every addressed fact still satisfies its original absent/revision precondition. Snapshot-derived clear/trim/consolidation operations instead reload the complete document and recompute their intent on a manifest conflict, with a bounded retry. Typed manifest/fact conflict subclasses keep that decision independent of exception text, and same-ID creates and stale same-fact writes fail. Scope-lock objects are weakly cached so inactive users do not grow a process-lifetime map. Cache validation does not scale with the fact-file count: its token combines the shared JSON's `(mtime_ns, size, revision)`, so the persisted revision invalidates stale caches even when a coarse-mtime filesystem reports identical metadata for same-size writes; direct out-of-band Markdown edits require `reload()`. Atomic replacement also syncs the parent directory on POSIX so the rename is durable. DeerMem translates private storage conflict/corruption exceptions to the backend-neutral MemoryManager contract; the Gateway maps them to HTTP 409 and a stable HTTP 500 response respectively. A normal default-manager read automatically migrates legacy facts from the global JSON into `__default__`; it also adopts the earlier implicit `lead-agent` fact bucket only when that directory has no custom-agent `config.yaml`, and rejects unexpected files instead of deleting them. The v1-to-v2 migration is one-way for the running application: operators must stop DeerFlow and snapshot the configured storage root before upgrade. Before any destructive v2 write, every migrated JSON source is durably retained as `{manifest_filename}.v1.bak`; a missing-write or mismatched existing backup aborts without modifying v1 data. Legacy per-agent JSON is deleted only after its non-empty summaries are safely adopted or confirmed identical; summary conflicts keep the source file and fail loudly.
|
||||
- **Proactive Markdown migration CLI**: from `backend/`, run `PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run` to audit and omit `--dry-run` to migrate before serving traffic. Use repeated `--user-id` values when selecting exact original identities, especially standalone raw IDs containing `@` or other characters that are normalized in directory names; `--storage-path` selects a non-default DeerMem root. The CLI reuses `FileMemoryStorage.migrate`, is idempotent, continues across per-user failures, and exits non-zero if any user fails. It is optional because the first normal read still performs the same migration automatically.
|
||||
- `retrieval_adapter` owns indexing and retrieval. `fts5` is the DeerMem default and uses a persistent derived SQLite index under `.retrieval/`; an empty value disables the adapter and selects `substring_fallback`. File storage sends upsert/remove notifications for normal writes and both explicit and lazy migrations after releasing durable storage locks, then delegates search. Gateway startup schedules `DeerMem.warm_retrieval()` as a background full rebuild so readiness is not delayed, while a first search lazily rebuilds its exact scope until warm-up completes. Individual malformed facts are logged and skipped without triggering repeated full scans; only a fatal adapter rebuild failure keeps lazy retry enabled. During shutdown, the Gateway waits at most one second for this derived rebuild and leaves the full configured timeout to the canonical memory flush; if the rebuild is still active, its adapter remains open until process exit. Adapter failures mark the scope dirty and fall back to canonical substring search until rebuilding succeeds. `FileMemoryStorage` owns and closes the adapter so higher layers do not reach into private storage state.
|
||||
|
||||
@ -91,6 +91,8 @@ Async task delegation with concurrent execution:
|
||||
LLM-powered persistent context retention across conversations:
|
||||
|
||||
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
|
||||
- **Scope-safe writes**: Middleware extraction stores only durable, descriptive user-level facts; global summaries also require descriptive authority, while contradiction removals and consolidated facts fail closed when scope metadata is missing or task/project-local
|
||||
- **Atomic replacements**: A contradiction removal linked to a replacement runs only after the replacement survives scope/confidence gates, deduplication, and fact-limit trimming
|
||||
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
|
||||
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
|
||||
- **System prompt injection**: Top facts + context injected into agent prompts
|
||||
|
||||
@ -15,11 +15,12 @@ template: |-
|
||||
- SKIP: Facts are distinct enough to remain separate.
|
||||
|
||||
Add consolidation decisions to "factsToConsolidate" in your output JSON.
|
||||
Each entry: {{"sourceIds": ["fact_id_1", "fact_id_2"], "consolidated": {{"content": "...", "category": "...", "confidence": 0.9}}}}
|
||||
Each entry: {{"sourceIds": ["fact_id_1", "fact_id_2"], "consolidated": {{"content": "...", "category": "...", "confidence": 0.9, "scope": "user|thread|project", "durability": "durable|temporary", "authority": "descriptive|transactional"}}}}
|
||||
|
||||
Rules:
|
||||
- The consolidated fact must preserve ALL key details from source facts
|
||||
- Only consolidate facts that describe the same aspect of the user
|
||||
- Classify every consolidated fact. It is eligible only when it is user-scoped, durable, descriptive, and safe to inject into unrelated future threads.
|
||||
- Confidence of consolidated fact = max of source confidences
|
||||
- Be conservative - when in doubt, keep facts separate
|
||||
- Maximum {max_groups} consolidation groups per cycle
|
||||
|
||||
@ -12,12 +12,21 @@ messages:
|
||||
|
||||
Before extracting facts, perform a structured reflection on the conversation:
|
||||
1. Error/Retry Detection: Did the agent encounter errors, require retries, or produce incorrect results?
|
||||
If yes, record the root cause and correct approach as a high-confidence fact with category "correction".
|
||||
If yes, record the root cause and correct approach only when it is a durable user-level working pattern.
|
||||
2. User Correction Detection: Did the user correct the agent's direction, understanding, or output?
|
||||
If yes, record the correct interpretation or approach as a high-confidence fact with category "correction".
|
||||
If yes, distinguish a reusable user-level correction from a correction to the current task's facts or files.
|
||||
Include what went wrong in "sourceError" only when category is "correction" and the mistake is explicit in the conversation.
|
||||
3. Project Constraint Discovery: Were any project-specific constraints discovered during the conversation?
|
||||
If yes, record them as facts with the most appropriate category and confidence.
|
||||
If yes, classify them as project-scoped and do not promote them to user memory.
|
||||
|
||||
Scope and Safety Classification:
|
||||
- scope="user": a property, preference, background detail, ongoing personal state, or durable working pattern of the user that is safe and useful to inject into an unrelated future thread, task, project, or repository.
|
||||
- scope="thread": information limited to the current request or conversation, including one-off constraints for a reply, file, email, trip, PR, test, or action.
|
||||
- scope="project": a rule, decision, state, or constraint that is meaningful only inside a particular project or repository, even if it may span several threads.
|
||||
- durability="durable": expected to remain true across future conversations. durability="temporary": current, short-lived, or one-off information.
|
||||
- authority="transactional": an instruction, grant, permission, or authorization to perform an action such as editing, deleting, pushing, closing, publishing, or force-pushing. Transactional content must never become long-term memory.
|
||||
- authority="descriptive": describes the user without granting authority for an action.
|
||||
- When uncertain, use thread/project or temporary; never guess user+durable.
|
||||
|
||||
Memory Section Guidelines:
|
||||
|
||||
@ -43,6 +52,9 @@ messages:
|
||||
Include: Core expertise, longstanding interests, fundamental working style
|
||||
|
||||
**Facts Extraction**:
|
||||
- Every new fact MUST include scope, durability, and authority. These labels are evaluated by a deterministic write gate and are not persisted.
|
||||
- Only facts classified as scope="user", durability="durable", authority="descriptive" are eligible for storage.
|
||||
- Do not create facts for current-task objectives, acceptance criteria, workspace state, exact current file/commit/error state, project-only constraints, or one-time action permissions.
|
||||
- Extract specific, quantifiable details (e.g., "16k+ GitHub stars", "200+ datasets")
|
||||
- Include proper nouns (company names, project names, technology names)
|
||||
- Preserve technical terminology and version numbers
|
||||
@ -85,25 +97,27 @@ messages:
|
||||
Output Format (JSON):
|
||||
{{
|
||||
"user": {{
|
||||
"workContext": {{ "summary": "...", "shouldUpdate": true/false }},
|
||||
"personalContext": {{ "summary": "...", "shouldUpdate": true/false }},
|
||||
"topOfMind": {{ "summary": "...", "shouldUpdate": true/false }}
|
||||
"workContext": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }},
|
||||
"personalContext": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }},
|
||||
"topOfMind": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }}
|
||||
}},
|
||||
"history": {{
|
||||
"recentMonths": {{ "summary": "...", "shouldUpdate": true/false }},
|
||||
"earlierContext": {{ "summary": "...", "shouldUpdate": true/false }},
|
||||
"longTermBackground": {{ "summary": "...", "shouldUpdate": true/false }}
|
||||
"recentMonths": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }},
|
||||
"earlierContext": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }},
|
||||
"longTermBackground": {{ "summary": "...", "shouldUpdate": true/false, "scope": "user|thread|project", "authority": "descriptive|transactional" }}
|
||||
}},
|
||||
"newFacts": [
|
||||
{{ "content": "...", "category": "preference|knowledge|context|behavior|goal|correction", "confidence": 0.0-1.0, "expected_valid_days": 90 }}
|
||||
{{ "content": "...", "category": "preference|knowledge|context|behavior|goal|correction", "confidence": 0.0-1.0, "expected_valid_days": 90, "scope": "user|thread|project", "durability": "durable|temporary", "authority": "descriptive|transactional" }}
|
||||
],
|
||||
"factsToRemove": [
|
||||
{{ "id": "fact_id_1", "scope": "user|thread|project", "reason": "explicit user-level contradiction or retraction", "replacementFactIndex": 0 }}
|
||||
],
|
||||
"factsToRemove": ["fact_id_1", "fact_id_2"],
|
||||
"staleFactsToRemove": [{{ "id": "fact_id", "reason": "brief explanation" }}],
|
||||
"staleFactsToExtend": [{{ "id": "fact_id", "extend_by_days": 365, "reason": "brief explanation" }}],
|
||||
"factsToConsolidate": [
|
||||
{{
|
||||
"sourceIds": ["fact_id_1", "fact_id_2"],
|
||||
"consolidated": {{ "content": "synthesized fact", "category": "knowledge", "confidence": 0.9 }}
|
||||
"consolidated": {{ "content": "synthesized fact", "category": "knowledge", "confidence": 0.9, "scope": "user|thread|project", "durability": "durable|temporary", "authority": "descriptive|transactional" }}
|
||||
}}
|
||||
]
|
||||
}}
|
||||
@ -115,7 +129,9 @@ messages:
|
||||
- Only add facts that are clearly stated (0.9+) or strongly implied (0.7+)
|
||||
- Use category "correction" for explicit agent mistakes or user corrections; assign confidence >= 0.95 when the correction is explicit
|
||||
- Include "sourceError" only for explicit correction facts when the prior mistake or wrong approach is clearly stated; omit it otherwise
|
||||
- Remove facts that are contradicted by new information
|
||||
- Remove an existing fact only when the user explicitly contradicts or retracts it at user scope. A thread/project-local exception does not contradict a user-level fact.
|
||||
- factsToRemove entries MUST include scope and reason. Use replacementFactIndex when a removal depends on a replacement in newFacts; the zero-based index must identify that replacement. Omit replacementFactIndex only for a pure user-level retraction with no replacement.
|
||||
- Every summary with shouldUpdate=true MUST include scope and authority. A summary is user-scoped only when the entire prose block is safe to inject into unrelated future threads; otherwise classify it as thread/project so the write gate rejects it. Any summary containing an instruction, grant, permission, or authorization is transactional and must be rejected even when it describes a user-wide or recurring policy.
|
||||
- When updating topOfMind, integrate new focus areas while removing completed/abandoned ones
|
||||
Keep 3-5 concurrent focus themes that are still active and relevant
|
||||
- For history sections, integrate new information chronologically into appropriate time period
|
||||
|
||||
@ -126,6 +126,52 @@ def _extract_text(content: Any) -> str:
|
||||
|
||||
|
||||
_REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS = frozenset({"user", "history", "newFacts"})
|
||||
_FACT_CLASSIFICATION_FIELDS = ("scope", "durability", "authority")
|
||||
|
||||
|
||||
def _normalize_gate_label(value: Any) -> str | None:
|
||||
"""Normalize a model-produced scope-gate label without validating policy."""
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
normalized = value.strip().lower()
|
||||
return normalized or None
|
||||
|
||||
|
||||
def _fact_scope_gate_reason(fact: dict[str, Any]) -> str | None:
|
||||
"""Return the deterministic rejection reason for a model-extracted fact."""
|
||||
if any(_normalize_gate_label(fact.get(field)) is None for field in _FACT_CLASSIFICATION_FIELDS):
|
||||
return "missing"
|
||||
if _normalize_gate_label(fact.get("scope")) != "user":
|
||||
return "scope"
|
||||
if _normalize_gate_label(fact.get("durability")) != "durable":
|
||||
return "durability"
|
||||
if _normalize_gate_label(fact.get("authority")) != "descriptive":
|
||||
return "authority"
|
||||
return None
|
||||
|
||||
|
||||
def _summary_scope_gate_reason(section_data: dict[str, Any]) -> str | None:
|
||||
"""Return the deterministic rejection reason for a summary update."""
|
||||
scope = _normalize_gate_label(section_data.get("scope"))
|
||||
authority = _normalize_gate_label(section_data.get("authority"))
|
||||
if scope is None or authority is None:
|
||||
return "missing"
|
||||
if scope != "user":
|
||||
return "scope"
|
||||
if authority != "descriptive":
|
||||
return "authority"
|
||||
return None
|
||||
|
||||
|
||||
def _removal_scope_gate_reason(removal: dict[str, Any]) -> str | None:
|
||||
"""Return the deterministic rejection reason for a contradiction removal."""
|
||||
scope = _normalize_gate_label(removal.get("scope"))
|
||||
reason = removal.get("reason")
|
||||
if scope is None or not isinstance(reason, str) or not reason.strip():
|
||||
return "missing"
|
||||
if scope != "user":
|
||||
return "scope"
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_memory_update_fact(fact: Any) -> dict[str, Any] | None:
|
||||
@ -180,6 +226,14 @@ def _normalize_memory_update_fact(fact: Any) -> dict[str, Any] | None:
|
||||
if evd is not None:
|
||||
normalized_fact["expected_valid_days"] = evd
|
||||
|
||||
# Scope classification is extraction-only metadata. Preserve it through
|
||||
# structural normalization so _apply_updates can fail closed per item, but
|
||||
# never copy it into the persisted fact_entry.
|
||||
for field in _FACT_CLASSIFICATION_FIELDS:
|
||||
normalized_value = _normalize_gate_label(fact.get(field))
|
||||
if normalized_value is not None:
|
||||
normalized_fact[field] = normalized_value
|
||||
|
||||
return normalized_fact
|
||||
|
||||
|
||||
@ -189,7 +243,35 @@ def _normalize_memory_update_data(update_data: dict[str, Any]) -> dict[str, Any]
|
||||
history = update_data.get("history")
|
||||
new_facts = update_data.get("newFacts")
|
||||
facts_to_remove = update_data.get("factsToRemove")
|
||||
normalized_facts_to_remove = [fact_id for fact_id in facts_to_remove if isinstance(fact_id, str)] if isinstance(facts_to_remove, list) else []
|
||||
normalized_facts_to_remove: list[dict[str, Any]] = []
|
||||
if isinstance(facts_to_remove, list):
|
||||
for entry in facts_to_remove:
|
||||
# Preserve the legacy string form as an unclassified removal. The
|
||||
# apply-layer gate will reject it as missing instead of continuing
|
||||
# to allow an unscoped destructive mutation.
|
||||
if isinstance(entry, str):
|
||||
fact_id = entry.strip()
|
||||
if fact_id:
|
||||
normalized_facts_to_remove.append({"id": fact_id})
|
||||
continue
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
raw_id = entry.get("id")
|
||||
if not isinstance(raw_id, str) or not raw_id.strip():
|
||||
continue
|
||||
normalized_removal: dict[str, Any] = {"id": raw_id.strip()}
|
||||
scope = _normalize_gate_label(entry.get("scope"))
|
||||
if scope is not None:
|
||||
normalized_removal["scope"] = scope
|
||||
reason = entry.get("reason")
|
||||
if isinstance(reason, str) and reason.strip():
|
||||
normalized_removal["reason"] = reason.strip()
|
||||
if "replacementFactIndex" in entry:
|
||||
# Preserve invalid values too: the apply layer must reject an
|
||||
# invalid dependency rather than silently treating it as a pure
|
||||
# removal and deleting the old fact.
|
||||
normalized_removal["replacementFactIndex"] = entry.get("replacementFactIndex")
|
||||
normalized_facts_to_remove.append(normalized_removal)
|
||||
normalized_new_facts = []
|
||||
dropped_new_fact = not isinstance(new_facts, list)
|
||||
if isinstance(new_facts, list):
|
||||
@ -290,6 +372,7 @@ def _normalize_memory_update_data(update_data: dict[str, Any]) -> dict[str, Any]
|
||||
"content": content.strip(),
|
||||
"category": _norm_cat,
|
||||
"confidence": _norm_conf,
|
||||
**{field: normalized for field in _FACT_CLASSIFICATION_FIELDS if (normalized := _normalize_gate_label(consolidated.get(field))) is not None},
|
||||
},
|
||||
}
|
||||
)
|
||||
@ -1021,25 +1104,24 @@ class MemoryUpdater:
|
||||
if "correction" in signals:
|
||||
hints.append(
|
||||
"IMPORTANT: Explicit correction signals were detected in this conversation. "
|
||||
"Pay special attention to what the agent got wrong, what the user corrected, "
|
||||
"and record the correct approach as a fact with category "
|
||||
'"correction" and confidence >= 0.95 when appropriate.'
|
||||
"Record a correction with confidence >= 0.95 only when it describes a durable, user-level "
|
||||
"working preference that is safe to reuse across unrelated tasks. A correction to facts, files, "
|
||||
"directions, or constraints in the current task is thread- or project-scoped and must not be stored."
|
||||
)
|
||||
if "reinforcement" in signals:
|
||||
hints.append(
|
||||
"IMPORTANT: Positive reinforcement signals were detected in this conversation. "
|
||||
"The user explicitly confirmed the agent's approach was correct or helpful. "
|
||||
"Record the confirmed approach, style, or preference as a fact with category "
|
||||
'"preference" or "behavior" and confidence >= 0.9 when appropriate.'
|
||||
"Record the confirmed approach, style, or preference with high confidence only if it is a durable, "
|
||||
"user-level pattern. Approval of the current result or current task is thread-scoped and must not be stored."
|
||||
)
|
||||
if "preference" in signals:
|
||||
hints.append('IMPORTANT: A preference signal was detected. Record the user\'s stated preference or dislike as a fact with category "preference" and high confidence.')
|
||||
hints.append("IMPORTANT: A preference signal was detected. Record it with high confidence only when it is a durable, user-level preference; a one-off choice for the current task is thread-scoped and must not be stored.")
|
||||
if "identity" in signals:
|
||||
hints.append('IMPORTANT: An identity signal was detected. Record the user\'s stated role, profession, or background as a fact with category "identity" and high confidence.')
|
||||
hints.append("IMPORTANT: An identity signal was detected. Record the user's stated role, profession, or background only when it is user-level and durable across tasks.")
|
||||
if "goal" in signals:
|
||||
hints.append('IMPORTANT: A goal signal was detected. Record the user\'s stated objective or intent as a fact with category "goal" and high confidence.')
|
||||
hints.append("IMPORTANT: A goal signal was detected. Record only a durable, user-level goal that remains useful across unrelated tasks; the objective of the current task, sprint, PR, or thread must not be stored.")
|
||||
if "decision" in signals:
|
||||
hints.append('IMPORTANT: A decision signal was detected. Record the user\'s decision or chosen option as a fact with category "decision" and high confidence.')
|
||||
hints.append("IMPORTANT: A decision signal was detected. Record only a durable, user-level decision or working pattern; a choice made for the current task, file, PR, or thread must not be stored.")
|
||||
return "\n".join(hints)
|
||||
|
||||
def _prepare_update_prompt(
|
||||
@ -1534,22 +1616,34 @@ class MemoryUpdater:
|
||||
update_data: Updates from LLM.
|
||||
thread_id: Optional thread ID for tracking.
|
||||
metrics: Optional observability dict. When provided, populated with
|
||||
``facts_passed_confidence`` / ``rejected_low_confidence`` counted
|
||||
at the real confidence-filter site below (the only acceptance
|
||||
gate for new facts), so the metric cannot drift from the actual
|
||||
filter the way a re-derived count in the caller could.
|
||||
confidence and scope-gate counters counted at their real filter
|
||||
sites, so observability cannot drift from actual acceptance.
|
||||
|
||||
Returns:
|
||||
Updated memory data.
|
||||
"""
|
||||
config = self._config
|
||||
now = utc_now_iso_z()
|
||||
scope_gate_rejections: dict[str, dict[str, int]] = {
|
||||
"facts": {"missing": 0, "scope": 0, "durability": 0, "authority": 0},
|
||||
"summaries": {"missing": 0, "scope": 0, "authority": 0},
|
||||
"removals": {"missing": 0, "scope": 0, "replacement": 0},
|
||||
"consolidations": {"missing": 0, "scope": 0, "durability": 0, "authority": 0},
|
||||
}
|
||||
|
||||
def reject_by_scope_gate(kind: str, reason: str) -> None:
|
||||
scope_gate_rejections[kind][reason] += 1
|
||||
|
||||
# Update user sections
|
||||
user_updates = update_data.get("user", {})
|
||||
for section in ["workContext", "personalContext", "topOfMind"]:
|
||||
section_data = user_updates.get(section, {})
|
||||
if section_data.get("shouldUpdate") and section_data.get("summary"):
|
||||
if not isinstance(section_data, dict) or not section_data.get("shouldUpdate") or not section_data.get("summary"):
|
||||
continue
|
||||
rejection_reason = _summary_scope_gate_reason(section_data)
|
||||
if rejection_reason is not None:
|
||||
reject_by_scope_gate("summaries", rejection_reason)
|
||||
else:
|
||||
current_memory["user"][section] = {
|
||||
"summary": section_data["summary"],
|
||||
"updatedAt": now,
|
||||
@ -1559,17 +1653,17 @@ class MemoryUpdater:
|
||||
history_updates = update_data.get("history", {})
|
||||
for section in ["recentMonths", "earlierContext", "longTermBackground"]:
|
||||
section_data = history_updates.get(section, {})
|
||||
if section_data.get("shouldUpdate") and section_data.get("summary"):
|
||||
if not isinstance(section_data, dict) or not section_data.get("shouldUpdate") or not section_data.get("summary"):
|
||||
continue
|
||||
rejection_reason = _summary_scope_gate_reason(section_data)
|
||||
if rejection_reason is not None:
|
||||
reject_by_scope_gate("summaries", rejection_reason)
|
||||
else:
|
||||
current_memory["history"][section] = {
|
||||
"summary": section_data["summary"],
|
||||
"updatedAt": now,
|
||||
}
|
||||
|
||||
# Remove facts (contradiction-based)
|
||||
facts_to_remove = set(update_data.get("factsToRemove", []))
|
||||
if facts_to_remove:
|
||||
current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in facts_to_remove]
|
||||
|
||||
# ── Staleness review: removals + lifetime extensions ──
|
||||
# Both operations share one staleness-candidate guardrail pass and one
|
||||
# candidate_ids set. proposed_remove_ids is hoisted out of the removals
|
||||
@ -1672,63 +1766,103 @@ class MemoryUpdater:
|
||||
# Creation-time lifetime cap shared with the consolidation path below, so
|
||||
# both fact-creation sites apply the identical bound in one place.
|
||||
creation_cap = int(config.staleness_age_days * config.staleness_max_lifetime_multiplier)
|
||||
# Counted at the confidence-gate site (the only real accept filter for new
|
||||
# facts) so the ``facts_passed_confidence`` metric mirrors the actual
|
||||
# filter and cannot drift from it. Facts below the threshold are the
|
||||
# reject count; duplicate / empty / over-cap facts that pass the
|
||||
# threshold are still counted here -- the metric is a confidence-gate
|
||||
# signal (the host's rejection-rate warning monitors confidence
|
||||
# filtering, not dedup / over-cap), not a persisted-fact count.
|
||||
# Two independent accept filters govern new facts: the deterministic
|
||||
# scope gate and this confidence threshold. Each is counted at its own
|
||||
# filter site so neither metric can drift from the filter it reports:
|
||||
# ``facts_passed_confidence`` counts threshold-passers even when the
|
||||
# scope gate rejects them, and the scope-gate counters increment
|
||||
# whether or not the confidence check passes. Duplicate / empty /
|
||||
# over-cap facts that pass the threshold are still counted here -- the
|
||||
# metric is a confidence-gate signal (the host's rejection-rate
|
||||
# warning monitors confidence filtering, not dedup / over-cap), not a
|
||||
# persisted-fact count.
|
||||
passed_threshold = 0
|
||||
for fact in new_facts:
|
||||
replacement_fact_keys: dict[int, str] = {}
|
||||
for fact_index, fact in enumerate(new_facts):
|
||||
confidence = fact.get("confidence", 0.5)
|
||||
if confidence >= config.fact_confidence_threshold:
|
||||
passed_threshold += 1
|
||||
raw_content = fact.get("content", "")
|
||||
if not isinstance(raw_content, str):
|
||||
continue
|
||||
normalized_content = raw_content.strip()
|
||||
fact_key = _fact_content_key(normalized_content)
|
||||
if fact_key is None:
|
||||
# Empty / whitespace-only content: skip it the same way the
|
||||
# non-string guard above does, instead of appending a blank
|
||||
# fact that violates the non-empty-content invariant.
|
||||
continue
|
||||
if fact_key in existing_fact_keys:
|
||||
continue
|
||||
rejection_reason = _fact_scope_gate_reason(fact)
|
||||
if rejection_reason is not None:
|
||||
reject_by_scope_gate("facts", rejection_reason)
|
||||
continue
|
||||
if confidence < config.fact_confidence_threshold:
|
||||
continue
|
||||
raw_content = fact.get("content", "")
|
||||
if not isinstance(raw_content, str):
|
||||
continue
|
||||
normalized_content = raw_content.strip()
|
||||
fact_key = _fact_content_key(normalized_content)
|
||||
if fact_key is None:
|
||||
# Empty / whitespace-only content: skip it the same way the
|
||||
# non-string guard above does, instead of appending a blank
|
||||
# fact that violates the non-empty-content invariant.
|
||||
continue
|
||||
# Remember every eligible replacement's content key even when it is
|
||||
# already present. A paired removal is safe only if the post-trim
|
||||
# memory contains this content under an ID other than its target.
|
||||
replacement_fact_keys[fact_index] = fact_key
|
||||
if fact_key in existing_fact_keys:
|
||||
continue
|
||||
|
||||
fact_entry = {
|
||||
"id": f"fact_{uuid.uuid4().hex[:8]}",
|
||||
"content": normalized_content,
|
||||
"category": fact.get("category", "context"),
|
||||
"confidence": confidence,
|
||||
"createdAt": now,
|
||||
"source": thread_id or "unknown",
|
||||
}
|
||||
source_error = fact.get("sourceError")
|
||||
if isinstance(source_error, str):
|
||||
normalized_source_error = source_error.strip()
|
||||
if normalized_source_error:
|
||||
fact_entry["sourceError"] = normalized_source_error
|
||||
evd = _read_expected_valid_days(fact)
|
||||
if evd is not None:
|
||||
# Apply the creation-time cap so the LLM cannot assign an
|
||||
# unbounded lifetime that defers staleness review indefinitely.
|
||||
# Extensions (staleFactsToExtend) bypass this cap via their own
|
||||
# staleness_max_extension_days ceiling because they represent a
|
||||
# deliberate review decision, not an unchecked initial assignment.
|
||||
fact_entry["expected_valid_days"] = min(evd, creation_cap)
|
||||
current_memory["facts"].append(fact_entry)
|
||||
if fact_key is not None:
|
||||
existing_fact_keys.add(fact_key)
|
||||
|
||||
if metrics is not None:
|
||||
metrics["facts_passed_confidence"] = passed_threshold
|
||||
metrics["rejected_low_confidence"] = len(new_facts) - passed_threshold
|
||||
fact_entry = {
|
||||
"id": f"fact_{uuid.uuid4().hex[:8]}",
|
||||
"content": normalized_content,
|
||||
"category": fact.get("category", "context"),
|
||||
"confidence": confidence,
|
||||
"createdAt": now,
|
||||
"source": thread_id or "unknown",
|
||||
}
|
||||
source_error = fact.get("sourceError")
|
||||
if isinstance(source_error, str):
|
||||
normalized_source_error = source_error.strip()
|
||||
if normalized_source_error:
|
||||
fact_entry["sourceError"] = normalized_source_error
|
||||
evd = _read_expected_valid_days(fact)
|
||||
if evd is not None:
|
||||
# Apply the creation-time cap so the LLM cannot assign an
|
||||
# unbounded lifetime that defers staleness review indefinitely.
|
||||
# Extensions (staleFactsToExtend) bypass this cap via their own
|
||||
# staleness_max_extension_days ceiling because they represent a
|
||||
# deliberate review decision, not an unchecked initial assignment.
|
||||
fact_entry["expected_valid_days"] = min(evd, creation_cap)
|
||||
current_memory["facts"].append(fact_entry)
|
||||
existing_fact_keys.add(fact_key)
|
||||
|
||||
# Enforce max facts limit (coerced confidence -- see _trim_facts_to_max).
|
||||
current_memory["facts"] = _trim_facts_to_max(current_memory["facts"], config.max_facts)
|
||||
|
||||
# Remove contradicted facts only after replacements have passed both
|
||||
# gates and survived deduplication/trimming. Task-local contradictions
|
||||
# cannot delete user memory, and a failed paired replacement cannot
|
||||
# degrade into a delete-only update.
|
||||
fact_ids_to_remove: set[str] = set()
|
||||
for removal in update_data.get("factsToRemove", []):
|
||||
if not isinstance(removal, dict):
|
||||
reject_by_scope_gate("removals", "missing")
|
||||
continue
|
||||
rejection_reason = _removal_scope_gate_reason(removal)
|
||||
if rejection_reason is not None:
|
||||
reject_by_scope_gate("removals", rejection_reason)
|
||||
continue
|
||||
fact_id = removal.get("id")
|
||||
if not isinstance(fact_id, str) or not fact_id:
|
||||
reject_by_scope_gate("removals", "missing")
|
||||
continue
|
||||
if "replacementFactIndex" in removal:
|
||||
replacement_index = removal.get("replacementFactIndex")
|
||||
if not isinstance(replacement_index, int) or isinstance(replacement_index, bool) or replacement_index < 0:
|
||||
reject_by_scope_gate("removals", "replacement")
|
||||
continue
|
||||
replacement_key = replacement_fact_keys.get(replacement_index)
|
||||
if replacement_key is None or not any(fact.get("id") != fact_id and _fact_content_key(fact.get("content")) == replacement_key for fact in current_memory.get("facts", [])):
|
||||
reject_by_scope_gate("removals", "replacement")
|
||||
continue
|
||||
fact_ids_to_remove.add(fact_id)
|
||||
|
||||
if fact_ids_to_remove:
|
||||
current_memory["facts"] = [fact for fact in current_memory.get("facts", []) if fact.get("id") not in fact_ids_to_remove]
|
||||
|
||||
# ── Memory consolidation ──
|
||||
# Runs after the max_facts trim so source facts that were just evicted
|
||||
# (low confidence, pushed out by high-confidence newFacts) are absent
|
||||
@ -1789,6 +1923,10 @@ class MemoryUpdater:
|
||||
content = consolidated.get("content", "")
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
continue
|
||||
rejection_reason = _fact_scope_gate_reason(consolidated)
|
||||
if rejection_reason is not None:
|
||||
reject_by_scope_gate("consolidations", rejection_reason)
|
||||
continue
|
||||
|
||||
source_confidences = [_coerce_source_confidence(fact_index[sid]) for sid in source_ids]
|
||||
# _coerce_source_confidence already clamps each value to [0, 1],
|
||||
@ -1899,4 +2037,11 @@ class MemoryUpdater:
|
||||
current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in ids_consumed]
|
||||
current_memory["facts"].extend(new_consolidated)
|
||||
|
||||
if metrics is not None:
|
||||
metrics["facts_passed_confidence"] = passed_threshold
|
||||
metrics["rejected_low_confidence"] = len(new_facts) - passed_threshold
|
||||
metrics["facts_passed_scope_gate"] = len(new_facts) - sum(scope_gate_rejections["facts"].values())
|
||||
metrics["rejected_by_scope_gate"] = sum(count for reasons in scope_gate_rejections.values() for count in reasons.values())
|
||||
metrics["scope_gate_rejections"] = scope_gate_rejections
|
||||
|
||||
return current_memory
|
||||
|
||||
@ -694,6 +694,8 @@ def _host_default_extraction_callback(payload: Any) -> None:
|
||||
extracted = payload.get("facts_extracted")
|
||||
passed_confidence = payload.get("facts_passed_confidence")
|
||||
rejected = payload.get("rejected_low_confidence", 0)
|
||||
rejected_by_scope = payload.get("rejected_by_scope_gate", 0)
|
||||
scope_breakdown = payload.get("scope_gate_rejections")
|
||||
thread_id = payload.get("thread_id")
|
||||
model_name = payload.get("model_name")
|
||||
if isinstance(extracted, int) and isinstance(passed_confidence, int) and extracted > 0:
|
||||
@ -721,6 +723,22 @@ def _host_default_extraction_callback(payload: Any) -> None:
|
||||
payload.get("success"),
|
||||
payload.get("token_usage"),
|
||||
)
|
||||
if isinstance(scope_breakdown, dict):
|
||||
logger.info(
|
||||
"Memory scope-gate metrics: thread=%s model=%s rejected=%s breakdown=%s",
|
||||
thread_id,
|
||||
model_name,
|
||||
rejected_by_scope,
|
||||
scope_breakdown,
|
||||
)
|
||||
fact_breakdown = scope_breakdown.get("facts")
|
||||
fact_scope_rejected = sum(value for value in fact_breakdown.values() if isinstance(value, int)) if isinstance(fact_breakdown, dict) else 0
|
||||
if isinstance(extracted, int) and extracted > 0 and fact_scope_rejected / extracted > 0.6:
|
||||
logger.warning(
|
||||
"Memory fact scope-gate rejection rate %.0f%% exceeds 60%% - review extraction model classification / prompt (thread=%s)",
|
||||
fact_scope_rejected / extracted * 100,
|
||||
thread_id,
|
||||
)
|
||||
|
||||
|
||||
def _collect_host_hooks() -> dict[str, Any]:
|
||||
|
||||
@ -240,7 +240,7 @@ def test_trace_id_threads_through_to_callbacks(deermem_data_dir):
|
||||
|
||||
|
||||
def test_default_passive_update_persists_fact_in_reserved_default_bucket(deermem_data_dir):
|
||||
dm = _deermem_with_fake_llm(payload='{"user":{},"history":{},"newFacts":[{"content":"Default agent fact","category":"context","confidence":0.9}],"factsToRemove":[]}')
|
||||
dm = _deermem_with_fake_llm(payload='{"user":{},"history":{},"newFacts":[{"content":"Default agent fact","category":"context","confidence":0.9,"scope":"user","durability":"durable","authority":"descriptive"}],"factsToRemove":[]}')
|
||||
|
||||
dm.add(
|
||||
thread_id="default-thread",
|
||||
|
||||
@ -35,6 +35,12 @@ from deerflow.agents.memory.backends.deermem.deermem.core.updater import (
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
_DURABLE_USER_CLASSIFICATION = {
|
||||
"scope": "user",
|
||||
"durability": "durable",
|
||||
"authority": "descriptive",
|
||||
}
|
||||
|
||||
|
||||
def _memory_config(**overrides: object) -> DeerMemConfig:
|
||||
"""Build a DeerMemConfig with test overrides (validation bypassed via setattr).
|
||||
@ -247,6 +253,7 @@ class TestNormalizeFactsToConsolidate:
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {
|
||||
**_DURABLE_USER_CLASSIFICATION,
|
||||
"content": "User is a full-stack engineer",
|
||||
"category": "knowledge",
|
||||
"confidence": 0.9,
|
||||
@ -287,7 +294,7 @@ class TestNormalizeFactsToConsolidate:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_only"],
|
||||
"consolidated": {"content": "should be skipped", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "should be skipped", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -304,7 +311,7 @@ class TestNormalizeFactsToConsolidate:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": " ", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": " ", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -359,6 +366,7 @@ class TestApplyUpdatesConsolidation:
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b", "fact_c"],
|
||||
"consolidated": {
|
||||
**_DURABLE_USER_CLASSIFICATION,
|
||||
"content": "Full-stack: React frontend, Python backend, PostgreSQL",
|
||||
"category": "knowledge",
|
||||
"confidence": 0.9,
|
||||
@ -398,9 +406,9 @@ class TestApplyUpdatesConsolidation:
|
||||
"factsToRemove": [],
|
||||
"staleFactsToRemove": [],
|
||||
"factsToConsolidate": [
|
||||
{"sourceIds": ["f_0", "f_1"], "consolidated": {"content": "Group 1", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["f_2", "f_3"], "consolidated": {"content": "Group 2", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["f_4", "f_5"], "consolidated": {"content": "Group 3", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["f_0", "f_1"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Group 1", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["f_2", "f_3"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Group 2", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["f_4", "f_5"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Group 3", "category": "knowledge", "confidence": 0.8}},
|
||||
],
|
||||
}
|
||||
|
||||
@ -433,7 +441,7 @@ class TestApplyUpdatesConsolidation:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_hallucinated"],
|
||||
"consolidated": {"content": "Should not apply", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Should not apply", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -461,7 +469,7 @@ class TestApplyUpdatesConsolidation:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": [f"f_{i}" for i in range(10)], # 10 sources, cap is 5
|
||||
"consolidated": {"content": "Over-merged", "category": "knowledge", "confidence": 0.8},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Over-merged", "category": "knowledge", "confidence": 0.8},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -494,8 +502,8 @@ class TestApplyUpdatesConsolidation:
|
||||
"factsToRemove": [],
|
||||
"staleFactsToRemove": [],
|
||||
"factsToConsolidate": [
|
||||
{"sourceIds": ["fact_a", "fact_b"], "consolidated": {"content": "AB", "category": "knowledge", "confidence": 0.9}},
|
||||
{"sourceIds": ["fact_b", "fact_c"], "consolidated": {"content": "BC", "category": "knowledge", "confidence": 0.8}},
|
||||
{"sourceIds": ["fact_a", "fact_b"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "AB", "category": "knowledge", "confidence": 0.9}},
|
||||
{"sourceIds": ["fact_b", "fact_c"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "BC", "category": "knowledge", "confidence": 0.8}},
|
||||
],
|
||||
}
|
||||
|
||||
@ -529,10 +537,10 @@ class TestApplyUpdatesConsolidation:
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": ["fact_contradicted"],
|
||||
"factsToRemove": [{"id": "fact_contradicted", "scope": "user", "reason": "Explicit contradiction in test fixture"}],
|
||||
"staleFactsToRemove": [{"id": "fact_stale", "reason": "outdated"}],
|
||||
"factsToConsolidate": [
|
||||
{"sourceIds": ["fact_a", "fact_b"], "consolidated": {"content": "React + Python", "category": "knowledge", "confidence": 0.9}},
|
||||
{"sourceIds": ["fact_a", "fact_b"], "consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "React + Python", "category": "knowledge", "confidence": 0.9}},
|
||||
],
|
||||
}
|
||||
|
||||
@ -559,7 +567,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_a"],
|
||||
"consolidated": {"content": "Rewritten", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Rewritten", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -596,11 +604,11 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "Merged", "category": " knowledge ", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged", "category": " knowledge ", "confidence": 0.9},
|
||||
},
|
||||
{
|
||||
"sourceIds": ["fact_c", "fact_d"],
|
||||
"consolidated": {"content": "Also merged", "category": " ", "confidence": 0.85},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Also merged", "category": " ", "confidence": 0.85},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -632,15 +640,15 @@ class TestReviewerFindings:
|
||||
"newFacts": [
|
||||
# 2 high-confidence new facts that push us to max_facts=3,
|
||||
# forcing the trim to evict low_a and low_b
|
||||
{"content": "New high 1", "category": "knowledge", "confidence": 0.98},
|
||||
{"content": "New high 2", "category": "knowledge", "confidence": 0.97},
|
||||
{**_DURABLE_USER_CLASSIFICATION, "content": "New high 1", "category": "knowledge", "confidence": 0.98},
|
||||
{**_DURABLE_USER_CLASSIFICATION, "content": "New high 2", "category": "knowledge", "confidence": 0.97},
|
||||
],
|
||||
"factsToRemove": [],
|
||||
"staleFactsToRemove": [],
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["low_a", "low_b"],
|
||||
"consolidated": {"content": "Merged low", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged low", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -680,7 +688,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "Merged AB", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged AB", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -714,7 +722,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["corr_0", "corr_1"],
|
||||
"consolidated": {"content": "Merged corrections", "category": "correction", "confidence": 0.95},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged corrections", "category": "correction", "confidence": 0.95},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -752,7 +760,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "Merged", "category": "knowledge", "confidence": 1.0},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged", "category": "knowledge", "confidence": 1.0},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -777,7 +785,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_c", "fact_d"],
|
||||
"consolidated": {"content": "Below threshold", "category": "knowledge", "confidence": 1.0},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Below threshold", "category": "knowledge", "confidence": 1.0},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -809,7 +817,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "Should not merge", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Should not merge", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -852,7 +860,7 @@ class TestReviewerFindings:
|
||||
{
|
||||
"sourceIds": ["fact_null", "fact_b"],
|
||||
# LLM returns 1.0; cap = max(0.5, 0.9) = 0.9
|
||||
"consolidated": {"content": "Merged", "category": "knowledge", "confidence": 1.0},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged", "category": "knowledge", "confidence": 1.0},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -888,7 +896,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_old", "fact_new"],
|
||||
"consolidated": {"content": "Old and new merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Old and new merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -935,7 +943,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "A and B merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "A and B merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -975,7 +983,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1021,7 +1029,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1069,7 +1077,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_legacy", "fact_stable"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1128,7 +1136,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_stable", "fact_volatile"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1176,7 +1184,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_bad", "fact_stable"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1226,7 +1234,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_bad", "fact_stable"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1273,7 +1281,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_old", "fact_fresh"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1316,7 +1324,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_stable", "fact_volatile"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1358,7 +1366,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1400,7 +1408,7 @@ class TestReviewerFindings:
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {"content": "merged stable skill", "category": "knowledge", "confidence": 0.9},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "merged stable skill", "category": "knowledge", "confidence": 0.9},
|
||||
},
|
||||
],
|
||||
}
|
||||
@ -1440,7 +1448,7 @@ class TestReviewerFindings:
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
# LLM omits the confidence field entirely
|
||||
"consolidated": {"content": "Merged without confidence", "category": "knowledge"},
|
||||
"consolidated": {**_DURABLE_USER_CLASSIFICATION, "content": "Merged without confidence", "category": "knowledge"},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
411
backend/tests/test_memory_scope_gate.py
Normal file
411
backend/tests/test_memory_scope_gate.py
Normal file
@ -0,0 +1,411 @@
|
||||
import copy
|
||||
import logging
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig
|
||||
from deerflow.agents.memory.backends.deermem.deermem.core.storage import MemoryStorage
|
||||
from deerflow.agents.memory.backends.deermem.deermem.core.updater import MemoryUpdater, _extract_text, _normalize_memory_update_data
|
||||
from deerflow.agents.memory.manager import _host_default_extraction_callback
|
||||
|
||||
|
||||
def _memory(facts: list[dict[str, object]] | None = None) -> dict[str, object]:
|
||||
return {
|
||||
"version": "1.0",
|
||||
"revision": 0,
|
||||
"lastUpdated": "",
|
||||
"user": {
|
||||
"workContext": {"summary": "", "updatedAt": ""},
|
||||
"personalContext": {"summary": "", "updatedAt": ""},
|
||||
"topOfMind": {"summary": "", "updatedAt": ""},
|
||||
},
|
||||
"history": {
|
||||
"recentMonths": {"summary": "", "updatedAt": ""},
|
||||
"earlierContext": {"summary": "", "updatedAt": ""},
|
||||
"longTermBackground": {"summary": "", "updatedAt": ""},
|
||||
},
|
||||
"facts": copy.deepcopy(facts or []),
|
||||
}
|
||||
|
||||
|
||||
class _Storage(MemoryStorage):
|
||||
def load(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, object]:
|
||||
return _memory()
|
||||
|
||||
def reload(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, object]:
|
||||
return self.load(agent_name, user_id=user_id)
|
||||
|
||||
def save(
|
||||
self,
|
||||
memory_data: dict[str, object],
|
||||
agent_name: str | None = None,
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
expected_revision: int | None = None,
|
||||
) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _updater(**config_overrides: object) -> MemoryUpdater:
|
||||
config = DeerMemConfig()
|
||||
for key, value in config_overrides.items():
|
||||
setattr(config, key, value)
|
||||
return MemoryUpdater(config, _Storage(), llm=None)
|
||||
|
||||
|
||||
def _fact(content: str, **overrides: object) -> dict[str, object]:
|
||||
fact: dict[str, object] = {
|
||||
"content": content,
|
||||
"category": "preference",
|
||||
"confidence": 0.9,
|
||||
"scope": "user",
|
||||
"durability": "durable",
|
||||
"authority": "descriptive",
|
||||
}
|
||||
fact.update(overrides)
|
||||
return fact
|
||||
|
||||
|
||||
def _stored_fact(fact_id: str, content: str) -> dict[str, object]:
|
||||
return {
|
||||
"id": fact_id,
|
||||
"content": content,
|
||||
"category": "preference",
|
||||
"confidence": 0.9,
|
||||
"createdAt": "2026-01-01T00:00:00Z",
|
||||
"source": "thread-old",
|
||||
}
|
||||
|
||||
|
||||
def test_normalize_preserves_extraction_only_classification_fields() -> None:
|
||||
normalized = _normalize_memory_update_data(
|
||||
{
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [_fact("User prefers concise answers")],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
)
|
||||
|
||||
assert normalized["newFacts"][0]["scope"] == "user"
|
||||
assert normalized["newFacts"][0]["durability"] == "durable"
|
||||
assert normalized["newFacts"][0]["authority"] == "descriptive"
|
||||
|
||||
|
||||
def test_fact_gate_accepts_only_durable_descriptive_user_facts() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7)
|
||||
metrics: dict[str, object] = {}
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"factsToRemove": [],
|
||||
"newFacts": [
|
||||
_fact("accepted"),
|
||||
_fact("thread only", scope="thread"),
|
||||
_fact("temporary", durability="temporary"),
|
||||
_fact("permission", authority="transactional"),
|
||||
{"content": "missing labels", "category": "context", "confidence": 0.9},
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(_memory(), update, thread_id="thread-new", metrics=metrics)
|
||||
|
||||
assert [fact["content"] for fact in result["facts"]] == ["accepted"]
|
||||
assert set(result["facts"][0]) == {"id", "content", "category", "confidence", "createdAt", "source"}
|
||||
assert metrics["rejected_by_scope_gate"] == 4
|
||||
assert metrics["scope_gate_rejections"] == {
|
||||
"facts": {"missing": 1, "scope": 1, "durability": 1, "authority": 1},
|
||||
"summaries": {"missing": 0, "scope": 0, "authority": 0},
|
||||
"removals": {"missing": 0, "scope": 0, "replacement": 0},
|
||||
"consolidations": {"missing": 0, "scope": 0, "durability": 0, "authority": 0},
|
||||
}
|
||||
|
||||
|
||||
def test_summary_gate_requires_user_scope_and_descriptive_authority() -> None:
|
||||
updater = _updater()
|
||||
current = _memory()
|
||||
current["user"]["personalContext"]["summary"] = "Existing summary"
|
||||
metrics: dict[str, object] = {}
|
||||
update = {
|
||||
"user": {
|
||||
"workContext": {"summary": "User is a software engineer", "shouldUpdate": True, "scope": "user", "authority": "descriptive"},
|
||||
"personalContext": {"summary": "Constraint for this PR", "shouldUpdate": True, "scope": "project", "authority": "descriptive"},
|
||||
"topOfMind": {"summary": "User granted push access", "shouldUpdate": True, "scope": "user", "authority": "transactional"},
|
||||
},
|
||||
"history": {
|
||||
"recentMonths": {"summary": "Missing authority label", "shouldUpdate": True, "scope": "user"},
|
||||
},
|
||||
"newFacts": [],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update, metrics=metrics)
|
||||
|
||||
assert result["user"]["workContext"]["summary"] == "User is a software engineer"
|
||||
assert set(result["user"]["workContext"]) == {"summary", "updatedAt"}
|
||||
assert result["user"]["personalContext"]["summary"] == "Existing summary"
|
||||
assert result["user"]["topOfMind"]["summary"] == ""
|
||||
assert result["history"]["recentMonths"]["summary"] == ""
|
||||
assert metrics["scope_gate_rejections"]["summaries"] == {"missing": 1, "scope": 1, "authority": 1}
|
||||
|
||||
|
||||
def test_thread_scoped_removal_cannot_delete_user_fact() -> None:
|
||||
updater = _updater()
|
||||
current = _memory([_stored_fact("fact_api", "User generally prefers API compatibility")])
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": [{"id": "fact_api", "scope": "thread", "reason": "This PR may break the API"}],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update)
|
||||
|
||||
assert [fact["id"] for fact in result["facts"]] == ["fact_api"]
|
||||
|
||||
|
||||
def test_unreasoned_user_scoped_removal_fails_closed() -> None:
|
||||
updater = _updater()
|
||||
current = _memory([_stored_fact("fact_api", "User generally prefers API compatibility")])
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": [{"id": "fact_api", "scope": "user"}],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update)
|
||||
|
||||
assert [fact["id"] for fact in result["facts"]] == ["fact_api"]
|
||||
|
||||
|
||||
def test_paired_removal_is_skipped_when_replacement_fails_scope_gate() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7)
|
||||
current = _memory([_stored_fact("fact_api", "User generally prefers API compatibility")])
|
||||
metrics: dict[str, object] = {}
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [_fact("This PR may break the API", scope="thread", durability="temporary")],
|
||||
"factsToRemove": [
|
||||
{
|
||||
"id": "fact_api",
|
||||
"scope": "user",
|
||||
"reason": "Preference changed",
|
||||
"replacementFactIndex": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update, metrics=metrics)
|
||||
|
||||
assert [fact["id"] for fact in result["facts"]] == ["fact_api"]
|
||||
assert metrics["scope_gate_rejections"]["removals"]["replacement"] == 1
|
||||
|
||||
|
||||
def test_paired_removal_is_skipped_when_replacement_fails_confidence_gate() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7)
|
||||
current = _memory([_stored_fact("fact_api", "User generally prefers API compatibility")])
|
||||
metrics: dict[str, object] = {}
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [_fact("User no longer requires API compatibility", confidence=0.69)],
|
||||
"factsToRemove": [
|
||||
{
|
||||
"id": "fact_api",
|
||||
"scope": "user",
|
||||
"reason": "Preference changed",
|
||||
"replacementFactIndex": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update, metrics=metrics)
|
||||
|
||||
assert [fact["id"] for fact in result["facts"]] == ["fact_api"]
|
||||
assert metrics["facts_passed_scope_gate"] == 1
|
||||
assert metrics["rejected_low_confidence"] == 1
|
||||
assert metrics["scope_gate_rejections"]["removals"]["replacement"] == 1
|
||||
|
||||
|
||||
def test_paired_removal_is_atomic_when_replacement_is_persisted() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7, max_facts=100)
|
||||
current = _memory([_stored_fact("fact_editor", "User prefers Vim")])
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [_fact("User now prefers VS Code")],
|
||||
"factsToRemove": [
|
||||
{
|
||||
"id": "fact_editor",
|
||||
"scope": "user",
|
||||
"reason": "User changed their durable editor preference",
|
||||
"replacementFactIndex": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update)
|
||||
|
||||
assert [fact["content"] for fact in result["facts"]] == ["User now prefers VS Code"]
|
||||
|
||||
|
||||
def test_paired_removal_is_skipped_when_replacement_is_trimmed() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7, max_facts=1)
|
||||
current = _memory([_stored_fact("fact_editor", "User prefers Vim")])
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [_fact("User now prefers VS Code", confidence=0.8)],
|
||||
"factsToRemove": [
|
||||
{
|
||||
"id": "fact_editor",
|
||||
"scope": "user",
|
||||
"reason": "User changed their durable editor preference",
|
||||
"replacementFactIndex": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update)
|
||||
|
||||
assert [fact["content"] for fact in result["facts"]] == ["User prefers Vim"]
|
||||
|
||||
|
||||
def test_missing_classification_rejects_one_fact_without_aborting_other_updates() -> None:
|
||||
updater = _updater(fact_confidence_threshold=0.7, max_facts=100)
|
||||
current = _memory([_stored_fact("fact_old", "Old durable preference")])
|
||||
normalized = _normalize_memory_update_data(
|
||||
{
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [
|
||||
_fact("Accepted durable preference"),
|
||||
{"content": "Unclassified replacement", "category": "preference", "confidence": 0.9},
|
||||
],
|
||||
"factsToRemove": [
|
||||
{
|
||||
"id": "fact_old",
|
||||
"scope": "user",
|
||||
"reason": "Replace it",
|
||||
"replacementFactIndex": 1,
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
result = updater._apply_updates(current, normalized)
|
||||
|
||||
assert {fact["content"] for fact in result["facts"]} == {
|
||||
"Old durable preference",
|
||||
"Accepted durable preference",
|
||||
}
|
||||
|
||||
|
||||
def test_legacy_string_removal_fails_closed_and_reports_missing_scope() -> None:
|
||||
updater = _updater()
|
||||
current = _memory([_stored_fact("fact_api", "User generally prefers API compatibility")])
|
||||
metrics: dict[str, object] = {}
|
||||
normalized = _normalize_memory_update_data(
|
||||
{
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": ["fact_api"],
|
||||
}
|
||||
)
|
||||
|
||||
result = updater._apply_updates(current, normalized, metrics=metrics)
|
||||
|
||||
assert normalized["factsToRemove"] == [{"id": "fact_api"}]
|
||||
assert [fact["id"] for fact in result["facts"]] == ["fact_api"]
|
||||
assert metrics["scope_gate_rejections"]["removals"]["missing"] == 1
|
||||
|
||||
|
||||
def test_signal_hints_require_cross_task_user_scope() -> None:
|
||||
updater = _updater()
|
||||
|
||||
hints = updater._build_signal_hints(frozenset({"correction", "goal", "decision"}))
|
||||
|
||||
assert "user-level" in hints
|
||||
assert "current task" in hints
|
||||
assert "thread" in hints
|
||||
|
||||
|
||||
def test_prompt_requires_scope_labels_for_every_mutating_path() -> None:
|
||||
updater = _updater()
|
||||
message = MagicMock()
|
||||
message.type = "human"
|
||||
message.content = "Remember that I prefer concise answers."
|
||||
|
||||
prepared = updater._prepare_update_prompt([message], agent_name="lead-agent", signals=frozenset())
|
||||
|
||||
assert prepared is not None
|
||||
_, prompt = prepared
|
||||
prompt_text = "\n".join(_extract_text(getattr(item, "content", item)) for item in prompt)
|
||||
assert 'scope="user"' in prompt_text
|
||||
assert 'durability="durable"' in prompt_text
|
||||
assert 'authority="transactional"' in prompt_text
|
||||
assert '"scope": "user|thread|project", "authority": "descriptive|transactional"' in prompt_text
|
||||
assert "replacementFactIndex" in prompt_text
|
||||
assert "unrelated future thread" in prompt_text
|
||||
|
||||
|
||||
def test_unclassified_consolidation_keeps_sources_and_reports_rejection() -> None:
|
||||
updater = _updater(
|
||||
consolidation_enabled=True,
|
||||
consolidation_min_facts=2,
|
||||
consolidation_max_groups_per_cycle=1,
|
||||
consolidation_max_sources=4,
|
||||
)
|
||||
current = _memory(
|
||||
[
|
||||
_stored_fact("fact_a", "User uses Python"),
|
||||
_stored_fact("fact_b", "User uses Rust"),
|
||||
]
|
||||
)
|
||||
metrics: dict[str, object] = {}
|
||||
update = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": [],
|
||||
"factsToConsolidate": [
|
||||
{
|
||||
"sourceIds": ["fact_a", "fact_b"],
|
||||
"consolidated": {
|
||||
"content": "User uses Python and Rust",
|
||||
"category": "preference",
|
||||
"confidence": 0.9,
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = updater._apply_updates(current, update, metrics=metrics)
|
||||
|
||||
assert {fact["id"] for fact in result["facts"]} == {"fact_a", "fact_b"}
|
||||
assert metrics["scope_gate_rejections"]["consolidations"]["missing"] == 1
|
||||
|
||||
|
||||
def test_default_observability_warns_when_fact_scope_gate_rejects_most_items(caplog) -> None:
|
||||
payload = {
|
||||
"thread_id": "thread-scope",
|
||||
"model_name": "test-model",
|
||||
"success": True,
|
||||
"facts_extracted": 2,
|
||||
"facts_passed_confidence": 2,
|
||||
"rejected_low_confidence": 0,
|
||||
"rejected_by_scope_gate": 2,
|
||||
"scope_gate_rejections": {
|
||||
"facts": {"missing": 2, "scope": 0, "durability": 0, "authority": 0},
|
||||
"summaries": {"missing": 0, "scope": 0, "authority": 0},
|
||||
"removals": {"missing": 0, "scope": 0, "replacement": 0},
|
||||
"consolidations": {"missing": 0, "scope": 0, "durability": 0, "authority": 0},
|
||||
},
|
||||
}
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
_host_default_extraction_callback(payload)
|
||||
|
||||
assert "scope-gate rejection rate 100% exceeds 60%" in caplog.text
|
||||
@ -33,6 +33,12 @@ from deerflow.agents.memory.backends.deermem.deermem.core.updater import (
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
_DURABLE_USER_FACT = {
|
||||
"scope": "user",
|
||||
"durability": "durable",
|
||||
"authority": "descriptive",
|
||||
}
|
||||
|
||||
|
||||
def _memory_config(**overrides: object) -> DeerMemConfig:
|
||||
config = DeerMemConfig()
|
||||
@ -569,7 +575,7 @@ class TestApplyUpdatesStaleness:
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [],
|
||||
"factsToRemove": ["fact_contradicted"],
|
||||
"factsToRemove": [{"id": "fact_contradicted", "scope": "user", "reason": "Explicit contradiction in test fixture"}],
|
||||
"staleFactsToRemove": [{"id": "fact_stale", "reason": "old"}],
|
||||
}
|
||||
|
||||
@ -924,7 +930,7 @@ class TestNewFactsExpectedValidDays:
|
||||
update_data = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [{"content": "User speaks Spanish natively", "category": "knowledge", "confidence": 0.95, "expected_valid_days": 180}],
|
||||
"newFacts": [{**_DURABLE_USER_FACT, "content": "User speaks Spanish natively", "category": "knowledge", "confidence": 0.95, "expected_valid_days": 180}],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
|
||||
@ -947,7 +953,7 @@ class TestNewFactsExpectedValidDays:
|
||||
update_data = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [{"content": "User prefers Python", "category": "knowledge", "confidence": 0.9, "expected_valid_days": 3650}],
|
||||
"newFacts": [{**_DURABLE_USER_FACT, "content": "User prefers Python", "category": "knowledge", "confidence": 0.9, "expected_valid_days": 3650}],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
|
||||
@ -962,7 +968,7 @@ class TestNewFactsExpectedValidDays:
|
||||
update_data = {
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [{"content": "User uses Python", "category": "knowledge", "confidence": 0.9}],
|
||||
"newFacts": [{**_DURABLE_USER_FACT, "content": "User uses Python", "category": "knowledge", "confidence": 0.9}],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
|
||||
|
||||
@ -48,6 +48,13 @@ def _memory_config(**overrides: object) -> DeerMemConfig:
|
||||
return config
|
||||
|
||||
|
||||
_DURABLE_USER_FACT = {
|
||||
"scope": "user",
|
||||
"durability": "durable",
|
||||
"authority": "descriptive",
|
||||
}
|
||||
|
||||
|
||||
class _MemoryStorage(MemoryStorage):
|
||||
def __init__(self, memory: dict[str, object] | None = None, *, save_result: bool = True):
|
||||
self.memory = copy.deepcopy(memory or _make_memory())
|
||||
@ -119,9 +126,9 @@ def test_apply_updates_skips_existing_duplicate_and_preserves_removals() -> None
|
||||
]
|
||||
)
|
||||
update_data = {
|
||||
"factsToRemove": ["fact_remove"],
|
||||
"factsToRemove": [{"id": "fact_remove", "scope": "user", "reason": "Explicit retraction in test fixture"}],
|
||||
"newFacts": [
|
||||
{"content": "User likes Python", "category": "preference", "confidence": 0.95},
|
||||
{**_DURABLE_USER_FACT, "content": "User likes Python", "category": "preference", "confidence": 0.95},
|
||||
],
|
||||
}
|
||||
|
||||
@ -136,8 +143,8 @@ def test_apply_updates_skips_whitespace_only_facts() -> None:
|
||||
current_memory = _make_memory()
|
||||
update_data = {
|
||||
"newFacts": [
|
||||
{"content": " ", "category": "context", "confidence": 0.9},
|
||||
{"content": "User prefers dark mode", "category": "preference", "confidence": 0.9},
|
||||
{**_DURABLE_USER_FACT, "content": " ", "category": "context", "confidence": 0.9},
|
||||
{**_DURABLE_USER_FACT, "content": "User prefers dark mode", "category": "preference", "confidence": 0.9},
|
||||
],
|
||||
}
|
||||
|
||||
@ -226,9 +233,9 @@ def test_apply_updates_skips_same_batch_duplicates_and_keeps_source_metadata() -
|
||||
current_memory = _make_memory()
|
||||
update_data = {
|
||||
"newFacts": [
|
||||
{"content": "User prefers dark mode", "category": "preference", "confidence": 0.91},
|
||||
{"content": "User prefers dark mode", "category": "preference", "confidence": 0.92},
|
||||
{"content": "User works on DeerFlow", "category": "context", "confidence": 0.87},
|
||||
{**_DURABLE_USER_FACT, "content": "User prefers dark mode", "category": "preference", "confidence": 0.91},
|
||||
{**_DURABLE_USER_FACT, "content": "User prefers dark mode", "category": "preference", "confidence": 0.92},
|
||||
{**_DURABLE_USER_FACT, "content": "User works on DeerFlow", "category": "context", "confidence": 0.87},
|
||||
],
|
||||
}
|
||||
|
||||
@ -266,9 +273,9 @@ def test_apply_updates_preserves_threshold_and_max_facts_trimming() -> None:
|
||||
)
|
||||
update_data = {
|
||||
"newFacts": [
|
||||
{"content": "User prefers dark mode", "category": "preference", "confidence": 0.9},
|
||||
{"content": "User uses uv", "category": "context", "confidence": 0.85},
|
||||
{"content": "User likes noisy logs", "category": "behavior", "confidence": 0.6},
|
||||
{**_DURABLE_USER_FACT, "content": "User prefers dark mode", "category": "preference", "confidence": 0.9},
|
||||
{**_DURABLE_USER_FACT, "content": "User uses uv", "category": "context", "confidence": 0.85},
|
||||
{**_DURABLE_USER_FACT, "content": "User likes noisy logs", "category": "behavior", "confidence": 0.6},
|
||||
],
|
||||
}
|
||||
|
||||
@ -292,6 +299,7 @@ def test_apply_updates_preserves_source_error() -> None:
|
||||
"category": "correction",
|
||||
"confidence": 0.95,
|
||||
"sourceError": "The agent previously suggested npm start.",
|
||||
**_DURABLE_USER_FACT,
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -312,6 +320,7 @@ def test_apply_updates_ignores_empty_source_error() -> None:
|
||||
"category": "correction",
|
||||
"confidence": 0.95,
|
||||
"sourceError": " ",
|
||||
**_DURABLE_USER_FACT,
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -936,7 +945,9 @@ class TestUpdateMemoryStructuredResponse:
|
||||
|
||||
def test_wrapped_json_responses_parse(self):
|
||||
"""Memory update should tolerate provider wrappers around valid JSON."""
|
||||
valid_json = '{"user": {}, "history": {}, "newFacts": [{"content": "User prefers concise updates", "category": "preference", "confidence": 0.9}], "factsToRemove": []}'
|
||||
valid_json = (
|
||||
'{"user": {}, "history": {}, "newFacts": [{"content": "User prefers concise updates", "category": "preference", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}], "factsToRemove": []}'
|
||||
)
|
||||
response_variants = [
|
||||
f"<think>Analyze the conversation first.</think>\n{valid_json}",
|
||||
f"<think>Analyze the conversation first.\n{valid_json}",
|
||||
@ -953,7 +964,7 @@ class TestUpdateMemoryStructuredResponse:
|
||||
|
||||
def test_ignores_unrelated_json_before_memory_update(self):
|
||||
"""Parser should not select unrelated JSON objects before the memory update."""
|
||||
valid_json = '{"user": {}, "history": {}, "newFacts": [{"content": "Remember the actual update", "category": "context", "confidence": 0.9}], "factsToRemove": []}'
|
||||
valid_json = '{"user": {}, "history": {}, "newFacts": [{"content": "Remember the actual update", "category": "context", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}], "factsToRemove": []}'
|
||||
response = f'Example object: {{"user": "alice"}}\nActual memory update:\n{valid_json}'
|
||||
|
||||
result, storage = self._run_update_with_response(response)
|
||||
@ -970,7 +981,11 @@ class TestUpdateMemoryStructuredResponse:
|
||||
|
||||
def test_schema_guard_ignores_invalid_update_fields(self):
|
||||
"""Parsed JSON with bad field types should not break the memory update."""
|
||||
response = '{"user": "bad", "history": [], "newFacts": ["bad", {"content": "User works on DeerFlow", "category": "context", "confidence": 0.91}], "factsToRemove": "bad"}'
|
||||
response = (
|
||||
'{"user": "bad", "history": [], "newFacts": ["bad", '
|
||||
'{"content": "User works on DeerFlow", "category": "context", "confidence": 0.91, '
|
||||
'"scope": "user", "durability": "durable", "authority": "descriptive"}], "factsToRemove": "bad"}'
|
||||
)
|
||||
|
||||
result, storage = self._run_update_with_response(response)
|
||||
|
||||
@ -981,7 +996,7 @@ class TestUpdateMemoryStructuredResponse:
|
||||
"""Malformed fact entries should be normalized per fact, not fail the whole update."""
|
||||
response = (
|
||||
'{"user": {}, "history": {}, "newFacts": ['
|
||||
'{"content": " User likes async updates ", "category": 9, "confidence": "0.91", "sourceError": " parse issue "}, '
|
||||
'{"content": " User likes async updates ", "category": 9, "confidence": "0.91", "sourceError": " parse issue ", "scope": "user", "durability": "durable", "authority": "descriptive"}, '
|
||||
'{"content": "skip invalid confidence", "category": "context", "confidence": "high"}, '
|
||||
'{"content": 12, "category": "context", "confidence": 0.9}, '
|
||||
'{"content": " ", "category": "context", "confidence": 0.9}'
|
||||
@ -1175,7 +1190,7 @@ class TestFactDeduplicationCaseInsensitive:
|
||||
update_data = {
|
||||
"factsToRemove": [],
|
||||
"newFacts": [
|
||||
{"content": "user prefers python", "category": "preference", "confidence": 0.95},
|
||||
{**_DURABLE_USER_FACT, "content": "user prefers python", "category": "preference", "confidence": 0.95},
|
||||
],
|
||||
}
|
||||
|
||||
@ -1202,7 +1217,7 @@ class TestFactDeduplicationCaseInsensitive:
|
||||
update_data = {
|
||||
"factsToRemove": [],
|
||||
"newFacts": [
|
||||
{"content": "User prefers Go", "category": "preference", "confidence": 0.85},
|
||||
{**_DURABLE_USER_FACT, "content": "User prefers Go", "category": "preference", "confidence": 0.85},
|
||||
],
|
||||
}
|
||||
|
||||
@ -1293,7 +1308,7 @@ class TestFinalizeCacheIsolation:
|
||||
{
|
||||
"user": {},
|
||||
"history": {},
|
||||
"newFacts": [{"content": "new fact", "category": "context", "confidence": 0.9}],
|
||||
"newFacts": [{**_DURABLE_USER_FACT, "content": "new fact", "category": "context", "confidence": 0.9}],
|
||||
"factsToRemove": [],
|
||||
}
|
||||
)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user