diff --git a/README.md b/README.md index 913d6d982..692046782 100644 --- a/README.md +++ b/README.md @@ -810,6 +810,23 @@ Across sessions, DeerFlow builds a persistent memory of your profile, preference Memory updates now skip duplicate fact entries at apply time, so repeated preferences and context do not accumulate endlessly across sessions. +File-backed memory now separates global user context from agent facts. Each user has one `memory.json` containing only the project-independent `user` and `history` summaries; every fact is a canonical Markdown file below `agents/{agent_name}/facts/`. Existing lead-agent middleware, API, Settings, import/export, and embedded-client calls that omit `agent_name` resolve inside DeerMem to the reserved `__default__` bucket. That bucket is outside the valid custom-agent name grammar, so a real custom agent named `lead-agent` has a separate fact repository and deleting a custom agent cannot delete a memory-only directory without `config.yaml`. Public agent identifiers are case-insensitive and canonicalized to lowercase. Runtime/API readers still receive a compatibility `facts` array for the selected/default agent, so the frontend does not read agent facts from `memory.json`; structured Markdown `source` metadata is projected to the historical string field at the MemoryManager boundary. An unscoped Clear All first migrates facts from unread legacy per-agent JSON without adopting its soon-to-be-cleared summaries, then removes shared summaries and facts from every agent bucket while preserving agent configuration files, so a later read cannot resurrect skipped legacy facts; an explicitly agent-scoped clear removes only that agent's facts. On first normal read, old facts embedded in the user JSON are migrated automatically to `__default__`; facts written to the earlier implicit `lead-agent` bucket are also moved when that directory is not a real custom agent. Migration and normal writes notify a configured retrieval adapter only after durable storage locks are released. Journaled writes, a shared user lock, and optimistic user-memory revisions prevent silent lost updates. Retrieval engines remain optional behind the memory retrieval-adapter contract, with an explicit local substring fallback when no adapter is configured. + +Single-fact repository operations are genuinely incremental: an upsert/delete reads, journals, writes, and re-indexes only the addressed fact files, and returns an explicit incomplete delta rather than a cache-dependent fake full document. Summary change sets merge the supplied `user`/`history` child keys over the persisted sections so a partial update cannot erase omitted siblings; full imports normalize both sections to the complete compatibility schema before applying replacement values. Manager/API compatibility methods materialize a fresh full document only when their public response contract requires one. Fact-level point operations use separate expected user-memory and fact revisions and may explicitly rebase when every addressed fact precondition still holds. Snapshot-derived operations such as scoped clear, capped create, consolidation, and trimming never replay stale delete/trim sets: a manifest conflict reloads the complete document and recomputes the operation, with a bounded retry. Fact paths use the first two hexadecimal characters of `SHA-256(fact_id)` so generated `fact_*` IDs distribute across shards. The cache token combines the shared JSON's nanosecond mtime, size, and persisted revision; this prevents coarse-mtime same-size writes from returning stale data without scanning fact files. Direct out-of-band Markdown edits require an explicit reload. Storage-specific conflicts and corruption are translated at the MemoryManager boundary; the Gateway returns conflict as HTTP 409 and a stable, non-sensitive corruption error as HTTP 500. Full-document `save()` remains a compatibility API and computes a diff before writing; malformed or missing `facts` can no longer silently erase an agent's Markdown files. Legacy migration preserves non-empty `user`/`history` before deleting an agent `memory.json`; conflicting summaries keep the legacy file and fail loudly instead of choosing a winner. + +Legacy facts in `memory.json` migrate automatically into the reserved `__default__` Markdown bucket on the user's first normal memory read. Operators who prefer to audit or complete the migration before serving traffic can run the optional idempotent CLI from `backend/`: + +```bash +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users +# A custom DeerMem root or original non-directory-safe identity can be explicit: +PYTHONPATH=. python scripts/migrate_memory_markdown.py --storage-path /path/to/deerflow-home --user-id 'test@example.com' +``` + +The v1-to-v2 storage migration is one-way for a running application: pre-PR code does not read Markdown facts. Before upgrading a persistent deployment, stop DeerFlow and take a filesystem snapshot or full backup of the configured memory storage root. The migration also durably retains each destructive JSON source beside the original path as `{manifest_filename}.v1.bak` before writing v2 data; an existing mismatched backup or a backup-write failure stops migration without modifying the v1 source. This local backup preserves pre-migration data but is not a substitute for a full snapshot and does not contain facts created after the upgrade. + +`--user-id` may be repeated. `--all-users` discovers the existing directory-safe buckets below the selected storage root; standalone integrations that passed raw IDs containing characters such as `@` should use the original value with `--user-id`. A failed user's migration is reported without hiding the rest of the audit, and the command exits non-zero when any user fails. The automatic first-read path remains enabled, so running this CLI is not required for startup. + ## Recommended Models DeerFlow is model-agnostic — it works with any LLM that implements the OpenAI-compatible API. That said, it performs best with models that support: diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 6e283355d..d7abf0f79 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -604,15 +604,15 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_ ### Memory System (`packages/harness/deerflow/agents/memory/`) **Components**: -- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O +- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication, optimistic revision checks, and repository change sets - `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary - `prompt.py` - Prompt templates for memory updates -- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple +- `storage.py` - File repository with one user-global summary JSON, agent-owned single-fact Markdown, target-only journaled changes, strict fact validation, shared-user plus per-fact optimistic revisions, lock-protected migration, deep-copy caching, and an optional retrieval adapter - `tools.py` - Tool-driven memory mode (`memory_search`, `memory_add`, `memory_update`, `memory_delete`) using the same storage/update primitives **Per-User Isolation**: - Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json` -- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json` +- Per-agent facts at `{base_dir}/users/{user_id}/agents/{agent_name}/facts/{sha256-prefix}/{fact-id}.md`, where the prefix is the first two hexadecimal characters of `SHA-256(fact_id)`; there is no per-agent `memory.json` - Custom agent definitions (`SOUL.md` + `config.yaml`) are also per-user at `{base_dir}/users/{user_id}/agents/{agent_name}/`. The legacy shared layout `{base_dir}/agents/{agent_name}/` remains read-only fallback for unmigrated installations - Middleware mode captures `user_id` via `get_effective_user_id()` at enqueue time; tool mode resolves `user_id` and `agent_name` from `ToolRuntime.context` via `resolve_runtime_user_id(runtime)` so tool calls stay scoped to the authenticated user and active custom agent - The `/api/memory*` endpoints resolve the owner through `_resolve_memory_user_id(request)`: trusted internal callers (IM channel workers carrying the `X-DeerFlow-Owner-User-Id` header, e.g. a bound `/memory` command) act for the connection owner; browser/API callers fall back to `get_effective_user_id()`. The header is only honored after `AuthMiddleware` validated the internal token, mirroring `get_trusted_internal_owner_user_id` used by the threads router @@ -620,16 +620,23 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_ - Absolute `storage_path` in config opts out of per-user isolation - **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json`, `threads/`, and `agents/` into per-user layout. Supports `--dry-run` (preview changes) and `--user-id USER_ID` (assign unowned legacy data to a user, defaults to `default`). -**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`): +**Data Structure**: - **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries) - **History**: `recentMonths`, `earlierContext`, `longTermBackground` -- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source` +- **Global JSON**: `{base_dir}/users/{user_id}/memory.json` stores only `version`, shared revision/time, `user`, and `history`; it never stores facts or a fact index +- **Facts**: Schema-v2 Markdown documents under `agents/{agent_name}/facts/{sha256-prefix}/{fact-id}.md`; YAML front matter contains structure and the body contains the atomic fact +- **Default agent compatibility**: DeerMem resolves an omitted `agent_name` to the reserved `__default__` fact bucket at the manager boundary. The sentinel is accepted only by DeerMem storage and is outside the custom-agent name grammar, so a real custom `lead-agent` remains isolated. Public agent identifiers are case-insensitive and canonicalized to lowercase before storage +- **Compatibility view**: direct global storage reads return `facts: []`, while DeerMem Manager/API reads select the explicit agent or reserved default and return its facts, so existing Settings and embedded-client schemas remain stable. Markdown keeps structured `source` metadata internally; the manager projects it to the historical string field before returning a public document +- **Incremental result contract**: `FileMemoryStorage.apply_changes()` returns `complete: false` plus `upsertedFacts`/`deletedFactIds`; it never presents a partial cache as a complete memory document. Public compatibility callers explicitly reload a fresh complete view only where their response contract requires it, including after successful disjoint-create rebases +- **Repository**: `get/list/upsert/delete_fact`, `apply_changes`, summary operations, migration, index lifecycle/status, and scoped search. `apply_changes` and direct fact CRUD touch only target Markdown files; direct fact CRUD accepts separate expected user-memory and fact revisions. Supplied summary child keys merge over their persisted section, while import normalizes complete replacement sections first. Whole-document `load/save` remains for compatibility but validates the complete `facts` list and diffs it before persistence. An unscoped manager clear first migrates facts from unread legacy agent JSON without adopting potentially conflicting summaries, then removes the global summaries and every agent's canonical facts while preserving agent configuration; an explicit agent clear removes only that bucket's facts and preserves the shared summaries **Workflow**: - `memory.mode: middleware` (default) keeps the passive path: `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, queues conversation with the captured `user_id`, and the debounced background thread invokes the LLM to extract context updates and facts using the stored `user_id`. - `memory.mode: tool` skips `MemoryMiddleware` and registers `memory_search`, `memory_add`, `memory_update`, and `memory_delete` on the agent. The model decides when to search, add, update, or delete facts; this is opt-in/experimental and should not be described as better than middleware mode without eval evidence. - Both modes share `FileMemoryStorage`, per-user/per-agent isolation, prompt injection, manual CRUD primitives, and the updater backend. -- Middleware mode queue debounces (30s default), batches updates, deduplicates per-thread, applies updates atomically (temp file + rename) with cache invalidation, and skips duplicate fact content before append. +- Middleware mode queue debounces (30s default), batches updates, and commits global summaries plus the selected/default agent's fact delta through a user-level lock, optimistic user-memory revisions, per-fact revisions, and a recoverable target-file journal. Only explicitly marked point operations may rebase a stale shared revision, and only while every addressed fact still satisfies its original absent/revision precondition. Snapshot-derived clear/trim/consolidation operations instead reload the complete document and recompute their intent on a manifest conflict, with a bounded retry. Typed manifest/fact conflict subclasses keep that decision independent of exception text, and same-ID creates and stale same-fact writes fail. Scope-lock objects are weakly cached so inactive users do not grow a process-lifetime map. Cache validation does not scale with the fact-file count: its token combines the shared JSON's `(mtime_ns, size, revision)`, so the persisted revision invalidates stale caches even when a coarse-mtime filesystem reports identical metadata for same-size writes; direct out-of-band Markdown edits require `reload()`. Atomic replacement also syncs the parent directory on POSIX so the rename is durable. DeerMem translates private storage conflict/corruption exceptions to the backend-neutral MemoryManager contract; the Gateway maps them to HTTP 409 and a stable HTTP 500 response respectively. A normal default-manager read automatically migrates legacy facts from the global JSON into `__default__`; it also adopts the earlier implicit `lead-agent` fact bucket only when that directory has no custom-agent `config.yaml`, and rejects unexpected files instead of deleting them. The v1-to-v2 migration is one-way for the running application: operators must stop DeerFlow and snapshot the configured storage root before upgrade. Before any destructive v2 write, every migrated JSON source is durably retained as `{manifest_filename}.v1.bak`; a missing-write or mismatched existing backup aborts without modifying v1 data. Legacy per-agent JSON is deleted only after its non-empty summaries are safely adopted or confirmed identical; summary conflicts keep the source file and fail loudly. +- **Proactive Markdown migration CLI**: from `backend/`, run `PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run` to audit and omit `--dry-run` to migrate before serving traffic. Use repeated `--user-id` values when selecting exact original identities, especially standalone raw IDs containing `@` or other characters that are normalized in directory names; `--storage-path` selects a non-default DeerMem root. The CLI reuses `FileMemoryStorage.migrate`, is idempotent, continues across per-user failures, and exits non-zero if any user fails. It is optional because the first normal read still performs the same migration automatically. +- A configured `retrieval_adapter` owns indexing and semantic retrieval. File storage sends upsert/remove notifications for normal writes and both explicit and lazy migrations after releasing durable storage locks, then delegates search; without an adapter it declares and uses `substring_fallback`. - Staleness pass (same LLM invocation as the regular updater, no extra API call): when `staleness_review_enabled` is `true` and at least `staleness_min_candidates` aged facts exist, `_select_stale_candidates` selects facts older than their individual review window (`expected_valid_days`, or the global `staleness_age_days` fallback) that are not in `staleness_protected_categories` (default: `correction`), surfaces them in the prompt with a `valid:Nd` annotation, and the LLM judges each as KEEP, REMOVE, or EXTEND. REMOVE entries go in `staleFactsToRemove`; EXTEND entries go in `staleFactsToExtend` with an `extend_by_days` value, which sets the fact's `expected_valid_days` to `min(days_since_created + extend_by_days, staleness_max_extension_days)`. The LLM assigns `expected_valid_days` when creating a fact; it is clamped at write time to `staleness_age_days × staleness_max_lifetime_multiplier` (creation cap). `_apply_updates` enforces the guardrail unconditionally at apply time: it intersects both the removal and extension sets with `_select_stale_candidates` output before applying the per-cycle cap (`staleness_max_removals_per_cycle`), so protected and non-aged facts can never be targeted regardless of model behavior or the feature flag setting. Facts the LLM proposed for removal are excluded from extension even if the per-cycle cap prevented their actual deletion that cycle. Extensions use an absolute ceiling (`staleness_max_extension_days`) rather than the creation multiplier so a deliberate review decision can advance the window beyond the initial cap while preventing `timedelta` overflow from a malformed `extend_by_days`. - Consolidation pass (same LLM invocation as the regular updater, no extra API call): when `consolidation_enabled` is `true` and at least one category holds `consolidation_min_facts` or more facts, `_select_consolidation_candidates` identifies fragmented categories and surfaces at most `consolidation_max_groups_per_cycle` of them (largest first) in the prompt. The LLM decides which groups to merge and proposes a synthesised fact per group. `_apply_updates` enforces guardrails: source IDs must exist and must not overlap across groups, group size is capped at `consolidation_max_sources`, the merged fact's confidence cannot exceed the source maximum, and facts below `fact_confidence_threshold` are not written. The merged fact carries the newest source's `createdAt` (so the staleness clock reflects the underlying information, not synthesis time) and inherits `expected_valid_days` set so the merged fact is re-reviewed at the earliest source review deadline (`min(createdAt + effective_lifetime)` across sources, where a source's effective lifetime is its `expected_valid_days` or the global `staleness_age_days` fallback for legacy facts without one - so a legacy source's default window is not swallowed by a long-lived sibling), relative to the merged `createdAt`, clamped to a minimal positive window if a source is already past its deadline, then capped at the creation-time `staleness_max_lifetime_multiplier`; this keeps a volatile or legacy sub-detail from inheriting a stable source's long window and escaping staleness review for years, while a merge of uniformly stable sources does not re-enter review prematurely. - Next interaction injects selected facts + context into `` tags in the system prompt when `injection_enabled` is true. @@ -650,7 +657,12 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_ **Configuration** (`config.yaml` → `memory`): - `enabled` / `injection_enabled` - Master switches - `mode` - Operation mode: `middleware` (default passive background extraction) or `tool` (experimental model-driven memory tools). Modes are mutually exclusive. -- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation) +- `storage_path` - DeerMem storage root; one global summary JSON lives under each user and Markdown facts remain under agent buckets +- `storage_class` - `file` or a dotted `MemoryStorage` class; invalid persistent backends fail fast +- `strict_user_scope` - Require `user_id` for all storage access (default `false` for no-auth/legacy compatibility) +- `manifest_filename` - User-global summary JSON filename (kept for configuration compatibility) +- `file_lock_timeout_seconds` - Scope-lock wait; Markdown facts and the recovery journal are required storage invariants rather than configurable modes +- `retrieval_adapter` - Optional dotted factory receiving `DeerMemConfig` and returning a retrieval-port implementation - `debounce_seconds` - Wait time before processing (default: 30) - `shutdown_flush_timeout_seconds` - Host-shared hard budget (seconds) to drain the memory backend's pending-update buffer on Gateway graceful shutdown (default: 30; 1–300). Each pending item does one LLM call, so large IM batches may need more. The Gateway lifespan calls `MemoryManager.shutdown_flush(timeout)` after channels/scheduler stop; the backend short-circuits on an idle buffer, so the host calls it unconditionally (no pending/processing gate). Must fit inside the pod's K8s `terminationGracePeriodSeconds` (gateway Helm chart sets this; default 45s) or K8s SIGKILLs the drain mid-flight. - `model_name` - LLM for updates (null = default model) diff --git a/backend/app/gateway/routers/agents.py b/backend/app/gateway/routers/agents.py index 0b28f503e..98a932665 100644 --- a/backend/app/gateway/routers/agents.py +++ b/backend/app/gateway/routers/agents.py @@ -470,6 +470,8 @@ async def delete_agent(name: str) -> None: if not agent_dir.exists(): outcome = "legacy" if paths.agent_dir(name).exists() else "missing" return outcome, str(agent_dir) + if not (agent_dir / "config.yaml").is_file(): + return "not-custom-agent", str(agent_dir) shutil.rmtree(agent_dir) return "deleted", str(agent_dir) @@ -486,5 +488,10 @@ async def delete_agent(name: str) -> None: ) if outcome == "missing": raise HTTPException(status_code=404, detail=f"Agent '{name}' not found") + if outcome == "not-custom-agent": + raise HTTPException( + status_code=409, + detail=(f"Directory for '{name}' contains memory data but is not a custom agent because config.yaml is missing; it was preserved."), + ) logger.info(f"Deleted agent '{name}' from {agent_dir}") diff --git a/backend/app/gateway/routers/memory.py b/backend/app/gateway/routers/memory.py index 8c701005e..73c44153a 100644 --- a/backend/app/gateway/routers/memory.py +++ b/backend/app/gateway/routers/memory.py @@ -1,12 +1,12 @@ """Memory API router for retrieving and managing global memory data.""" -from typing import Literal +from typing import Any, Literal from fastapi import APIRouter, HTTPException, Request -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, field_validator from app.gateway.internal_auth import get_trusted_internal_owner_user_id -from deerflow.agents.memory import get_memory_manager +from deerflow.agents.memory import MemoryConflictError, MemoryCorruptionError, get_memory_manager from deerflow.config.memory_config import get_memory_config from deerflow.config.paths import make_safe_user_id from deerflow.runtime.user_context import get_effective_user_id @@ -65,16 +65,44 @@ class Fact(BaseModel): id: str = Field(..., description="Unique identifier for the fact") content: str = Field(..., description="Fact content") category: str = Field(default="context", description="Fact category") + categoryExtension: str | None = Field(default=None, description="Extension category when category is 'other'") + topics: list[str] | None = Field(default=None, description="Retrieval-oriented topic labels") confidence: float = Field(default=0.5, description="Confidence score (0-1)") createdAt: str = Field(default="", description="Creation timestamp") - source: str = Field(default="unknown", description="Source thread ID") + source: str = Field(default="unknown", description="Legacy source string; structured metadata remains internal to storage") sourceError: str | None = Field(default=None, description="Optional description of the prior mistake or wrong approach") + schemaVersion: int | None = Field(default=None, description="Per-fact schema version") + status: str | None = Field(default=None, description="Fact lifecycle status") + scope: dict[str, str | None] | None = Field(default=None, description="Canonical user/agent scope") + revision: int | None = Field(default=None, description="Fact optimistic revision") + updatedAt: str | None = Field(default=None, description="Last fact update timestamp") + consolidatedAt: str | None = None + consolidatedFrom: list[str] | None = None + + @field_validator("source", mode="before") + @classmethod + def _legacy_source_string(cls, value: Any) -> str: + """Keep the HTTP contract stable while Markdown stores rich metadata.""" + if isinstance(value, str): + return value + if not isinstance(value, dict): + return "unknown" + source_type = value.get("type") + thread_id = value.get("threadId") + if source_type == "conversation" and isinstance(thread_id, str) and thread_id: + return thread_id + if isinstance(source_type, str) and source_type: + return source_type + if isinstance(thread_id, str) and thread_id: + return thread_id + return "unknown" class MemoryResponse(BaseModel): """Response model for memory data.""" version: str = Field(default="1.0", description="Memory schema version") + revision: int | None = Field(default=None, description="Manifest revision") lastUpdated: str = Field(default="", description="Last update timestamp") user: UserContext = Field(default_factory=UserContext) history: HistoryContext = Field(default_factory=HistoryContext) @@ -85,11 +113,20 @@ def _map_memory_fact_value_error(exc: ValueError) -> HTTPException: """Convert updater validation errors into stable API responses.""" if exc.args and exc.args[0] == "confidence": detail = "Invalid confidence value; must be between 0 and 1." + elif exc.args and exc.args[0] == "agent_name": + detail = "An agent name is required for fact operations; user-global memory stores summaries only." else: detail = "Memory fact content cannot be empty." return HTTPException(status_code=400, detail=detail) +def _map_memory_manager_error(exc: MemoryConflictError | MemoryCorruptionError) -> HTTPException: + """Map backend-neutral manager errors without importing a storage plugin.""" + if isinstance(exc, MemoryConflictError): + return HTTPException(status_code=409, detail="Memory changed concurrently; reload and retry.") + return HTTPException(status_code=500, detail="Stored memory data is corrupted.") + + def _require_capability(name: str, *, label: str): """Return a DeerMem-internal capability (bound method) or raise 501. @@ -183,7 +220,10 @@ async def get_memory(http_request: Request) -> MemoryResponse: } ``` """ - memory_data = get_memory_manager().get_memory(user_id=_resolve_memory_user_id(http_request)) + try: + memory_data = get_memory_manager().get_memory(user_id=_resolve_memory_user_id(http_request)) + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc return MemoryResponse(**memory_data) @@ -205,14 +245,17 @@ async def reload_memory(http_request: Request) -> MemoryResponse: """ user_id = _resolve_memory_user_id(http_request) manager = get_memory_manager() - if hasattr(manager, "reload_memory"): - memory_data = manager.reload_memory(user_id=user_id) - else: - # Non-DeerMem backends have no reload concept; return current memory. - # (Asymmetry vs fact CRUD, which raises 501 when unsupported: reload is a - # read-only refresh, so degrading to get_memory is safe and still useful; - # silently no-op'ing a write would hide data loss, so writes fail loud.) - memory_data = manager.get_memory(user_id=user_id) + try: + if hasattr(manager, "reload_memory"): + memory_data = manager.reload_memory(user_id=user_id) + else: + # Non-DeerMem backends have no reload concept; return current memory. + # (Asymmetry vs fact CRUD, which raises 501 when unsupported: reload is a + # read-only refresh, so degrading to get_memory is safe and still useful; + # silently no-op'ing a write would hide data loss, so writes fail loud.) + memory_data = manager.get_memory(user_id=user_id) + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc return MemoryResponse(**memory_data) @@ -227,6 +270,8 @@ async def clear_memory(http_request: Request) -> MemoryResponse: """Clear all persisted memory data.""" try: memory_data = get_memory_manager().clear_memory(user_id=_resolve_memory_user_id(http_request)) + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc except OSError as exc: raise HTTPException(status_code=500, detail="Failed to clear memory data.") from exc @@ -252,6 +297,8 @@ async def create_memory_fact_endpoint(request: FactCreateRequest, http_request: ) except ValueError as exc: raise _map_memory_fact_value_error(exc) from exc + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc except OSError as exc: raise HTTPException(status_code=500, detail="Failed to create memory fact.") from exc @@ -275,6 +322,8 @@ async def delete_memory_fact_endpoint(fact_id: str, http_request: Request) -> Me memory_data = delete_fact(fact_id, user_id=_resolve_memory_user_id(http_request)) except KeyError as exc: raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc except OSError as exc: raise HTTPException(status_code=500, detail="Failed to delete memory fact.") from exc @@ -303,6 +352,8 @@ async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest, h raise _map_memory_fact_value_error(exc) from exc except KeyError as exc: raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc except OSError as exc: raise HTTPException(status_code=500, detail="Failed to update memory fact.") from exc @@ -318,7 +369,10 @@ async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest, h ) async def export_memory(http_request: Request) -> MemoryResponse: """Export the current memory data.""" - memory_data = get_memory_manager().get_memory(user_id=_resolve_memory_user_id(http_request)) + try: + memory_data = get_memory_manager().get_memory(user_id=_resolve_memory_user_id(http_request)) + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc return MemoryResponse(**memory_data) @@ -332,7 +386,9 @@ async def export_memory(http_request: Request) -> MemoryResponse: async def import_memory(request: MemoryResponse, http_request: Request) -> MemoryResponse: """Import and persist memory data.""" try: - memory_data = get_memory_manager().import_memory(request.model_dump(), user_id=_resolve_memory_user_id(http_request)) + memory_data = get_memory_manager().import_memory(request.model_dump(exclude_none=True), user_id=_resolve_memory_user_id(http_request)) + except (MemoryConflictError, MemoryCorruptionError) as exc: + raise _map_memory_manager_error(exc) from exc except OSError as exc: raise HTTPException(status_code=500, detail="Failed to import memory data.") from exc diff --git a/backend/packages/harness/deerflow/agents/memory/__init__.py b/backend/packages/harness/deerflow/agents/memory/__init__.py index 677fb9b08..22188cf49 100644 --- a/backend/packages/harness/deerflow/agents/memory/__init__.py +++ b/backend/packages/harness/deerflow/agents/memory/__init__.py @@ -13,13 +13,19 @@ them directly from ``deerflow.agents.memory.backends.deermem.deermem.core.*``. """ from deerflow.agents.memory.manager import ( + MemoryConflictError, + MemoryCorruptionError, MemoryManager, + MemoryManagerError, get_memory_manager, reset_memory_manager, ) __all__ = [ "MemoryManager", + "MemoryManagerError", + "MemoryConflictError", + "MemoryCorruptionError", "get_memory_manager", "reset_memory_manager", ] diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py index ddb4ffd7d..f0023c3e0 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py @@ -23,10 +23,11 @@ never breaks those modules at import time (see MemoryManager plan, step 8). from __future__ import annotations +import copy import logging from typing import Any -from deerflow.agents.memory.manager import MemoryManager +from deerflow.agents.memory.manager import MemoryConflictError, MemoryCorruptionError, MemoryManager from .deermem.config import DeerMemConfig from .deermem.core.llm import build_llm @@ -36,14 +37,56 @@ from .deermem.core.message_processing import ( filter_messages_for_memory, load_patterns, ) +from .deermem.core.paths import DEFAULT_AGENT_BUCKET from .deermem.core.prompt import format_memory_for_injection, load_prompt, load_prompt_messages, warm_tiktoken_cache from .deermem.core.queue import MemoryUpdateQueue -from .deermem.core.storage import create_storage +from .deermem.core.storage import MemoryRevisionConflict, MemoryStorageCorruption, create_storage from .deermem.core.updater import MemoryUpdater, _coerce_source_confidence logger = logging.getLogger(__name__) +def _resolve_agent_name(agent_name: str | None) -> str: + """Return DeerFlow's case-insensitive canonical agent identifier.""" + return agent_name.lower() if agent_name is not None else DEFAULT_AGENT_BUCKET + + +def _call_backend(operation): + """Translate DeerMem-private storage errors into the public manager contract.""" + try: + return operation() + except MemoryRevisionConflict as exc: + raise MemoryConflictError(str(exc)) from exc + except MemoryStorageCorruption as exc: + raise MemoryCorruptionError(str(exc)) from exc + + +def _legacy_source_value(source: Any) -> str: + """Project structured source metadata back to the legacy public string.""" + if isinstance(source, str): + return source + if not isinstance(source, dict): + return "unknown" + source_type = source.get("type") + thread_id = source.get("threadId") + if source_type == "conversation" and isinstance(thread_id, str) and thread_id: + return thread_id + if isinstance(source_type, str) and source_type: + return source_type + if isinstance(thread_id, str) and thread_id: + return thread_id + return "unknown" + + +def _compat_document(memory_data: dict[str, Any]) -> dict[str, Any]: + """Return the historical Manager/API shape without changing persistence.""" + result = copy.deepcopy(memory_data) + for fact in result.get("facts", []): + if isinstance(fact, dict): + fact["source"] = _legacy_source_value(fact.get("source")) + return result + + class DeerMem(MemoryManager): """Default memory backend: file-backed facts + debounced LLM extraction.""" @@ -109,7 +152,7 @@ class DeerMem(MemoryManager): self._queue.add( thread_id=thread_id, messages=filtered, - agent_name=agent_name, + agent_name=_resolve_agent_name(agent_name), user_id=user_id, trace_id=trace_id, correction_detected=correction_detected, @@ -136,7 +179,7 @@ class DeerMem(MemoryManager): self._queue.add_nowait( thread_id=thread_id, messages=filtered, - agent_name=agent_name, + agent_name=_resolve_agent_name(agent_name), user_id=user_id, correction_detected=correction_detected, reinforcement_detected=reinforcement_detected, @@ -179,7 +222,7 @@ class DeerMem(MemoryManager): ``injection_enabled`` gate and the ```` wrapping stay at the call site (``_get_memory_context``); this returns only the body. """ - memory_data = self._updater.get_memory_data(agent_name=agent_name, user_id=user_id) + memory_data = _call_backend(lambda: self._updater.get_memory_data(agent_name=_resolve_agent_name(agent_name), user_id=user_id)) return format_memory_for_injection( memory_data, max_tokens=self._config.max_injection_tokens, @@ -210,10 +253,26 @@ class DeerMem(MemoryManager): if not query or not query.strip() or top_k <= 0: return [] query_lower = query.strip().lower() - memory_data = self._updater.get_memory_data(agent_name=agent_name, user_id=user_id) + search_facts = getattr(self._storage, "search_facts", None) + resolved_agent_name = _resolve_agent_name(agent_name) + scopes = [{"userId": user_id, "agentName": resolved_agent_name}] + indexed = ( + search_facts( + query, + scopes=scopes, + top_k=top_k, + mode="hybrid", + filters={"category": category} if category else None, + ) + if callable(search_facts) + else [] + ) + if indexed: + return [_compat_document({"facts": [result.get("fact", result)]})["facts"][0] for result in indexed] + memory_data = _call_backend(lambda: self._updater.get_memory_data(agent_name=resolved_agent_name, user_id=user_id)) matched = [fact for fact in memory_data.get("facts", []) if isinstance(fact.get("content"), str) and query_lower in fact["content"].lower() and (category is None or fact.get("category") == category)] matched.sort(key=_coerce_source_confidence, reverse=True) - return matched[:top_k] + return _compat_document({"facts": matched[:top_k]})["facts"] # ── Manage ─────────────────────────────────────────────────────────── def get_memory( @@ -222,7 +281,8 @@ class DeerMem(MemoryManager): user_id: str | None = None, agent_name: str | None = None, ) -> dict[str, Any]: - return self._updater.get_memory_data(agent_name=agent_name, user_id=user_id) + memory_data = _call_backend(lambda: self._updater.get_memory_data(agent_name=_resolve_agent_name(agent_name), user_id=user_id)) + return _compat_document(memory_data) def delete_memory( self, @@ -239,7 +299,11 @@ class DeerMem(MemoryManager): user_id: str | None = None, agent_name: str | None = None, ) -> dict[str, Any]: - return self._updater.clear_memory_data(agent_name=agent_name, user_id=user_id) + if agent_name is None: + memory_data = _call_backend(lambda: self._updater.clear_all_memory_data(user_id=user_id)) + else: + memory_data = _call_backend(lambda: self._updater.clear_memory_data(agent_name=_resolve_agent_name(agent_name), user_id=user_id)) + return _compat_document(memory_data) def import_memory( self, @@ -248,7 +312,14 @@ class DeerMem(MemoryManager): user_id: str | None = None, agent_name: str | None = None, ) -> dict[str, Any]: - return self._updater.import_memory_data(memory_data, agent_name=agent_name, user_id=user_id) + imported = _call_backend( + lambda: self._updater.import_memory_data( + memory_data, + agent_name=_resolve_agent_name(agent_name), + user_id=user_id, + ) + ) + return _compat_document(imported) def export_memory( self, @@ -295,7 +366,13 @@ class DeerMem(MemoryManager): agent_name: str | None = None, ) -> dict[str, Any]: """Drop the cached memory document and reload from disk.""" - return self._updater.reload_memory_data(agent_name=agent_name, user_id=user_id) + memory_data = _call_backend( + lambda: self._updater.reload_memory_data( + agent_name=_resolve_agent_name(agent_name), + user_id=user_id, + ) + ) + return _compat_document(memory_data) def create_fact( self, @@ -306,13 +383,16 @@ class DeerMem(MemoryManager): agent_name: str | None = None, user_id: str | None = None, ) -> tuple[dict[str, Any], str | None]: - return self._updater.create_memory_fact( - content, - category=category, - confidence=confidence, - agent_name=agent_name, - user_id=user_id, + memory_data, fact_id = _call_backend( + lambda: self._updater.create_memory_fact( + content, + category=category, + confidence=confidence, + agent_name=_resolve_agent_name(agent_name), + user_id=user_id, + ) ) + return _compat_document(memory_data), fact_id def delete_fact( self, @@ -321,7 +401,14 @@ class DeerMem(MemoryManager): agent_name: str | None = None, user_id: str | None = None, ) -> dict[str, Any]: - return self._updater.delete_memory_fact(fact_id, agent_name=agent_name, user_id=user_id) + memory_data = _call_backend( + lambda: self._updater.delete_memory_fact( + fact_id, + agent_name=_resolve_agent_name(agent_name), + user_id=user_id, + ) + ) + return _compat_document(memory_data) def update_fact( self, @@ -333,11 +420,14 @@ class DeerMem(MemoryManager): agent_name: str | None = None, user_id: str | None = None, ) -> dict[str, Any]: - return self._updater.update_memory_fact( - fact_id, - content=content, - category=category, - confidence=confidence, - agent_name=agent_name, - user_id=user_id, + memory_data = _call_backend( + lambda: self._updater.update_memory_fact( + fact_id, + content=content, + category=category, + confidence=confidence, + agent_name=_resolve_agent_name(agent_name), + user_id=user_id, + ) ) + return _compat_document(memory_data) diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py index 136396cf8..a8a32eef8 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py @@ -52,6 +52,24 @@ class DeerMemConfig(BaseModel): default="", description="Dotted class path for an alternative storage provider; empty (default) = FileMemoryStorage (no importlib, portable).", ) + strict_user_scope: bool = Field( + default=False, + description="Require user_id for every storage scope. False preserves no-auth and legacy callers.", + ) + manifest_filename: str = Field( + default="memory.json", + description="User-global summary JSON filename. Kept under this name for config compatibility; must be a plain .json filename.", + ) + file_lock_timeout_seconds: int = Field( + default=10, + ge=1, + le=120, + description="Maximum wait for the per-scope cross-process advisory file lock.", + ) + retrieval_adapter: str = Field( + default="", + description="Optional dotted retrieval-adapter factory. It receives DeerMemConfig and must implement RetrievalPort.", + ) # ── Queue ──────────────────────────────────────────────────────────── debounce_seconds: int = Field( default=30, @@ -254,6 +272,7 @@ class DeerMemConfig(BaseModel): """ if not backend_config: return cls() + backend_config = dict(backend_config) known = {k: v for k, v in backend_config.items() if k in cls.model_fields and v is not None} unknown = sorted(k for k in backend_config if k not in cls.model_fields) if unknown: diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/paths.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/paths.py index 28748ac7a..59c8211a5 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/paths.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/paths.py @@ -2,9 +2,9 @@ The host no longer dictates where DeerMem stores data. Root = ``config.storage_path`` (if set, absolute or relative) or ``$DEERMEM_DATA_DIR`` or ``~/.deermem/``. -Per-user / per-agent / legacy layouts live under the root, mirroring the -pre-abstraction paths so a one-time data migration (old ``{base_dir}/users/*`` --> DeerMem root) is a plain move. +Each user has one global ``memory.json`` for project-independent summaries. +Agent-specific facts live below ``agents/{agent_name}/facts`` and never add a +fact index to that JSON document. user_id is sanitized in-process (``[A-Za-z0-9_-]`` + SHA-256 digest for lossy ids) and agent_name validated against an inlined pattern -- DeerMem does not @@ -31,6 +31,9 @@ _SAFE_USER_ID_DIGEST_HEX_LEN = 16 # agent_name validation (inlined; was deer-flow's AGENT_NAME_PATTERN). AGENT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9-]+$") +# Internal bucket used when callers omit ``agent_name``. The underscores keep +# it outside the public custom-agent namespace accepted by AGENT_NAME_PATTERN. +DEFAULT_AGENT_BUCKET = "__default__" def safe_user_id(raw: str) -> str: @@ -54,7 +57,7 @@ def validate_agent_name(name: str) -> None: """Validate that the agent name is safe to use in filesystem paths.""" if not name: raise ValueError("Agent name must be a non-empty string.") - if not AGENT_NAME_PATTERN.match(name): + if name != DEFAULT_AGENT_BUCKET and not AGENT_NAME_PATTERN.match(name): raise ValueError(f"Invalid agent name {name!r}: names must match {AGENT_NAME_PATTERN.pattern}") @@ -74,22 +77,40 @@ def memory_file_path( ) -> Path: """Resolve the memory file path under DeerMem's own data root. - ``config.storage_path`` (absolute or relative) is the root; per-user / - per-agent / legacy layouts live under it. Empty -> default root + ``config.storage_path`` (absolute or relative) is the root. Empty -> default root (``$DEERMEM_DATA_DIR`` / ``~/.deermem/``). The host (deer-flow factory) injects an absolute base_dir as ``storage_path`` so memory lands at ``{base_dir}/users/{user_id}/memory.json`` (CWD-independent). """ root = Path(config.storage_path) if config.storage_path else _default_root() + if config.strict_user_scope and user_id is None: + raise ValueError("user_id is required when strict_user_scope is enabled.") + manifest_filename = config.manifest_filename + if Path(manifest_filename).name != manifest_filename or not manifest_filename.endswith(".json"): + raise ValueError("manifest_filename must be a plain .json filename.") if user_id is not None: uid = safe_user_id(user_id) if agent_name is not None: validate_agent_name(agent_name) - return root / "users" / uid / "agents" / agent_name.lower() / "memory.json" - return root / "users" / uid / "memory.json" + bucket = root / "users" / uid + return bucket / manifest_filename # Legacy: no user_id if agent_name is not None: validate_agent_name(agent_name) - return root / "agents" / agent_name.lower() / "memory.json" - return root / "memory.json" + bucket = root + return bucket / manifest_filename + + +def agent_facts_directory(memory_path: Path, agent_name: str) -> Path: + """Return the fact root for one required agent below a user's memory file.""" + validate_agent_name(agent_name) + return memory_path.parent / "agents" / agent_name.lower() / "facts" + + +def fact_file_path(memory_path: Path, fact_id: str, *, agent_name: str) -> Path: + """Return the sharded Markdown path for one agent-owned fact.""" + if not fact_id or not re.fullmatch(r"[A-Za-z0-9_-]+", fact_id): + raise ValueError("Fact id may contain only letters, numbers, '_' and '-'.") + prefix = hashlib.sha256(fact_id.encode("utf-8")).hexdigest()[:2] + return agent_facts_directory(memory_path, agent_name) / prefix / f"{fact_id}.md" diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py index 38ff6fb1e..53c1230ac 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py @@ -1,29 +1,87 @@ -"""Memory storage providers.""" +"""Memory storage providers. + +The file backend stores only project-independent user/history summaries in one +user-level ``memory.json``. Each fact is canonical in one Markdown file below +its required agent name. The +public ``load``/``save`` compatibility surface still exposes the historical +document shape (``facts`` is a list), so updater and gateway callers can move +to the fact repository API incrementally. +""" + +from __future__ import annotations import abc +import copy +import hashlib +import importlib import json import logging +import os +import shutil import threading +import time import uuid +import weakref +from collections.abc import Iterator +from contextlib import contextmanager from datetime import UTC, datetime from pathlib import Path -from typing import Any +from typing import Any, Protocol + +import yaml from ..config import DeerMemConfig -from .paths import memory_file_path +from .paths import DEFAULT_AGENT_BUCKET, agent_facts_directory, fact_file_path, memory_file_path, safe_user_id, validate_agent_name logger = logging.getLogger(__name__) +DOCUMENT_VERSION = "2.0" +CORE_CATEGORIES = frozenset({"preference", "correction", "context", "goal", "behavior", "identity", "constraint", "decision", "other"}) + + +class MemoryStorageError(RuntimeError): + """Base error for persistent memory failures.""" + + +class MemoryStorageCorruption(MemoryStorageError): + """The global memory JSON or a canonical fact cannot be parsed safely.""" + + +class MemoryRevisionConflict(MemoryStorageError): + """A stale writer attempted to overwrite a newer user-memory revision.""" + + +class MemoryManifestRevisionConflict(MemoryRevisionConflict): + """The shared user-memory revision changed before a transaction committed.""" + + +class MemoryFactRevisionConflict(MemoryRevisionConflict): + """A fact no longer satisfies its expected absence or revision.""" + + +class RetrievalPort(Protocol): + """Storage-facing adapter implemented by the independent retrieval module.""" + + def upsert(self, fact: dict[str, Any], *, scope: dict[str, str | None], path: str) -> None: ... + + def remove(self, fact_id: str, *, scope: dict[str, str | None]) -> None: ... + + def search(self, query: str, *, scopes: list[dict[str, str | None]], top_k: int, mode: str, filters: dict[str, Any] | None) -> list[dict[str, Any]]: ... + + +RetrievalNotification = tuple[str, dict[str, Any] | str, str | None] +ScopedRetrievalNotifications = tuple[str, list[RetrievalNotification]] + def utc_now_iso_z() -> str: - """Current UTC time as ISO-8601 with ``Z`` suffix (matches prior naive-UTC output).""" return datetime.now(UTC).isoformat().removesuffix("+00:00") + "Z" def create_empty_memory() -> dict[str, Any]: - """Create an empty memory structure.""" + """Return the compatibility document shape used by updater/injection.""" return { "version": "1.0", + "revision": 0, "lastUpdated": utc_now_iso_z(), "user": { "workContext": {"summary": "", "updatedAt": ""}, @@ -39,163 +97,1381 @@ def create_empty_memory() -> dict[str, Any]: } +def _has_meaningful_data(value: Any) -> bool: + """Return whether a legacy summary value contains anything worth preserving.""" + if isinstance(value, dict): + return any(_has_meaningful_data(item) for item in value.values()) + if isinstance(value, (list, tuple, set)): + return any(_has_meaningful_data(item) for item in value) + return value not in (None, "", False) + + +def _merge_legacy_summary_section(*, canonical: Any, legacy: Any, section: str, legacy_path: Path) -> Any: + """Adopt a legacy section only when doing so cannot overwrite live data.""" + if canonical == legacy or not _has_meaningful_data(legacy): + return copy.deepcopy(canonical) + if not _has_meaningful_data(canonical): + return copy.deepcopy(legacy) + raise MemoryStorageCorruption(f"Legacy {section} summary migration conflict in {legacy_path}; the legacy file was kept") + + +def _scope_dict(user_id: str | None, agent_name: str | None) -> dict[str, str | None]: + return {"userId": user_id, "agentName": agent_name} + + +def _content_hash(raw: bytes) -> str: + return f"sha256:{hashlib.sha256(raw).hexdigest()}" + + +def _file_signature(path: Path) -> tuple[int, int] | None: + """Use nanosecond mtime plus size so cache validation is not mtime-only.""" + try: + stat = path.stat() + return (stat.st_mtime_ns, stat.st_size) + except OSError: + return None + + +def _ensure_migration_backup(source_path: Path) -> Path: + """Durably preserve one immutable pre-migration JSON source beside it.""" + backup_path = source_path.with_name(f"{source_path.name}.v1.bak") + try: + source_bytes = source_path.read_bytes() + if backup_path.exists(): + if backup_path.read_bytes() != source_bytes: + raise MemoryStorageCorruption(f"Existing migration backup {backup_path} differs from source {source_path}; the original backup was kept and migration was stopped") + return backup_path + _atomic_write(backup_path, source_bytes) + return backup_path + except MemoryStorageCorruption: + raise + except OSError as exc: + raise OSError(f"Failed to create durable migration backup {backup_path}: {exc}") from exc + + +def _normalize_category(fact: dict[str, Any]) -> None: + raw_category = fact.get("category", "context") + if not isinstance(raw_category, str): + raise ValueError("fact.category must be a string") + category = raw_category or "context" + if category not in CORE_CATEGORIES: + fact.setdefault("categoryExtension", category) + fact["category"] = "other" + + +def _require_string_list(fact: dict[str, Any], field: str) -> None: + value = fact.get(field, []) + if not isinstance(value, list) or any(not isinstance(item, str) for item in value): + raise ValueError(f"fact.{field} must be a list of strings") + fact[field] = value + + +def _normalize_fact( + fact: dict[str, Any], + *, + scope: dict[str, str | None], + existing: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Validate one fact and derive its per-item revision. + + The shared JSON revision protects the multi-file transaction. The fact's + own revision protects one Markdown object when a disjoint transaction is + safely rebased after that shared revision changed. + """ + if not isinstance(fact, dict): + raise ValueError("fact must be an object") + normalized = copy.deepcopy(fact) + normalized["id"] = str(normalized.get("id") or f"fact_{uuid.uuid4().hex}") + # Validate the id through the canonical path builder's public contract. + if not normalized["id"] or any(character not in "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-" for character in normalized["id"]): + raise ValueError("fact.id may contain only letters, numbers, '_' and '-'") + normalized["schemaVersion"] = 2 + if not isinstance(normalized.get("content"), str): + raise ValueError("fact.content must be a string") + normalized["content"] = normalized["content"].strip() + if not normalized["content"]: + raise ValueError("fact.content must not be empty") + _normalize_category(normalized) + confidence = normalized.get("confidence", 0.5) + if isinstance(confidence, bool) or not isinstance(confidence, (int, float)) or not 0 <= confidence <= 1: + raise ValueError("fact.confidence must be a number between 0 and 1") + normalized["confidence"] = float(confidence) + status = normalized.get("status", "active") + if status != "active": + raise ValueError("fact.status must be 'active'; deletion is physical") + normalized["status"] = "active" + normalized["scope"] = copy.deepcopy(scope) + _require_string_list(normalized, "topics") + _require_string_list(normalized, "consolidatedFrom") + revision = normalized.get("revision", 1) + if isinstance(revision, bool) or not isinstance(revision, int) or revision < 1: + raise ValueError("fact.revision must be an integer >= 1") + source = normalized.get("source") + if isinstance(source, str): + if source in {"manual", "consolidation", "import", "unknown"}: + normalized["source"] = {"type": source, "threadId": None} + else: + normalized["source"] = {"type": "conversation", "threadId": source} + elif not isinstance(source, dict): + normalized["source"] = {"type": "unknown", "threadId": None} + else: + normalized["source"].setdefault("type", "unknown") + if not isinstance(normalized["source"].get("type"), str): + raise ValueError("fact.source.type must be a string") + if normalized["source"].get("threadId") is not None and not isinstance(normalized["source"].get("threadId"), str): + raise ValueError("fact.source.threadId must be a string or null") + normalized["title"] = _fact_title(normalized) + now = utc_now_iso_z() + if existing is None: + normalized.setdefault("createdAt", now) + normalized.setdefault("updatedAt", normalized["createdAt"]) + normalized["revision"] = revision + else: + existing_revision = existing.get("revision") + if not isinstance(existing_revision, int) or existing_revision < 1: + raise MemoryStorageCorruption(f"Stored fact {normalized['id']!r} has an invalid revision") + if revision != existing_revision: + raise MemoryFactRevisionConflict(f"Expected fact {normalized['id']!r} revision {revision}, found {existing_revision}") + normalized["createdAt"] = existing.get("createdAt") or normalized.get("createdAt") or now + comparison_keys = {"revision", "updatedAt"} + incoming_material = {key: value for key, value in normalized.items() if key not in comparison_keys} + existing_material = {key: value for key, value in existing.items() if key not in comparison_keys} + if incoming_material == existing_material: + normalized["revision"] = existing_revision + normalized["updatedAt"] = existing.get("updatedAt") or normalized["createdAt"] + else: + normalized["revision"] = existing_revision + 1 + normalized["updatedAt"] = now + if not isinstance(normalized.get("createdAt"), str) or not isinstance(normalized.get("updatedAt"), str): + raise ValueError("fact.createdAt and fact.updatedAt must be strings") + if normalized["consolidatedFrom"]: + normalized.setdefault("consolidatedAt", normalized["updatedAt"]) + return normalized + + +def _safe_relative_path(root: Path, relative: str, *, label: str) -> Path: + """Resolve an untrusted persisted relative path without leaving root.""" + candidate = Path(relative) + if candidate.is_absolute(): + raise MemoryStorageCorruption(f"{label} path escapes the user memory directory: {relative!r}") + root_resolved = root.resolve() + resolved = (root / candidate).resolve() + try: + resolved.relative_to(root_resolved) + except ValueError as exc: + raise MemoryStorageCorruption(f"{label} path escapes the user memory directory: {relative!r}") from exc + return resolved + + +def _fact_title(fact: dict[str, Any]) -> str: + explicit = str(fact.get("title") or "").strip() + if explicit: + return explicit.replace("\n", " ")[:160] + first = str(fact.get("content") or "Memory fact").splitlines()[0].strip() + return (first or "Memory fact")[:160] + + +def _render_fact_markdown(fact: dict[str, Any]) -> bytes: + metadata = {key: copy.deepcopy(value) for key, value in fact.items() if key not in {"content", "title"}} + scope = metadata.pop("scope", {}) + if isinstance(scope, dict): + metadata["user_id"] = scope.get("userId") + metadata["agent_name"] = scope.get("agentName") + front_matter = yaml.safe_dump(metadata, allow_unicode=True, sort_keys=False).strip() + text = f"---\n{front_matter}\n---\n\n# {_fact_title(fact)}\n\n{fact['content'].rstrip()}\n" + return text.encode("utf-8") + + +def _parse_fact_markdown(path: Path) -> dict[str, Any]: + try: + text = path.read_text(encoding="utf-8") + if not text.startswith("---\n"): + raise ValueError("missing YAML front matter") + front, body = text[4:].split("\n---\n", 1) + metadata = yaml.safe_load(front) or {} + if not isinstance(metadata, dict): + raise ValueError("front matter is not a mapping") + body = body.lstrip("\n") + lines = body.splitlines() + title = "" + if lines and lines[0].startswith("# "): + title = lines.pop(0)[2:].strip() + if lines and not lines[0].strip(): + lines.pop(0) + metadata["title"] = title + metadata["content"] = "\n".join(lines).rstrip("\n") + metadata["scope"] = { + "userId": metadata.pop("user_id", None), + "agentName": metadata.pop("agent_name", None), + } + return metadata + except (OSError, UnicodeError, ValueError, yaml.YAMLError) as exc: + raise MemoryStorageCorruption(f"Failed to parse canonical fact {path}: {exc}") from exc + + +def _fsync_parent_directory(directory: Path) -> None: + """Make a completed rename durable on POSIX filesystems.""" + if os.name == "nt": + return + descriptor = os.open(directory, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _atomic_write(path: Path, raw: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp") + try: + with open(temp, "wb") as handle: + handle.write(raw) + handle.flush() + os.fsync(handle.fileno()) + temp.replace(path) + _fsync_parent_directory(path.parent) + finally: + try: + temp.unlink(missing_ok=True) + except OSError: + pass + + +@contextmanager +def _process_file_lock(lock_path: Path, timeout_seconds: float) -> Iterator[None]: + """Cross-process advisory lock for one scope, using only the stdlib.""" + lock_path.parent.mkdir(parents=True, exist_ok=True) + handle = open(lock_path, "a+b") + deadline = time.monotonic() + timeout_seconds + acquired = False + try: + while not acquired: + try: + if os.name == "nt": + import msvcrt + + handle.seek(0, os.SEEK_END) + if handle.tell() == 0: + handle.write(b"0") + handle.flush() + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + acquired = True + except OSError: + if time.monotonic() >= deadline: + raise TimeoutError(f"Timed out acquiring memory scope lock {lock_path}") + time.sleep(0.05) + yield + finally: + if acquired: + try: + if os.name == "nt": + import msvcrt + + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + except OSError: + logger.warning("Failed to release memory scope lock %s", lock_path) + handle.close() + + class MemoryStorage(abc.ABC): - """Abstract base class for memory storage providers.""" + @abc.abstractmethod + def load(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: ... @abc.abstractmethod - def load(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Load memory data for the given agent.""" - pass + def reload(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: ... @abc.abstractmethod - def reload(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Force reload memory data for the given agent.""" - pass + def save( + self, + memory_data: dict[str, Any], + agent_name: str | None = None, + *, + user_id: str | None = None, + expected_revision: int | None = None, + ) -> bool: ... - @abc.abstractmethod - def save(self, memory_data: dict[str, Any], agent_name: str | None = None, *, user_id: str | None = None) -> bool: - """Save memory data for the given agent.""" - pass + def apply_changes(self, change_set: dict[str, Any], **scope: Any) -> dict[str, Any]: + """Apply one repository change set; providers may override atomically.""" + raise NotImplementedError + + def clear_all(self, *, user_id: str | None = None) -> dict[str, Any]: + """Clear global summaries and every agent fact bucket for one user.""" + raise NotImplementedError class FileMemoryStorage(MemoryStorage): - """File-based memory storage provider.""" - - def __init__(self, config: DeerMemConfig): - """Initialize the file memory storage with an injected DeerMemConfig.""" + def __init__(self, config: DeerMemConfig, retrieval: RetrievalPort | None = None): self._config = config - # Per-user/agent memory cache: keyed by (user_id, agent_name) tuple (None = global) - # Value: (memory_data, file_mtime) - self._memory_cache: dict[tuple[str | None, str | None], tuple[dict[str, Any], float | None]] = {} - # Guards all reads and writes to _memory_cache across concurrent callers. + self._retrieval = retrieval + self._memory_cache: dict[tuple[str | None, str | None], tuple[dict[str, Any], tuple[Any, ...]]] = {} self._cache_lock = threading.Lock() - - def _get_memory_file_path(self, agent_name: str | None = None, *, user_id: str | None = None) -> Path: - """Get the path to the memory file (DeerMem's own path resolution).""" - return memory_file_path(self._config, agent_name, user_id=user_id) - - def _load_memory_from_file(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Load memory data from file.""" - file_path = self._get_memory_file_path(agent_name, user_id=user_id) - - if not file_path.exists(): - return create_empty_memory() - - try: - with open(file_path, encoding="utf-8") as f: - data = json.load(f) - return data - except (json.JSONDecodeError, OSError) as e: - logger.warning("Failed to load memory file: %s", e) - return create_empty_memory() + self._scope_locks: weakref.WeakValueDictionary[tuple[str | None, str | None], threading.RLock] = weakref.WeakValueDictionary() @staticmethod def _cache_key(agent_name: str | None = None, *, user_id: str | None = None) -> tuple[str | None, str | None]: return (user_id, agent_name) - def load(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Load memory data (cached with file modification time check).""" - file_path = self._get_memory_file_path(agent_name, user_id=user_id) - cache_key = self._cache_key(agent_name, user_id=user_id) + def _scope_lock(self, key: tuple[str | None, str | None]) -> threading.RLock: + with self._cache_lock: + return self._scope_locks.setdefault(key, threading.RLock()) + def _get_memory_file_path(self, agent_name: str | None = None, *, user_id: str | None = None) -> Path: + return memory_file_path(self._config, agent_name, user_id=user_id) + + def _scope_signature(self, path: Path, agent_name: str | None) -> tuple[Any, ...]: + """Track supported writes without scanning the agent's fact files. + + Every storage-managed fact mutation advances and atomically replaces + the user-level JSON revision. Including that revision prevents stale + cache hits when a coarse-mtime filesystem reports identical metadata + for two same-size writes. Direct out-of-band Markdown edits require + ``reload()``. + """ + file_signature = _file_signature(path) + if file_signature is None: + return (None, None, None) + memory_file = self._load_memory_file(path) + revision = int((memory_file or {}).get("revision") or 0) + return (*file_signature, revision) + + def _dispatch_retrieval_notifications( + self, + notifications: list[RetrievalNotification], + *, + user_id: str | None, + agent_name: str | None, + ) -> None: + """Notify the optional index only after durable storage locks are released.""" + if self._retrieval is None: + return + scope = _scope_dict(user_id, agent_name) + for action, value, fact_path in notifications: + try: + if action == "upsert": + self._retrieval.upsert(copy.deepcopy(value), scope=scope, path=fact_path or "") + else: + self._retrieval.remove(str(value), scope=scope) + except Exception: + logger.exception("Retrieval notification failed for %s", value) + + @staticmethod + def _validate_loaded_fact( + fact: dict[str, Any], + fact_path: Path, + *, + user_id: str | None, + agent_name: str, + ) -> dict[str, Any]: + if str(fact.get("id")) != fact_path.stem: + raise MemoryStorageCorruption(f"Fact id mismatch for {fact_path}") + expected_scope = _scope_dict(user_id, agent_name) + actual_scope = fact.get("scope") + scope_matches = isinstance(actual_scope, dict) and actual_scope.get("userId") == user_id and isinstance(actual_scope.get("agentName"), str) and actual_scope["agentName"].lower() == agent_name.lower() + if not scope_matches: + raise MemoryStorageCorruption(f"Fact scope mismatch for {fact_path}: expected {expected_scope!r}, found {fact.get('scope')!r}") try: - current_mtime = file_path.stat().st_mtime if file_path.exists() else None + return _normalize_fact(fact, scope=copy.deepcopy(actual_scope)) + except ValueError as exc: + raise MemoryStorageCorruption(f"Invalid canonical fact {fact_path}: {exc}") from exc + + def _load_agent_facts(self, path: Path, agent_name: str | None, *, user_id: str | None) -> list[dict[str, Any]]: + if agent_name is None: + return [] + facts: list[dict[str, Any]] = [] + for fact_path in sorted(agent_facts_directory(path, agent_name).glob("**/*.md")): + fact = _parse_fact_markdown(fact_path) + facts.append(self._validate_loaded_fact(fact, fact_path, user_id=user_id, agent_name=agent_name)) + # Shard directories are an internal layout detail and must not change + # the stable fact order observed by callers. + return sorted(facts, key=lambda fact: str(fact["id"])) + + def _read_fact(self, path: Path, fact_id: str, *, user_id: str | None, agent_name: str) -> tuple[dict[str, Any] | None, Path]: + fact_path = fact_file_path(path, fact_id, agent_name=agent_name) + if not fact_path.exists(): + return None, fact_path + fact = _parse_fact_markdown(fact_path) + return self._validate_loaded_fact(fact, fact_path, user_id=user_id, agent_name=agent_name), fact_path + + def _agent_entries(self, path: Path, agent_name: str | None, *, user_id: str | None) -> dict[str, dict[str, str]]: + if agent_name is None: + return {} + entries: dict[str, dict[str, str]] = {} + for fact_path in sorted(agent_facts_directory(path, agent_name).glob("**/*.md")): + fact = _parse_fact_markdown(fact_path) + self._validate_loaded_fact(fact, fact_path, user_id=user_id, agent_name=agent_name) + fact_id = str(fact.get("id") or "") + if not fact_id or fact_id != fact_path.stem: + raise MemoryStorageCorruption(f"Fact id mismatch for {fact_path}") + entries[fact_id] = {"path": fact_path.relative_to(path.parent).as_posix()} + return entries + + def _legacy_agent_memory_path(self, path: Path, agent_name: str) -> Path: + return path.parent / "agents" / agent_name.lower() / path.name + + def _global_json_needs_migration(self, path: Path) -> bool: + memory_file = self._load_memory_file(path) + return memory_file is not None and ("facts" in memory_file or memory_file.get("version") != DOCUMENT_VERSION) + + def _migrate_previous_default_bucket_locked( + self, + path: Path, + *, + user_id: str | None, + ) -> tuple[bool, list[RetrievalNotification]]: + """Move facts written by the earlier ``lead-agent`` default mapping. + + A real custom ``lead-agent`` owns a ``config.yaml`` and is never + touched. A directory with any other unexpected file is also preserved + and rejected instead of being guessed at or recursively deleted. + """ + legacy_agent_name = "lead-agent" + legacy_agent_dir = path.parent / "agents" / legacy_agent_name + if not legacy_agent_dir.exists() or (legacy_agent_dir / "config.yaml").is_file(): + return False, [] + unexpected = [child for child in legacy_agent_dir.iterdir() if child.name != "facts"] + legacy_facts_dir = legacy_agent_dir / "facts" + if legacy_facts_dir.exists(): + unexpected.extend(child for child in legacy_facts_dir.glob("**/*") if child.is_file() and child.suffix.lower() != ".md") + if unexpected: + names = ", ".join(sorted(child.name for child in unexpected)) + raise MemoryStorageCorruption(f"Cannot migrate previous default bucket {legacy_agent_dir}: unexpected entries {names}") + + legacy_facts = self._load_agent_facts(path, legacy_agent_name, user_id=user_id) + upserts: list[dict[str, Any]] = [] + for legacy_fact in legacy_facts: + candidate = _normalize_fact(legacy_fact, scope=_scope_dict(user_id, DEFAULT_AGENT_BUCKET)) + existing, _ = self._read_fact( + path, + candidate["id"], + user_id=user_id, + agent_name=DEFAULT_AGENT_BUCKET, + ) + if existing is not None: + if not self._migration_equivalent(candidate, existing): + raise MemoryStorageCorruption(f"Fact migration conflict for {candidate['id']!r}") + continue + upserts.append(candidate) + + current_memory = self._load_memory_file(path) + _, notifications = self._commit_changes_locked( + path, + user_id=user_id, + agent_name=DEFAULT_AGENT_BUCKET, + upserts=upserts, + deletes=[], + summaries=None, + expected_revision=int((current_memory or {}).get("revision") or 0), + ) + # Delete only the source Markdown files that were parsed above. Never + # recursively remove this directory: if an unexpected file appears + # concurrently, the final rmdir simply leaves it in place. + for fact_path in legacy_facts_dir.glob("**/*.md"): + fact_path.unlink(missing_ok=True) + directories = sorted( + (child for child in legacy_agent_dir.glob("**/*") if child.is_dir()), + key=lambda child: len(child.parts), + reverse=True, + ) + for directory in directories: + try: + directory.rmdir() + except OSError: + pass + try: + legacy_agent_dir.rmdir() except OSError: - current_mtime = None + pass + return True, notifications + def _run_read_migrations_locked( + self, + path: Path, + agent_name: str | None, + *, + user_id: str | None, + ) -> list[ScopedRetrievalNotifications]: + """Finish journal recovery and all migrations reachable from reads.""" + notifications_by_agent: list[ScopedRetrievalNotifications] = [] + self._recover_if_needed(path) + if self._global_json_needs_migration(path): + _, _, notifications = self._migrate_locked( + path, + DEFAULT_AGENT_BUCKET, + user_id=user_id, + include_global=True, + ) + if notifications: + notifications_by_agent.append((DEFAULT_AGENT_BUCKET, notifications)) + if agent_name is not None: + legacy_path = self._legacy_agent_memory_path(path, agent_name) + if legacy_path.exists(): + _, _, notifications = self._migrate_locked(path, agent_name, user_id=user_id, include_global=False) + if notifications: + notifications_by_agent.append((agent_name, notifications)) + if agent_name == DEFAULT_AGENT_BUCKET: + _, notifications = self._migrate_previous_default_bucket_locked(path, user_id=user_id) + if notifications: + notifications_by_agent.append((DEFAULT_AGENT_BUCKET, notifications)) + return notifications_by_agent + + @staticmethod + def _migration_equivalent(left: dict[str, Any], right: dict[str, Any]) -> bool: + ignored = {"revision", "createdAt", "updatedAt", "scope", "title", "schemaVersion"} + return {key: value for key, value in left.items() if key not in ignored} == {key: value for key, value in right.items() if key not in ignored} + + def _load_memory_file(self, path: Path) -> dict[str, Any] | None: + if not path.exists(): + return None + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError, UnicodeError) as exc: + raise MemoryStorageCorruption(f"Failed to load global memory JSON {path}: {exc}") from exc + if not isinstance(value, dict): + raise MemoryStorageCorruption(f"Global memory JSON {path} is not an object") + return value + + def _recover_if_needed(self, path: Path) -> None: + """Recover or clean a previously journaled multi-file operation. + + Callers hold the scope's in-process and cross-process locks. + """ + journal_path = path.parent / ".memory.journal.json" + if not journal_path.exists(): + return + try: + journal = json.loads(journal_path.read_text(encoding="utf-8")) + operation_id = str(journal["operationId"]) + if not operation_id or Path(operation_id).name != operation_id: + raise TypeError("operationId must be a plain path component") + state = journal.get("state") + old_entries = journal.get("oldEntries", {}) + agent_name = journal.get("agentName") + if agent_name is not None and not isinstance(agent_name, str): + raise TypeError("agentName must be a string or null") + except (OSError, json.JSONDecodeError, KeyError, TypeError) as exc: + raise MemoryStorageCorruption(f"Invalid memory operation journal {journal_path}: {exc}") from exc + recovery_dir = path.parent / ".recovery" / operation_id + if state == "prepared": + backup_manifest = recovery_dir / "memory.json" + if backup_manifest.exists(): + _atomic_write(path, backup_manifest.read_bytes()) + elif int(journal.get("expectedRevision") or 0) == 0: + path.unlink(missing_ok=True) + if isinstance(old_entries, dict): + old_ids = set(old_entries) + for fact_id in journal.get("factIds", []): + if fact_id not in old_ids: + if agent_name is None: + raise MemoryStorageCorruption(f"Journal {journal_path} contains facts without agentName") + fact_file_path(path, str(fact_id), agent_name=agent_name).unlink(missing_ok=True) + for fact_id, entry in old_entries.items(): + if not isinstance(fact_id, str) or not fact_id or any(character not in "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-" for character in fact_id): + raise MemoryStorageCorruption(f"Journal {journal_path} contains an invalid fact id") + if not isinstance(entry, dict) or not isinstance(entry.get("path"), str): + continue + backup = recovery_dir / f"{fact_id}.md" + if backup.exists(): + _atomic_write(_safe_relative_path(path.parent, entry["path"], label="journal"), backup.read_bytes()) + elif state != "committed": + raise MemoryStorageCorruption(f"Unknown journal state {state!r} in {journal_path}") + if recovery_dir.exists(): + shutil.rmtree(recovery_dir) + journal_path.unlink(missing_ok=True) + + def _commit_changes_locked( + self, + path: Path, + *, + user_id: str | None, + agent_name: str | None, + upserts: list[dict[str, Any]], + deletes: list[str], + summaries: dict[str, Any] | None, + expected_revision: int | None, + delete_revisions: dict[str, int] | None = None, + upsert_revisions: dict[str, int | None] | None = None, + ) -> tuple[dict[str, Any], list[RetrievalNotification]]: + """Commit only the addressed fact files plus the shared summary JSON. + + Callers hold both locks and have already run journal recovery. This is + deliberately not implemented as load-all/replace-all: unchanged fact + files are neither opened for backup nor rewritten nor re-indexed. + """ + current_memory = self._load_memory_file(path) + current_revision = int((current_memory or {}).get("revision") or 0) + if expected_revision is not None and expected_revision != current_revision: + raise MemoryManifestRevisionConflict(f"Expected user-memory revision {expected_revision}, found {current_revision}") + if (upserts or deletes) and agent_name is None: + raise ValueError("agent_name is required for fact repository changes") + + scope = _scope_dict(user_id, agent_name) + prepared: dict[str, tuple[dict[str, Any], dict[str, Any] | None, Path]] = {} + for incoming in upserts: + if not isinstance(incoming, dict): + raise ValueError("change_set.upserts must contain fact objects") + candidate = copy.deepcopy(incoming) + candidate["id"] = str(candidate.get("id") or f"fact_{uuid.uuid4().hex}") + fact_id = candidate["id"] + if fact_id in prepared: + raise ValueError(f"Duplicate fact id {fact_id!r} in upserts") + if agent_name is None: # guarded above + raise ValueError("agent_name is required for fact repository changes") + existing, fact_path = self._read_fact(path, fact_id, user_id=user_id, agent_name=agent_name) + if upsert_revisions is not None and fact_id in upsert_revisions: + expected_fact_revision = upsert_revisions[fact_id] + if expected_fact_revision is None and existing is not None: + raise MemoryFactRevisionConflict(f"Fact {fact_id!r} must not already exist") + if expected_fact_revision is not None: + actual_fact_revision = None if existing is None else existing.get("revision") + if actual_fact_revision != expected_fact_revision: + raise MemoryFactRevisionConflict(f"Expected fact {fact_id!r} revision {expected_fact_revision}, found {actual_fact_revision}") + normalized = _normalize_fact(candidate, scope=scope, existing=existing) + if existing != normalized: + prepared[fact_id] = (normalized, existing, fact_path) + + delete_ids = [str(fact_id) for fact_id in deletes] + if len(delete_ids) != len(set(delete_ids)): + raise ValueError("Duplicate fact ids are not allowed in deletes") + removals: dict[str, tuple[dict[str, Any], Path]] = {} + for fact_id in delete_ids: + if fact_id in prepared: + raise ValueError(f"Fact {fact_id!r} cannot be upserted and deleted together") + if agent_name is None: # guarded above + raise ValueError("agent_name is required for fact repository changes") + existing, fact_path = self._read_fact(path, fact_id, user_id=user_id, agent_name=agent_name) + if existing is None: + continue + if delete_revisions and fact_id in delete_revisions and delete_revisions[fact_id] != existing.get("revision"): + raise MemoryFactRevisionConflict(f"Expected fact {fact_id!r} revision {delete_revisions[fact_id]}, found {existing.get('revision')}") + removals[fact_id] = (existing, fact_path) + + base = current_memory or create_empty_memory() + user_section = copy.deepcopy(base.get("user", {})) + history_section = copy.deepcopy(base.get("history", {})) + if summaries is not None: + if not isinstance(summaries, dict): + raise ValueError("change_set.summaries must be an object") + if "user" in summaries: + if not isinstance(summaries["user"], dict): + raise ValueError("change_set.summaries.user must be an object") + user_section.update(copy.deepcopy(summaries["user"])) + if "history" in summaries: + if not isinstance(summaries["history"], dict): + raise ValueError("change_set.summaries.history must be an object") + history_section.update(copy.deepcopy(summaries["history"])) + summaries_changed = user_section != base.get("user", {}) or history_section != base.get("history", {}) + needs_manifest_cleanup = current_memory is None or current_memory.get("version") != DOCUMENT_VERSION or "facts" in current_memory + if not prepared and not removals and not summaries_changed and not needs_manifest_cleanup: + memory_file = current_memory or { + "version": DOCUMENT_VERSION, + "revision": 0, + "lastUpdated": base.get("lastUpdated", utc_now_iso_z()), + "user": user_section, + "history": history_section, + } + return memory_file, [] + + next_revision = current_revision + 1 + old_entries: dict[str, dict[str, str]] = {} + for fact_id, (_, existing, fact_path) in prepared.items(): + if existing is not None: + old_entries[fact_id] = {"path": fact_path.relative_to(path.parent).as_posix()} + for fact_id, (_, fact_path) in removals.items(): + old_entries[fact_id] = {"path": fact_path.relative_to(path.parent).as_posix()} + fact_ids = list(prepared) + list(removals) + journal = { + "operationId": uuid.uuid4().hex, + "state": "prepared", + "agentName": agent_name, + "expectedRevision": current_revision, + "nextRevision": next_revision, + "factIds": fact_ids, + "oldEntries": old_entries, + } + journal_path = path.parent / ".memory.journal.json" + recovery_dir = path.parent / ".recovery" / journal["operationId"] + recovery_dir.mkdir(parents=True, exist_ok=True) + if current_memory is not None: + shutil.copy2(path, recovery_dir / "memory.json") + for fact_id, entry in old_entries.items(): + old_path = _safe_relative_path(path.parent, entry["path"], label="journal") + if old_path.exists(): + shutil.copy2(old_path, recovery_dir / f"{fact_id}.md") + _atomic_write(journal_path, json.dumps(journal, ensure_ascii=False, indent=2).encode("utf-8")) + + notifications: list[RetrievalNotification] = [] + for fact_id, (fact, _, fact_path) in prepared.items(): + _atomic_write(fact_path, _render_fact_markdown(fact)) + notifications.append(("upsert", fact, str(fact_path))) + for fact_id, (_, fact_path) in removals.items(): + fact_path.unlink(missing_ok=True) + notifications.append(("remove", fact_id, None)) + + memory_file = { + "version": DOCUMENT_VERSION, + "revision": next_revision, + "lastUpdated": utc_now_iso_z(), + "user": user_section, + "history": history_section, + } + _atomic_write(path, json.dumps(memory_file, ensure_ascii=False, indent=2).encode("utf-8")) + journal["state"] = "committed" + _atomic_write(journal_path, json.dumps(journal, ensure_ascii=False, indent=2).encode("utf-8")) + shutil.rmtree(recovery_dir, ignore_errors=True) + journal_path.unlink(missing_ok=True) with self._cache_lock: - cached = self._memory_cache.get(cache_key) - if cached is not None and cached[1] == current_mtime: - return cached[0] + for cache_key in [cache_key for cache_key in self._memory_cache if cache_key[0] == user_id]: + self._memory_cache.pop(cache_key, None) + return memory_file, notifications - memory_data = self._load_memory_from_file(agent_name, user_id=user_id) + def _migrate_locked( + self, + path: Path, + agent_name: str, + *, + user_id: str | None, + include_global: bool, + adopt_legacy_summaries: bool = True, + ) -> tuple[bool, str | None, list[RetrievalNotification]]: + """Merge legacy facts without overwriting an existing canonical fact.""" + sources: list[tuple[Path, dict[str, Any]]] = [] + legacy_path = self._legacy_agent_memory_path(path, agent_name) + legacy_memory = self._load_memory_file(legacy_path) + if legacy_memory is not None: + sources.append((legacy_path, legacy_memory)) + global_memory = self._load_memory_file(path) + from_version = None if global_memory is None else global_memory.get("version") + if include_global and global_memory is not None and ("facts" in global_memory or global_memory.get("version") != DOCUMENT_VERSION): + sources.append((path, global_memory)) + if not sources: + return False, from_version, [] + base = global_memory or create_empty_memory() + migrated_summaries = { + "user": copy.deepcopy(base.get("user", {})), + "history": copy.deepcopy(base.get("history", {})), + } + if legacy_memory is not None and adopt_legacy_summaries: + for section in ("user", "history"): + migrated_summaries[section] = _merge_legacy_summary_section( + canonical=migrated_summaries[section], + legacy=legacy_memory.get(section, {}), + section=section, + legacy_path=legacy_path, + ) + + upserts: list[dict[str, Any]] = [] + pending: dict[str, dict[str, Any]] = {} + for source_path, source_memory in sources: + source_document = self._document_from_memory_file( + source_memory, + source_path, + agent_name, + user_id=user_id, + allow_legacy_facts=True, + ) + for raw_fact in source_document.get("facts", []): + if not isinstance(raw_fact, dict): + raise MemoryStorageCorruption(f"Legacy fact in {source_path} is not an object") + candidate = _normalize_fact(raw_fact, scope=_scope_dict(user_id, agent_name)) + existing, _ = self._read_fact(path, candidate["id"], user_id=user_id, agent_name=agent_name) + if existing is not None: + if not self._migration_equivalent(candidate, existing): + raise MemoryStorageCorruption(f"Fact migration conflict for {candidate['id']!r}") + continue + previous = pending.get(candidate["id"]) + if previous is not None: + if not self._migration_equivalent(candidate, previous): + raise MemoryStorageCorruption(f"Fact migration conflict for {candidate['id']!r}") + continue + pending[candidate["id"]] = candidate + upserts.append(candidate) + + # Migration is intentionally one-way for the running application. + # Preserve every destructive v1 JSON source before the first v2 write; + # a failed/mismatched backup aborts while all source data is untouched. + for source_path, _ in sources: + _ensure_migration_backup(source_path) + + current_revision = int((global_memory or {}).get("revision") or 0) + _, notifications = self._commit_changes_locked( + path, + user_id=user_id, + agent_name=agent_name, + upserts=upserts, + deletes=[], + summaries=migrated_summaries, + expected_revision=current_revision, + ) + if legacy_memory is not None: + legacy_path.unlink(missing_ok=True) + return True, from_version, notifications + + def _document_from_memory_file( + self, + memory_file: dict[str, Any], + path: Path, + agent_name: str | None, + *, + user_id: str | None, + allow_legacy_facts: bool = False, + ) -> dict[str, Any]: + """Build the compatibility document without persisting facts in JSON.""" + legacy_facts = memory_file.get("facts") + contains_owned_legacy_facts = isinstance(legacy_facts, dict) or (isinstance(legacy_facts, list) and bool(legacy_facts)) + if contains_owned_legacy_facts and not allow_legacy_facts: + raise MemoryStorageCorruption(f"Legacy facts in {path} require explicit migrate(user_id=..., agent_name=...)") + if isinstance(legacy_facts, list): + facts = copy.deepcopy(legacy_facts) if agent_name is not None else [] + elif isinstance(legacy_facts, dict): + facts = [] + if agent_name is not None: + for fact_id, entry in legacy_facts.items(): + if not isinstance(entry, dict) or not isinstance(entry.get("path"), str): + raise MemoryStorageCorruption(f"Invalid legacy manifest entry for fact {fact_id!r}") + fact_path = _safe_relative_path(path.parent, entry["path"], label="legacy fact") + fact = _parse_fact_markdown(fact_path) + if str(fact.get("id")) != str(fact_id): + raise MemoryStorageCorruption(f"Legacy manifest id mismatch for fact {fact_id!r}") + if entry.get("contentHash") and entry.get("contentHash") != _content_hash(fact_path.read_bytes()): + raise MemoryStorageCorruption(f"Hash mismatch for canonical fact {fact_id!r}") + facts.append(fact) + elif legacy_facts is None: + facts = self._load_agent_facts(path, agent_name, user_id=user_id) + else: + raise MemoryStorageCorruption(f"Legacy facts in {path} must be a list or mapping") + result = {key: copy.deepcopy(value) for key, value in memory_file.items() if key != "facts"} + result.setdefault("revision", 0) + result["facts"] = facts + return result + + def _read_document(self, path: Path, agent_name: str | None, *, user_id: str | None) -> dict[str, Any]: + memory_file = self._load_memory_file(path) + if memory_file is None: + result = create_empty_memory() + result["facts"] = self._load_agent_facts(path, agent_name, user_id=user_id) + return result + return self._document_from_memory_file(memory_file, path, agent_name, user_id=user_id) + + def load(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + journal_path = path.parent / ".memory.journal.json" + legacy_path = self._legacy_agent_memory_path(path, agent_name) if agent_name is not None else None + previous_default_dir = path.parent / "agents" / "lead-agent" + needs_migration = ( + journal_path.exists() + or (legacy_path is not None and legacy_path.exists()) + or self._global_json_needs_migration(path) + or (agent_name == DEFAULT_AGENT_BUCKET and previous_default_dir.exists() and not (previous_default_dir / "config.yaml").is_file()) + ) + migration_notifications: list[ScopedRetrievalNotifications] = [] + if needs_migration: + with self._scope_lock(key), _process_file_lock(path.parent / ".memory.lock", float(getattr(self._config, "file_lock_timeout_seconds", 10))): + migration_notifications = self._run_read_migrations_locked(path, agent_name, user_id=user_id) + for notification_agent, notifications in migration_notifications: + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=notification_agent) + signature = self._scope_signature(path, agent_name) with self._cache_lock: - self._memory_cache[cache_key] = (memory_data, current_mtime) - - return memory_data + cached = self._memory_cache.get(key) + if cached is not None and cached[1] == signature: + return copy.deepcopy(cached[0]) + document = self._read_document(path, agent_name, user_id=user_id) + with self._cache_lock: + self._memory_cache[key] = (copy.deepcopy(document), signature) + return copy.deepcopy(document) def reload(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Reload memory data from file, forcing cache invalidation.""" - file_path = self._get_memory_file_path(agent_name, user_id=user_id) - memory_data = self._load_memory_from_file(agent_name, user_id=user_id) - cache_key = self._cache_key(agent_name, user_id=user_id) - - try: - mtime = file_path.stat().st_mtime if file_path.exists() else None - except OSError: - mtime = None - + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + legacy_path = self._legacy_agent_memory_path(path, agent_name) if agent_name is not None else None + previous_default_dir = path.parent / "agents" / "lead-agent" + needs_migration = ( + (path.parent / ".memory.journal.json").exists() + or (legacy_path is not None and legacy_path.exists()) + or self._global_json_needs_migration(path) + or (agent_name == DEFAULT_AGENT_BUCKET and previous_default_dir.exists() and not (previous_default_dir / "config.yaml").is_file()) + ) + migration_notifications: list[ScopedRetrievalNotifications] = [] + if needs_migration: + with self._scope_lock(key), _process_file_lock(path.parent / ".memory.lock", float(getattr(self._config, "file_lock_timeout_seconds", 10))): + migration_notifications = self._run_read_migrations_locked(path, agent_name, user_id=user_id) + for notification_agent, notifications in migration_notifications: + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=notification_agent) + document = self._read_document(path, agent_name, user_id=user_id) + signature = self._scope_signature(path, agent_name) with self._cache_lock: - self._memory_cache[cache_key] = (memory_data, mtime) - return memory_data + self._memory_cache[key] = (copy.deepcopy(document), signature) + return copy.deepcopy(document) - def save(self, memory_data: dict[str, Any], agent_name: str | None = None, *, user_id: str | None = None) -> bool: - """Save memory data to file and update cache.""" - file_path = self._get_memory_file_path(agent_name, user_id=user_id) - cache_key = self._cache_key(agent_name, user_id=user_id) + def migrate( + self, + *, + user_id: str | None = None, + agent_name: str | None = None, + ) -> dict[str, Any]: + """Run the idempotent version-driven migration for one exact scope.""" + if agent_name is None: + raise ValueError("agent_name is required to migrate legacy facts") + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + with self._scope_lock(key), _process_file_lock(path.parent / ".memory.lock", float(getattr(self._config, "file_lock_timeout_seconds", 10))): + self._recover_if_needed(path) + migrated, from_version, notifications = self._migrate_locked(path, agent_name, user_id=user_id, include_global=True) + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=agent_name) + document = self.reload(agent_name, user_id=user_id) + return { + "migrated": migrated, + "fromVersion": from_version, + "toVersion": document.get("version"), + "revision": document.get("revision", 0), + } + def save( + self, + memory_data: dict[str, Any], + agent_name: str | None = None, + *, + user_id: str | None = None, + expected_revision: int | None = None, + ) -> bool: + """Compatibility full replacement, diffed into per-fact operations. + + This API must scan the selected agent to determine which omitted facts + are deletions, but the commit writes only new/changed/deleted facts. + Repository callers should prefer ``apply_changes`` to avoid even that + full comparison scan. + """ + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + lock_path = path.parent / ".memory.lock" + notifications: list[RetrievalNotification] = [] try: - file_path.parent.mkdir(parents=True, exist_ok=True) - # Shallow-copy before adding lastUpdated so the caller's dict is not - # mutated as a side-effect, and the cache reference is not silently - # updated before the file write succeeds. - memory_data = {**memory_data, "lastUpdated": utc_now_iso_z()} - - temp_path = file_path.with_suffix(f".{uuid.uuid4().hex}.tmp") - with open(temp_path, "w", encoding="utf-8") as f: - json.dump(memory_data, f, indent=2, ensure_ascii=False) - - temp_path.replace(file_path) - - try: - mtime = file_path.stat().st_mtime - except OSError: - mtime = None - - with self._cache_lock: - self._memory_cache[cache_key] = (memory_data, mtime) - logger.info("Memory saved to %s", file_path) - return True - except OSError as e: - logger.error("Failed to save memory file: %s", e) + if not isinstance(memory_data, dict): + raise ValueError("memory_data must be an object") + if agent_name is not None and "facts" not in memory_data: + raise ValueError("memory_data.facts is required for an agent full save") + facts_raw = memory_data.get("facts", []) + if not isinstance(facts_raw, list): + raise ValueError("memory_data.facts must be a list") + if any(not isinstance(fact, dict) for fact in facts_raw): + raise ValueError("memory_data.facts must contain only fact objects") + if agent_name is None and facts_raw: + raise ValueError("agent_name is required to persist facts") + with self._scope_lock(key), _process_file_lock(lock_path, float(getattr(self._config, "file_lock_timeout_seconds", 10))): + self._recover_if_needed(path) + ids = [str(fact.get("id") or "") for fact in facts_raw] + if len(ids) != len(set(ids)): + raise ValueError("Duplicate fact ids are not allowed") + old_ids = set(self._agent_entries(path, agent_name, user_id=user_id)) if agent_name is not None else set() + summaries = None + if agent_name is None: + summaries = {"user": memory_data.get("user", {}), "history": memory_data.get("history", {})} + _, notifications = self._commit_changes_locked( + path, + user_id=user_id, + agent_name=agent_name, + upserts=copy.deepcopy(facts_raw), + deletes=sorted(old_ids - set(ids)), + summaries=summaries, + expected_revision=expected_revision, + ) + document = self._read_document(path, agent_name, user_id=user_id) + signature = self._scope_signature(path, agent_name) + with self._cache_lock: + self._memory_cache[key] = (copy.deepcopy(document), signature) + except MemoryRevisionConflict: + raise + except (OSError, ValueError, MemoryStorageCorruption) as exc: + logger.error("Failed to save memory scope %s: %s", key, exc) return False + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=agent_name) + return True -def create_storage(config: DeerMemConfig) -> MemoryStorage: - """Build the configured memory storage instance for ``config``. + def clear_all(self, *, user_id: str | None = None) -> dict[str, Any]: + """Clear one user's summaries and all agent facts, preserving agent configs.""" + path = self._get_memory_file_path(user_id=user_id) + key = self._cache_key(user_id=user_id) + notifications_by_agent: list[ScopedRetrievalNotifications] = [] + with ( + self._scope_lock(key), + _process_file_lock( + path.parent / ".memory.lock", + float(getattr(self._config, "file_lock_timeout_seconds", 10)), + ), + ): + self._recover_if_needed(path) + agents_root = path.parent / "agents" + if agents_root.exists(): + for agent_dir in sorted(child for child in agents_root.iterdir() if child.is_dir()): + agent_name = agent_dir.name + validate_agent_name(agent_name) + legacy_path = self._legacy_agent_memory_path(path, agent_name) + if legacy_path.exists(): + _, _, migration_notifications = self._migrate_locked( + path, + agent_name, + user_id=user_id, + include_global=False, + adopt_legacy_summaries=False, + ) + if migration_notifications: + notifications_by_agent.append((agent_name, migration_notifications)) + facts = self._load_agent_facts(path, agent_name, user_id=user_id) + if not facts: + continue + current_memory = self._load_memory_file(path) + _, notifications = self._commit_changes_locked( + path, + user_id=user_id, + agent_name=agent_name, + upserts=[], + deletes=[str(fact["id"]) for fact in facts], + summaries=None, + expected_revision=int((current_memory or {}).get("revision") or 0), + delete_revisions={str(fact["id"]): int(fact.get("revision") or 1) for fact in facts}, + ) + notifications_by_agent.append((agent_name, notifications)) - Replaces the old ``get_memory_storage()`` global singleton: the caller - (``DeerMem.__init__``) owns the returned instance. Empty ``storage_class`` - (default) -> ``FileMemoryStorage`` directly (no importlib, portable); a - dotted path is resolved and raises ``ValueError`` on failure (fail-fast: - memory is persistent state, so an unresolved ``storage_class`` is not - silently substituted with ``FileMemoryStorage`` -- mirrors the - ``manager_class`` resolution policy). - """ + empty = create_empty_memory() + current_memory = self._load_memory_file(path) + self._commit_changes_locked( + path, + user_id=user_id, + agent_name=None, + upserts=[], + deletes=[], + summaries={"user": empty["user"], "history": empty["history"]}, + expected_revision=int((current_memory or {}).get("revision") or 0), + ) + + for agent_name, notifications in notifications_by_agent: + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=agent_name) + return self.reload(DEFAULT_AGENT_BUCKET, user_id=user_id) + + @staticmethod + def _scope_kwargs(scope: dict[str, str | None]) -> dict[str, str]: + kwargs: dict[str, str] = {} + if scope.get("userId") is not None: + kwargs["user_id"] = str(scope["userId"]) + if scope.get("agentName") is not None: + kwargs["agent_name"] = str(scope["agentName"]) + return kwargs + + def get_fact( + self, + fact_id: str, + *, + user_id: str | None = None, + agent_name: str | None = None, + ) -> dict[str, Any] | None: + if agent_name is None: + raise ValueError("agent_name is required to get a fact") + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + legacy_path = self._legacy_agent_memory_path(path, agent_name) + notifications: list[RetrievalNotification] = [] + with self._scope_lock(key), _process_file_lock(path.parent / ".memory.lock", float(getattr(self._config, "file_lock_timeout_seconds", 10))): + self._recover_if_needed(path) + if legacy_path.exists(): + _, _, notifications = self._migrate_locked(path, agent_name, user_id=user_id, include_global=False) + fact, _ = self._read_fact(path, fact_id, user_id=user_id, agent_name=agent_name) + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=agent_name) + return copy.deepcopy(fact) + + def list_facts( + self, + *, + user_id: str | None = None, + agent_name: str | None = None, + filters: dict[str, Any] | None = None, + cursor: int = 0, + limit: int = 100, + ) -> list[dict[str, Any]]: + if cursor < 0 or limit < 1: + raise ValueError("cursor must be >= 0 and limit must be >= 1") + facts = self.load(agent_name, user_id=user_id).get("facts", []) + filters = filters or {} + matched = [fact for fact in facts if all(key in fact and fact.get(key) == value for key, value in filters.items())] + return copy.deepcopy(matched[cursor : cursor + limit]) + + def apply_changes( + self, + change_set: dict[str, Any], + *, + user_id: str | None = None, + agent_name: str | None = None, + expected_manifest_revision: int | None = None, + allow_manifest_rebase: bool = False, + ) -> dict[str, Any]: + """Commit an incremental change set and return only the applied delta. + + ``complete`` is deliberately false: callers that require the historical + full document must explicitly call ``load``. This prevents a fresh + process from presenting a one-fact cache snapshot as the whole agent + memory while keeping the mutation path free of full fact scans. + """ + has_fact_changes = bool(change_set.get("upserts") or change_set.get("deletes")) + if has_fact_changes and agent_name is None: + raise ValueError("agent_name is required for fact repository changes") + summaries = change_set.get("summaries") + upserts = copy.deepcopy(change_set.get("upserts", [])) + deletes = change_set.get("deletes", []) + delete_revisions = change_set.get("deleteRevisions") + upsert_revisions = change_set.get("upsertRevisions") + if not isinstance(upserts, list) or not isinstance(deletes, list): + raise ValueError("change_set.upserts and change_set.deletes must be lists") + if delete_revisions is not None and not isinstance(delete_revisions, dict): + raise ValueError("change_set.deleteRevisions must be an object") + if upsert_revisions is not None and not isinstance(upsert_revisions, dict): + raise ValueError("change_set.upsertRevisions must be an object") + + normalized_upsert_revisions: dict[str, int | None] = {} + for incoming in upserts: + if not isinstance(incoming, dict): + raise ValueError("change_set.upserts must contain fact objects") + incoming["id"] = str(incoming.get("id") or f"fact_{uuid.uuid4().hex}") + fact_id = incoming["id"] + if isinstance(upsert_revisions, dict) and fact_id in upsert_revisions: + expected_fact_revision = upsert_revisions[fact_id] + else: + expected_fact_revision = incoming.get("revision") if "revision" in incoming else None + if expected_fact_revision is not None and (isinstance(expected_fact_revision, bool) or not isinstance(expected_fact_revision, int) or expected_fact_revision < 1): + raise ValueError("change_set.upsertRevisions values must be null or integers >= 1") + normalized_upsert_revisions[fact_id] = expected_fact_revision + + path = self._get_memory_file_path(agent_name, user_id=user_id) + key = self._cache_key(agent_name, user_id=user_id) + expected = expected_manifest_revision + notifications: list[RetrievalNotification] = [] + memory_file: dict[str, Any] | None = None + safe_delete_rebase = not deletes or (isinstance(delete_revisions, dict) and all(str(fact_id) in delete_revisions for fact_id in deletes)) + safe_upsert_rebase = all(str(incoming["id"]) in normalized_upsert_revisions for incoming in upserts) + for attempt in range(3): + try: + with self._scope_lock(key), _process_file_lock(path.parent / ".memory.lock", float(getattr(self._config, "file_lock_timeout_seconds", 10))): + self._recover_if_needed(path) + memory_file, notifications = self._commit_changes_locked( + path, + user_id=user_id, + agent_name=agent_name, + upserts=upserts, + deletes=[str(fact_id) for fact_id in deletes], + summaries=copy.deepcopy(summaries), + expected_revision=expected, + delete_revisions=copy.deepcopy(delete_revisions), + upsert_revisions=normalized_upsert_revisions, + ) + break + except MemoryManifestRevisionConflict as exc: + can_rebase = allow_manifest_rebase and has_fact_changes and summaries is None and safe_delete_rebase and safe_upsert_rebase and attempt < 2 + if not can_rebase: + raise + current = self._load_memory_file(path) + expected = int((current or {}).get("revision") or 0) + logger.info("Rebasing disjoint memory fact change after revision conflict: %s", exc) + self._dispatch_retrieval_notifications(notifications, user_id=user_id, agent_name=agent_name) + if memory_file is None: # defensive: the bounded loop either commits or raises + raise MemoryStorageError("Memory repository change did not produce a result") + return { + "complete": False, + "version": memory_file.get("version", DOCUMENT_VERSION), + "revision": memory_file.get("revision", 0), + "lastUpdated": memory_file.get("lastUpdated", ""), + "upsertedFacts": [copy.deepcopy(value) for action, value, _ in notifications if action == "upsert" and isinstance(value, dict)], + "deletedFactIds": [str(value) for action, value, _ in notifications if action == "remove"], + } + + def upsert_fact( + self, + fact: dict[str, Any], + *, + user_id: str | None = None, + agent_name: str | None = None, + expected_manifest_revision: int | None = None, + expected_fact_revision: int | None = None, + ) -> dict[str, Any]: + if agent_name is None: + raise ValueError("agent_name is required to upsert a fact") + incoming = copy.deepcopy(fact) + incoming["id"] = str(incoming.get("id") or f"fact_{uuid.uuid4().hex}") + fact_id = incoming["id"] + return self.apply_changes( + {"upserts": [incoming], "upsertRevisions": {fact_id: expected_fact_revision}}, + user_id=user_id, + agent_name=agent_name, + expected_manifest_revision=expected_manifest_revision, + allow_manifest_rebase=True, + ) + + def delete_fact( + self, + fact_id: str, + *, + user_id: str | None = None, + agent_name: str | None = None, + expected_manifest_revision: int | None = None, + expected_fact_revision: int | None = None, + ) -> dict[str, Any]: + if agent_name is None: + raise ValueError("agent_name is required to delete a fact") + return self.apply_changes( + { + "deletes": [fact_id], + "deleteRevisions": ({fact_id: expected_fact_revision} if expected_fact_revision is not None else None), + }, + user_id=user_id, + agent_name=agent_name, + expected_manifest_revision=expected_manifest_revision, + allow_manifest_rebase=True, + ) + + def get_summaries( + self, + *, + user_id: str | None = None, + agent_name: str | None = None, + ) -> dict[str, Any]: + document = self.load(agent_name, user_id=user_id) + return {"user": copy.deepcopy(document.get("user", {})), "history": copy.deepcopy(document.get("history", {})), "revision": document.get("revision", 0)} + + def update_summaries( + self, + summaries: dict[str, Any], + *, + user_id: str | None = None, + agent_name: str | None = None, + expected_revision: int | None = None, + ) -> dict[str, Any]: + # Summaries are always user-global, never agent-specific. + document = self.load(user_id=user_id) + document.update({key: copy.deepcopy(value) for key, value in summaries.items() if key in {"user", "history"}}) + expected = int(document.get("revision") or 0) if expected_revision is None else expected_revision + if not self.save(document, user_id=user_id, expected_revision=expected): + raise MemoryStorageError("Failed to update global memory summaries") + return self.reload(user_id=user_id) + + def notify_fact_upsert(self, fact: dict[str, Any], *, path: str = "") -> bool: + if self._retrieval is None: + return False + scope = fact.get("scope") if isinstance(fact.get("scope"), dict) else {} + self._retrieval.upsert(copy.deepcopy(fact), scope=copy.deepcopy(scope), path=path) + return True + + def notify_fact_remove(self, fact_id: str, *, scope: dict[str, str | None]) -> bool: + if self._retrieval is None: + return False + self._retrieval.remove(fact_id, scope=copy.deepcopy(scope)) + return True + + def search_facts( + self, + query: str, + *, + scopes: list[dict[str, str | None]], + top_k: int = 10, + mode: str = "hybrid", + filters: dict[str, Any] | None = None, + ) -> list[dict[str, Any]]: + if self._retrieval is not None: + return self._retrieval.search(query, scopes=scopes, top_k=top_k, mode=mode, filters=filters) + query_lower = query.strip().lower() + if not query_lower or top_k <= 0: + return [] + results: list[dict[str, Any]] = [] + for scope in scopes: + facts = self.list_facts(filters=filters, **self._scope_kwargs(scope)) + for fact in facts: + content = fact.get("content") + if isinstance(content, str) and query_lower in content.lower(): + results.append({"fact": fact, "score": float(fact.get("confidence") or 0.5), "matchType": "substring"}) + results.sort(key=lambda result: result["score"], reverse=True) + return results[:top_k] + + def rebuild_index(self, scopes: list[dict[str, str | None]] | None = None) -> dict[str, Any]: + if self._retrieval is None: + return {"supported": False, "indexed": 0, "failed": 0, "reason": "retrieval_not_configured"} + indexed = 0 + failed = 0 + if scopes is None: + root = Path(self._config.storage_path) if self._config.storage_path else memory_file_path(self._config).parent + candidates = root.glob("**/facts/**/*.md") + for path in candidates: + try: + fact = _parse_fact_markdown(path) + relative_parts = path.relative_to(root).parts + agents_index = relative_parts.index("agents") + expected_agent = relative_parts[agents_index + 1] + expected_user_bucket = relative_parts[1] if len(relative_parts) > 1 and relative_parts[0] == "users" else None + fact_scope = fact.get("scope") if isinstance(fact.get("scope"), dict) else {} + original_user = fact_scope.get("userId") + if original_user is not None and not isinstance(original_user, str): + raise MemoryStorageCorruption(f"Fact scope userId is invalid for {path}") + if expected_user_bucket is not None and (original_user is None or safe_user_id(original_user) != expected_user_bucket): + raise MemoryStorageCorruption(f"Fact user scope does not match directory for {path}") + validate_agent_name(expected_agent) + self._validate_loaded_fact(fact, path, user_id=original_user, agent_name=expected_agent) + self.notify_fact_upsert(fact, path=str(path)) + indexed += 1 + except Exception: + logger.exception("Failed to rebuild retrieval index for %s", path) + failed += 1 + else: + for scope in scopes: + kwargs = self._scope_kwargs(scope) + memory_path = self._get_memory_file_path(**kwargs) + agent_name = kwargs.get("agent_name") + if agent_name is None: + continue + for fact in self.list_facts(**kwargs): + try: + self.notify_fact_upsert(fact, path=str(fact_file_path(memory_path, fact["id"], agent_name=agent_name))) + indexed += 1 + except Exception: + logger.exception("Failed to rebuild retrieval index for fact %s", fact.get("id")) + failed += 1 + return {"supported": True, "indexed": indexed, "failed": failed} + + def retrieval_status(self) -> dict[str, Any]: + return { + "configured": self._retrieval is not None, + "mode": "external" if self._retrieval is not None else "substring_fallback", + } + + def capabilities(self) -> set[str]: + capabilities = {"file", "markdown-facts", "global-summary-json", "revision", "journal", "fact-repository", "substring-fallback"} + if self._retrieval is not None: + capabilities.add("retrieval") + return capabilities + + +def create_storage(config: DeerMemConfig, retrieval: RetrievalPort | None = None) -> MemoryStorage: + if retrieval is None and config.retrieval_adapter: + try: + module_path, factory_name = config.retrieval_adapter.rsplit(".", 1) + factory = getattr(importlib.import_module(module_path), factory_name) + retrieval = factory(config) + except Exception as exc: + raise ValueError(f"backend_config.retrieval_adapter={config.retrieval_adapter!r} failed to load: {exc}") from exc storage_class_path = config.storage_class - if not storage_class_path: - return FileMemoryStorage(config) - + if not storage_class_path or storage_class_path == "file": + return FileMemoryStorage(config, retrieval=retrieval) try: module_path, class_name = storage_class_path.rsplit(".", 1) - import importlib - - module = importlib.import_module(module_path) - storage_class = getattr(module, class_name) - - # Validate that the configured storage is a MemoryStorage implementation - if not isinstance(storage_class, type): - raise TypeError(f"Configured memory storage '{storage_class_path}' is not a class: {storage_class!r}") - if not issubclass(storage_class, MemoryStorage): - raise TypeError(f"Configured memory storage '{storage_class_path}' is not a subclass of MemoryStorage") - + storage_class = getattr(importlib.import_module(module_path), class_name) + if not isinstance(storage_class, type) or not issubclass(storage_class, MemoryStorage): + raise TypeError(f"Configured memory storage '{storage_class_path}' is not a MemoryStorage class") return storage_class(config) - except Exception as e: - raise ValueError( - f"backend_config.storage_class={storage_class_path!r} failed to load: {e}. " - "Refusing to silently fall back to FileMemoryStorage - memory is persistent " - "state, so a wrong store is a silent data-integrity footgun (a misspelled " - "class path would otherwise write every fact to local JSON instead of the " - "intended backend)." - ) from e + except Exception as exc: + raise ValueError(f"backend_config.storage_class={storage_class_path!r} failed to load: {exc}. Refusing to silently fall back because memory is persistent state.") from exc diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py index e6b8b822c..44a11df6f 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py @@ -20,6 +20,7 @@ from .prompt import ( load_prompt_messages, ) from .storage import ( + MemoryManifestRevisionConflict, MemoryStorage, create_empty_memory, utc_now_iso_z, @@ -630,9 +631,19 @@ class MemoryUpdater: # ── Data access + fact CRUD (formerly module-level functions; use self._storage) ── - def _save_memory_to_file(self, memory_data: dict[str, Any], agent_name: str | None = None, *, user_id: str | None = None) -> bool: + def _save_memory_to_file( + self, + memory_data: dict[str, Any], + agent_name: str | None = None, + *, + user_id: str | None = None, + expected_revision: int | None = None, + ) -> bool: """Persist memory data via the injected storage.""" - return self._storage.save(memory_data, agent_name, user_id=user_id) + kwargs: dict[str, Any] = {"user_id": user_id} + if expected_revision is not None: + kwargs["expected_revision"] = expected_revision + return self._storage.save(memory_data, agent_name, **kwargs) def get_memory_data(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: """Get the current memory data via the injected storage.""" @@ -644,14 +655,90 @@ class MemoryUpdater: def import_memory_data(self, memory_data: dict[str, Any], agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: """Persist imported memory data via the injected storage.""" + if not isinstance(memory_data, dict): + raise ValueError("memory_data") + memory_data = copy.deepcopy(memory_data) + empty = create_empty_memory() + for section in ("user", "history"): + incoming_section = memory_data.get(section, {}) + if not isinstance(incoming_section, dict): + raise ValueError(f"memory_data.{section}") + complete_section = copy.deepcopy(empty[section]) + for key, value in incoming_section.items(): + if key in complete_section and isinstance(complete_section[key], dict) and isinstance(value, dict): + complete_section[key].update(copy.deepcopy(value)) + else: + complete_section[key] = copy.deepcopy(value) + memory_data[section] = complete_section + if agent_name is not None and getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + current = self.get_memory_data(agent_name, user_id=user_id) + incoming_facts = copy.deepcopy(memory_data.get("facts", [])) + if not isinstance(incoming_facts, list) or any(not isinstance(fact, dict) for fact in incoming_facts): + raise ValueError("memory_data.facts") + for fact in incoming_facts: + fact["id"] = str(fact.get("id") or f"fact_{uuid.uuid4().hex}") + fact["confidence"] = _coerce_source_confidence(fact) + current_by_id = {str(fact.get("id")): fact for fact in current.get("facts", []) if isinstance(fact, dict)} + incoming_ids = {str(fact.get("id")) for fact in incoming_facts} + self._storage.apply_changes( + { + "upserts": incoming_facts, + "upsertRevisions": {str(fact.get("id")): (int(current_by_id[str(fact.get("id"))].get("revision") or 1) if str(fact.get("id")) in current_by_id else None) for fact in incoming_facts}, + "deletes": [fact_id for fact_id in current_by_id if fact_id not in incoming_ids], + "deleteRevisions": {fact_id: int(fact.get("revision") or 1) for fact_id, fact in current_by_id.items() if fact_id not in incoming_ids}, + "summaries": {"user": copy.deepcopy(memory_data.get("user", {})), "history": copy.deepcopy(memory_data.get("history", {}))}, + }, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(current.get("revision") or 0), + ) + return self._storage.load(agent_name, user_id=user_id) + if agent_name is None: + memory_data["facts"] = [] if not self._storage.save(memory_data, agent_name, user_id=user_id): raise OSError("Failed to save imported memory data") return self._storage.load(agent_name, user_id=user_id) def clear_memory_data(self, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: - """Clear all stored memory data and persist an empty structure.""" + """Clear one selected agent's facts without resetting shared summaries.""" + if agent_name is not None and getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + for attempt in range(3): + current = self.get_memory_data(agent_name, user_id=user_id) if attempt == 0 else self.reload_memory_data(agent_name, user_id=user_id) + facts = [fact for fact in current.get("facts", []) if isinstance(fact, dict)] + try: + self._storage.apply_changes( + { + "deletes": [str(fact.get("id")) for fact in facts], + "deleteRevisions": {str(fact.get("id")): int(fact.get("revision") or 1) for fact in facts}, + }, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(current.get("revision") or 0), + ) + return self.reload_memory_data(agent_name, user_id=user_id) + except MemoryManifestRevisionConflict: + if attempt == 2: + raise + logger.info("Retrying scoped memory clear from a fresh snapshot after a revision conflict") + raise AssertionError("bounded scoped-clear retry did not return or raise") + current = self.get_memory_data(agent_name, user_id=user_id) + cleared_memory = copy.deepcopy(current) + cleared_memory["facts"] = [] + if not self._save_memory_to_file(cleared_memory, agent_name, user_id=user_id, expected_revision=int(current.get("revision") or 0)): + raise OSError("Failed to save cleared memory data") + return cleared_memory + + def clear_all_memory_data(self, *, user_id: str | None = None) -> dict[str, Any]: + """Clear global summaries and every agent fact bucket for one user.""" + if getattr(type(self._storage), "clear_all", None) is not MemoryStorage.clear_all: + return self._storage.clear_all(user_id=user_id) + current = self.get_memory_data(user_id=user_id) cleared_memory = create_empty_memory() - if not self._save_memory_to_file(cleared_memory, agent_name, user_id=user_id): + if not self._save_memory_to_file( + cleared_memory, + user_id=user_id, + expected_revision=int(current.get("revision") or 0), + ): raise OSError("Failed to save cleared memory data") return cleared_memory @@ -671,28 +758,54 @@ class MemoryUpdater: existence check (upstream's ``create_memory_fact_with_created_fact``), which the vendored copy had dropped together to avoid the dangling id. """ + if agent_name is None: + raise ValueError("agent_name") normalized_content = content.strip() if not normalized_content: raise ValueError("content") normalized_category = category.strip() or "context" validated_confidence = _validate_confidence(confidence) now = utc_now_iso_z() + fact_id = f"fact_{uuid.uuid4().hex[:8]}" + candidate = { + "id": fact_id, + "content": normalized_content, + "category": normalized_category, + "confidence": validated_confidence, + "createdAt": now, + "source": "manual", + } + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + for attempt in range(3): + memory_data = self.get_memory_data(agent_name, user_id=user_id) if attempt == 0 else self.reload_memory_data(agent_name, user_id=user_id) + updated_memory = dict(memory_data) + updated_memory["facts"] = _trim_facts_to_max([*memory_data.get("facts", []), copy.deepcopy(candidate)], self._config.max_facts) + kept_ids = {str(fact.get("id")) for fact in updated_memory["facts"]} + deletions = [str(fact.get("id")) for fact in memory_data.get("facts", []) if str(fact.get("id")) not in kept_ids] + try: + self._storage.apply_changes( + { + "upserts": [fact for fact in updated_memory["facts"] if fact.get("id") == fact_id], + "upsertRevisions": {fact_id: None}, + "deletes": deletions, + "deleteRevisions": {str(fact.get("id")): int(fact.get("revision") or 1) for fact in memory_data.get("facts", []) if str(fact.get("id")) in deletions}, + }, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(memory_data.get("revision") or 0), + ) + fresh_memory = self.reload_memory_data(agent_name, user_id=user_id) + stored = any(fact.get("id") == fact_id for fact in fresh_memory.get("facts", [])) + return fresh_memory, (fact_id if stored else None) + except MemoryManifestRevisionConflict: + if attempt == 2: + raise + logger.info("Retrying capped fact creation from a fresh snapshot after a revision conflict") + raise AssertionError("bounded create retry did not return or raise") memory_data = self.get_memory_data(agent_name, user_id=user_id) updated_memory = dict(memory_data) - facts = list(memory_data.get("facts", [])) - fact_id = f"fact_{uuid.uuid4().hex[:8]}" - facts.append( - { - "id": fact_id, - "content": normalized_content, - "category": normalized_category, - "confidence": validated_confidence, - "createdAt": now, - "source": "manual", - } - ) - updated_memory["facts"] = _trim_facts_to_max(facts, self._config.max_facts) - if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id): + updated_memory["facts"] = _trim_facts_to_max([*memory_data.get("facts", []), candidate], self._config.max_facts) + if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id, expected_revision=int(memory_data.get("revision") or 0)): raise OSError("Failed to save memory data after creating fact") # If the cap evicted the just-added (lower-confidence) fact, signal via # None so callers don't report a dangling id as "added". @@ -701,19 +814,68 @@ class MemoryUpdater: def delete_memory_fact(self, fact_id: str, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: """Delete a fact by its id and persist the updated memory data.""" + if agent_name is None: + raise ValueError("agent_name") + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes and hasattr(self._storage, "get_fact"): + deleted = self._storage.get_fact(fact_id, agent_name=agent_name, user_id=user_id) + if deleted is None: + raise KeyError(fact_id) + global_memory = self.get_memory_data(user_id=user_id) + self._storage.apply_changes( + {"deletes": [fact_id], "deleteRevisions": {fact_id: int(deleted.get("revision") or 1)}}, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(global_memory.get("revision") or 0), + allow_manifest_rebase=True, + ) + return self.get_memory_data(agent_name, user_id=user_id) memory_data = self.get_memory_data(agent_name, user_id=user_id) facts = memory_data.get("facts", []) updated_facts = [fact for fact in facts if fact.get("id") != fact_id] if len(updated_facts) == len(facts): raise KeyError(fact_id) + deleted = next(fact for fact in facts if fact.get("id") == fact_id) + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + self._storage.apply_changes( + {"deletes": [fact_id], "deleteRevisions": {fact_id: int(deleted.get("revision") or 1)}}, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(memory_data.get("revision") or 0), + allow_manifest_rebase=True, + ) + return self.get_memory_data(agent_name, user_id=user_id) updated_memory = dict(memory_data) updated_memory["facts"] = updated_facts - if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id): + if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id, expected_revision=int(memory_data.get("revision") or 0)): raise OSError(f"Failed to save memory data after deleting fact '{fact_id}'") return updated_memory def update_memory_fact(self, fact_id: str, content: str | None = None, category: str | None = None, confidence: float | None = None, agent_name: str | None = None, *, user_id: str | None = None) -> dict[str, Any]: """Update an existing fact and persist the updated memory data.""" + if agent_name is None: + raise ValueError("agent_name") + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes and hasattr(self._storage, "get_fact"): + updated_fact = self._storage.get_fact(fact_id, agent_name=agent_name, user_id=user_id) + if updated_fact is None: + raise KeyError(fact_id) + if content is not None: + normalized_content = content.strip() + if not normalized_content: + raise ValueError("content") + updated_fact["content"] = normalized_content + if category is not None: + updated_fact["category"] = category.strip() or "context" + if confidence is not None: + updated_fact["confidence"] = _validate_confidence(confidence) + global_memory = self.get_memory_data(user_id=user_id) + self._storage.apply_changes( + {"upserts": [updated_fact], "upsertRevisions": {fact_id: int(updated_fact.get("revision") or 1)}}, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(global_memory.get("revision") or 0), + allow_manifest_rebase=True, + ) + return self.get_memory_data(agent_name, user_id=user_id) memory_data = self.get_memory_data(agent_name, user_id=user_id) updated_memory = dict(memory_data) updated_facts: list[dict[str, Any]] = [] @@ -736,8 +898,18 @@ class MemoryUpdater: updated_facts.append(fact) if not found: raise KeyError(fact_id) + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + changed = next(fact for fact in updated_facts if fact.get("id") == fact_id) + self._storage.apply_changes( + {"upserts": [changed], "upsertRevisions": {fact_id: int(changed.get("revision") or 1)}}, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(memory_data.get("revision") or 0), + allow_manifest_rebase=True, + ) + return self.get_memory_data(agent_name, user_id=user_id) updated_memory["facts"] = updated_facts - if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id): + if not self._save_memory_to_file(updated_memory, agent_name, user_id=user_id, expected_revision=int(memory_data.get("revision") or 0)): raise OSError(f"Failed to save memory data after updating fact '{fact_id}'") return updated_memory @@ -829,11 +1001,53 @@ class MemoryUpdater: ) -> bool: """Parse the model response, apply updates, and persist memory.""" update_data = _parse_memory_update_response(response_content) + if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes: + for attempt in range(3): + # Deep-copy before in-place mutation so a failed commit cannot + # corrupt the cached snapshot. On a manifest conflict the + # complete extraction result is reapplied to a fresh document; + # its trim/consolidation/delete decisions are snapshot-wide and + # must never be replayed as disjoint point writes. + updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id) + updated_memory = _strip_upload_mentions_from_memory(updated_memory) + current_by_id = {str(fact.get("id")): fact for fact in current_memory.get("facts", [])} + updated_by_id = {str(fact.get("id")): fact for fact in updated_memory.get("facts", [])} + change_set = { + "upserts": [copy.deepcopy(fact) for fact_id, fact in updated_by_id.items() if current_by_id.get(fact_id) != fact], + "upsertRevisions": {fact_id: (int(current_by_id[fact_id].get("revision") or 1) if fact_id in current_by_id else None) for fact_id, fact in updated_by_id.items() if current_by_id.get(fact_id) != fact}, + "deletes": [fact_id for fact_id in current_by_id if fact_id not in updated_by_id], + "deleteRevisions": {fact_id: int(current_by_id[fact_id].get("revision") or 1) for fact_id in current_by_id if fact_id not in updated_by_id}, + } + summaries_changed = updated_memory.get("user", {}) != current_memory.get("user", {}) or updated_memory.get("history", {}) != current_memory.get("history", {}) + if summaries_changed: + change_set["summaries"] = { + "user": copy.deepcopy(updated_memory.get("user", {})), + "history": copy.deepcopy(updated_memory.get("history", {})), + } + try: + self._storage.apply_changes( + change_set, + agent_name=agent_name, + user_id=user_id, + expected_manifest_revision=int(current_memory.get("revision") or 0), + ) + return True + except MemoryManifestRevisionConflict: + if attempt == 2: + raise + current_memory = self.reload_memory_data(agent_name, user_id=user_id) + logger.info("Retrying extracted memory update from a fresh snapshot after a revision conflict") + raise AssertionError("bounded extracted-update retry did not return or raise") # Deep-copy before in-place mutation so a subsequent save() failure # cannot corrupt the still-cached original object reference. updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id) updated_memory = _strip_upload_mentions_from_memory(updated_memory) - return self._storage.save(updated_memory, agent_name, user_id=user_id) + return self._storage.save( + updated_memory, + agent_name, + user_id=user_id, + expected_revision=int(current_memory.get("revision") or 0), + ) async def aupdate_memory( self, diff --git a/backend/packages/harness/deerflow/agents/memory/manager.py b/backend/packages/harness/deerflow/agents/memory/manager.py index 85e8036a9..02c65a797 100644 --- a/backend/packages/harness/deerflow/agents/memory/manager.py +++ b/backend/packages/harness/deerflow/agents/memory/manager.py @@ -41,6 +41,18 @@ _backends_cache: dict[str, type[MemoryManager]] | None = None _manager_lock = threading.Lock() +class MemoryManagerError(RuntimeError): + """Backend-neutral base error exposed at the MemoryManager boundary.""" + + +class MemoryConflictError(MemoryManagerError): + """The requested write lost an optimistic-concurrency race.""" + + +class MemoryCorruptionError(MemoryManagerError): + """Persisted memory cannot be read safely.""" + + class MemoryManager(ABC): """Backend-neutral memory manager contract (9 methods). @@ -167,7 +179,12 @@ class MemoryManager(ABC): user_id: str | None = None, agent_name: str | None = None, ) -> dict[str, Any]: - """Clear the bucket's memory; return the cleared (now-empty) document.""" + """Clear memory and return the empty compatibility document. + + ``agent_name=None`` means all memory owned by the user. An explicit + agent name clears only that agent's memory and must preserve shared + user-level summaries. + """ @abstractmethod def import_memory( diff --git a/backend/scripts/migrate_memory_markdown.py b/backend/scripts/migrate_memory_markdown.py new file mode 100644 index 000000000..b7099b865 --- /dev/null +++ b/backend/scripts/migrate_memory_markdown.py @@ -0,0 +1,137 @@ +"""Proactively migrate legacy global JSON facts into Markdown fact files. + +Normal DeerMem reads already perform this migration lazily. This CLI lets an +operator preview or complete the same idempotent migration before serving +traffic, which is useful for multi-user upgrade audits. + +Usage from ``backend/``:: + + PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run + PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users + PYTHONPATH=. python scripts/migrate_memory_markdown.py --user-id alice +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + +from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig +from deerflow.agents.memory.backends.deermem.deermem.core.paths import DEFAULT_AGENT_BUCKET, memory_file_path +from deerflow.agents.memory.backends.deermem.deermem.core.storage import DOCUMENT_VERSION, FileMemoryStorage +from deerflow.config.runtime_paths import runtime_home + + +def discover_user_ids(storage_path: Path) -> list[str]: + """Return directory-safe user IDs found below one DeerMem storage root.""" + users_root = storage_path / "users" + if not users_root.is_dir(): + return [] + return sorted(entry.name for entry in users_root.iterdir() if entry.is_dir()) + + +def _inspect_legacy_global_json(config: DeerMemConfig, user_id: str) -> tuple[Path, bool]: + path = memory_file_path(config, user_id=user_id) + if not path.exists(): + return path, False + try: + document = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise ValueError(f"cannot parse {path}: {exc}") from exc + if not isinstance(document, dict): + raise ValueError(f"{path} must contain a JSON object") + return path, "facts" in document or document.get("version") != DOCUMENT_VERSION + + +def migrate_users( + config: DeerMemConfig, + user_ids: list[str], + *, + dry_run: bool = False, +) -> list[dict[str, Any]]: + """Migrate selected users independently and return an audit-friendly report.""" + storage = FileMemoryStorage(config) + report: list[dict[str, Any]] = [] + for user_id in user_ids: + entry: dict[str, Any] = {"user_id": user_id, "status": "current", "path": "", "error": None} + try: + path, needs_migration = _inspect_legacy_global_json(config, user_id) + entry["path"] = str(path) + if not needs_migration: + report.append(entry) + continue + if dry_run: + entry["status"] = "planned" + else: + result = storage.migrate(user_id=user_id, agent_name=DEFAULT_AGENT_BUCKET) + entry["status"] = "migrated" if result.get("migrated") else "current" + entry["from_version"] = result.get("fromVersion") + entry["to_version"] = result.get("toVersion") + except Exception as exc: # noqa: BLE001 - one bad user must not hide the rest of the audit + entry["status"] = "failed" + entry["error"] = str(exc) + report.append(entry) + return report + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=("Proactively migrate legacy facts from each user's memory.json into the reserved __default__ Markdown fact bucket.")) + selection = parser.add_mutually_exclusive_group(required=True) + selection.add_argument( + "--all-users", + action="store_true", + help="Migrate every directory-safe user bucket found under STORAGE_PATH/users.", + ) + selection.add_argument( + "--user-id", + action="append", + dest="user_ids", + metavar="USER_ID", + help="Migrate one original user ID; repeat this option for multiple users.", + ) + parser.add_argument( + "--storage-path", + type=Path, + default=None, + help="DeerMem root directory; defaults to DeerFlow's runtime home.", + ) + parser.add_argument("--dry-run", action="store_true", help="Report pending migrations without changing files.") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + storage_path = (args.storage_path or runtime_home()).resolve() + config = DeerMemConfig(storage_path=str(storage_path)) + user_ids = discover_user_ids(storage_path) if args.all_users else list(dict.fromkeys(args.user_ids or [])) + + print(f"Storage root: {storage_path}") + if not user_ids: + print("No user buckets found; nothing to migrate.") + return 0 + + report = migrate_users(config, user_ids, dry_run=args.dry_run) + for entry in report: + status = entry["status"] + user_id = entry["user_id"] + if status == "planned": + print(f"{user_id}: would migrate {entry['path']}") + elif status == "migrated": + print(f"{user_id}: migrated {entry['path']} ({entry.get('from_version')} -> {entry.get('to_version')})") + elif status == "failed": + print(f"{user_id}: FAILED: {entry['error']}") + else: + print(f"{user_id}: already current") + + migrated = sum(entry["status"] == "migrated" for entry in report) + planned = sum(entry["status"] == "planned" for entry in report) + current = sum(entry["status"] == "current" for entry in report) + failed = sum(entry["status"] == "failed" for entry in report) + print(f"Summary: migrated={migrated} planned={planned} current={current} failed={failed}") + return 1 if failed else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/tests/test_custom_agent.py b/backend/tests/test_custom_agent.py index e1c5fcd74..371a0ed8f 100644 --- a/backend/tests/test_custom_agent.py +++ b/backend/tests/test_custom_agent.py @@ -9,6 +9,9 @@ import pytest import yaml from fastapi.testclient import TestClient +from app.gateway.routers.agents import AGENT_NAME_PATTERN as GATEWAY_AGENT_NAME_PATTERN +from deerflow.agents.memory.backends.deermem.deermem.core.paths import AGENT_NAME_PATTERN as DEERMEM_AGENT_NAME_PATTERN +from deerflow.agents.memory.backends.deermem.deermem.core.paths import DEFAULT_AGENT_BUCKET, validate_agent_name from deerflow.config.agents_api_config import AgentsApiConfig, get_agents_api_config, set_agents_api_config # --------------------------------------------------------------------------- @@ -16,6 +19,12 @@ from deerflow.config.agents_api_config import AgentsApiConfig, get_agents_api_co # --------------------------------------------------------------------------- +def test_reserved_memory_bucket_stays_outside_both_public_agent_patterns() -> None: + assert GATEWAY_AGENT_NAME_PATTERN.fullmatch(DEFAULT_AGENT_BUCKET) is None + assert DEERMEM_AGENT_NAME_PATTERN.fullmatch(DEFAULT_AGENT_BUCKET) is None + validate_agent_name(DEFAULT_AGENT_BUCKET) # Internal storage sentinel remains usable. + + def _make_paths(base_dir: Path): """Return a Paths instance pointing to base_dir.""" from deerflow.config.paths import Paths @@ -477,16 +486,16 @@ class TestMemoryFilePath: assert path == tmp_path / "memory.json" def test_agent_memory_path(self, tmp_path, monkeypatch): - """Providing agent_name should return per-agent memory file.""" + """All agents share the user-global summary JSON path.""" from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig from deerflow.agents.memory.backends.deermem.deermem.core.storage import FileMemoryStorage monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) storage = FileMemoryStorage(DeerMemConfig()) path = storage._get_memory_file_path("code-reviewer") - assert path == tmp_path / "agents" / "code-reviewer" / "memory.json" + assert path == tmp_path / "memory.json" - def test_different_paths_for_different_agents(self, tmp_path, monkeypatch): + def test_agents_share_summary_path(self, tmp_path, monkeypatch): from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig from deerflow.agents.memory.backends.deermem.deermem.core.storage import FileMemoryStorage @@ -496,9 +505,7 @@ class TestMemoryFilePath: path_a = storage._get_memory_file_path("agent-a") path_b = storage._get_memory_file_path("agent-b") - assert path_global != path_a - assert path_global != path_b - assert path_a != path_b + assert path_global == path_a == path_b # =========================================================================== @@ -730,6 +737,22 @@ class TestAgentsAPI: response = agent_client.delete("/api/agents/does-not-exist") assert response.status_code == 404 + def test_delete_rejects_memory_only_directory_without_removing_facts(self, agent_client, tmp_path): + agent_dir = tmp_path / "users" / "test-user-autouse" / "agents" / "lead-agent" + facts_dir = agent_dir / "facts" + facts_dir.mkdir(parents=True) + fact_path = facts_dir / "fact_keep.md" + fact_path.write_text("memory data", encoding="utf-8") + + response = agent_client.delete("/api/agents/lead-agent") + + assert response.status_code == 409 + assert fact_path.read_text(encoding="utf-8") == "memory data" + + def test_reserved_default_bucket_cannot_be_created_as_custom_agent(self, agent_client): + response = agent_client.post("/api/agents", json={"name": "__default__", "soul": "must fail"}) + assert response.status_code == 422 + def test_create_agent_with_model_and_tool_groups(self, agent_client): payload = { "name": "specialized", diff --git a/backend/tests/test_deermem_self_contained.py b/backend/tests/test_deermem_self_contained.py index f5da58e25..9f3b2aca5 100644 --- a/backend/tests/test_deermem_self_contained.py +++ b/backend/tests/test_deermem_self_contained.py @@ -74,6 +74,52 @@ def test_zero_config_defaults_run_non_llm_ops(deermem_data_dir): assert dm.get_memory(user_id="u")["facts"][0]["content"] == "x" +def test_import_without_agent_name_persists_facts_in_default_markdown_bucket(deermem_data_dir): + dm = DeerMem(backend_config=None) + dm.import_memory( + { + "user": {}, + "history": {}, + "facts": [ + { + "id": "fact_default_import", + "content": "imported through the default manager scope", + "category": "context", + "confidence": 0.8, + "source": "import", + } + ], + }, + user_id="alice", + ) + + assert [fact["id"] for fact in dm.get_memory(user_id="alice")["facts"]] == ["fact_default_import"] + facts_root = deermem_data_dir / "users" / "alice" / "agents" / "__default__" / "facts" + assert [path.stem for path in facts_root.glob("**/*.md")] == ["fact_default_import"] + + +def test_import_empty_summary_sections_replace_existing_summaries_with_complete_defaults(deermem_data_dir): + dm = DeerMem(backend_config=None) + existing = dm.get_memory(user_id="alice") + existing["user"]["workContext"] = {"summary": "old work", "updatedAt": "old"} + existing["user"]["personalContext"] = {"summary": "old personal", "updatedAt": "old"} + existing["history"]["recentMonths"] = {"summary": "old history", "updatedAt": "old"} + dm.import_memory(existing, user_id="alice") + + imported = dm.import_memory({"user": {}, "history": {}, "facts": []}, user_id="alice") + + assert imported["user"] == { + "workContext": {"summary": "", "updatedAt": ""}, + "personalContext": {"summary": "", "updatedAt": ""}, + "topOfMind": {"summary": "", "updatedAt": ""}, + } + assert imported["history"] == { + "recentMonths": {"summary": "", "updatedAt": ""}, + "earlierContext": {"summary": "", "updatedAt": ""}, + "longTermBackground": {"summary": "", "updatedAt": ""}, + } + + def test_trace_id_threads_through_to_tracing_callback(deermem_data_dir): calls = [] @@ -92,6 +138,63 @@ def test_trace_id_threads_through_to_tracing_callback(deermem_data_dir): assert calls and calls[0] == ("t1", "trace-42", "gpt-x") +def test_default_passive_update_persists_fact_in_reserved_default_bucket(deermem_data_dir): + dm = _deermem_with_fake_llm(payload='{"user":{},"history":{},"newFacts":[{"content":"Default agent fact","category":"context","confidence":0.9}],"factsToRemove":[]}') + + dm.add( + thread_id="default-thread", + messages=[HumanMessage(content="remember this"), AIMessage(content="understood")], + user_id="alice", + ) + dm._queue.flush() + + assert [fact["content"] for fact in dm.get_memory(user_id="alice")["facts"]] == ["Default agent fact"] + facts_root = deermem_data_dir / "users" / "alice" / "agents" / "__default__" / "facts" + assert list(facts_root.glob("**/*.md")) + + +def test_clear_all_memory_removes_global_summaries_and_every_agent_fact(deermem_data_dir): + dm = DeerMem() + imported = dm.get_memory(user_id="alice") + imported["user"]["workContext"] = {"summary": "shared profile", "updatedAt": "now"} + imported["facts"] = [ + { + "id": "fact_default", + "content": "default fact", + "category": "context", + "confidence": 0.9, + "createdAt": "2026-01-01T00:00:00Z", + "source": "manual", + } + ] + dm.import_memory(imported, user_id="alice") + dm.create_fact("custom fact", agent_name="custom-agent", user_id="alice") + custom_dir = deermem_data_dir / "users" / "alice" / "agents" / "custom-agent" + config_path = custom_dir / "config.yaml" + config_path.write_text("name: custom-agent\n", encoding="utf-8") + + cleared = dm.clear_memory(user_id="alice") + + assert cleared["user"]["workContext"]["summary"] == "" + assert cleared["facts"] == [] + assert dm.get_memory(agent_name="custom-agent", user_id="alice")["facts"] == [] + assert config_path.read_text(encoding="utf-8") == "name: custom-agent\n" + + +def test_scoped_clear_preserves_shared_summaries(deermem_data_dir): + dm = DeerMem() + imported = dm.get_memory(user_id="alice") + imported["user"]["workContext"] = {"summary": "shared profile", "updatedAt": "now"} + dm.import_memory(imported, user_id="alice") + dm.create_fact("custom fact", agent_name="custom-agent", user_id="alice") + + cleared = dm.clear_memory(agent_name="custom-agent", user_id="alice") + + assert cleared["facts"] == [] + assert cleared["user"]["workContext"]["summary"] == "shared profile" + assert dm.get_memory(user_id="alice")["user"]["workContext"]["summary"] == "shared profile" + + def test_tracing_callback_optional_no_langfuse(deermem_data_dir): dm = _deermem_with_fake_llm({"model": {"provider": "openai", "model": "gpt-x", "api_key": "k", "base_url": "u"}}) assert dm._config.tracing_callback is None # langfuse not hard-required @@ -135,7 +238,7 @@ def test_portability_only_abc_contract_imports_deerflow(): deerflow_imports.append((p.relative_to(root).as_posix(), s)) assert len(deerflow_imports) == 1, deerflow_imports assert deerflow_imports[0][0] == "deer_mem.py" - assert "memory.manager import MemoryManager" in deerflow_imports[0][1] + assert "memory.manager import MemoryConflictError, MemoryCorruptionError, MemoryManager" in deerflow_imports[0][1] # Minimal vendored host contract (what another agent would ship). DeerMem only @@ -166,6 +269,9 @@ class MemoryManager(ABC): def import_memory(self, memory_data, *, user_id=None, agent_name=None) -> dict: ... @abstractmethod def export_memory(self, *, user_id=None, agent_name=None) -> dict: ... + +class MemoryConflictError(RuntimeError): ... +class MemoryCorruptionError(RuntimeError): ... ''' @@ -190,10 +296,11 @@ def test_portability_vendor_to_other_agent(tmp_path, monkeypatch): # Repoint the single ABC-contract import line to the vendored manager. deer_mem_file = dst_pkg / "deer_mem.py" text = deer_mem_file.read_text(encoding="utf-8") - assert "from deerflow.agents.memory.manager import MemoryManager" in text + contract_import = "from deerflow.agents.memory.manager import MemoryConflictError, MemoryCorruptionError, MemoryManager" + assert contract_import in text text = text.replace( - "from deerflow.agents.memory.manager import MemoryManager", - "from otheragent.manager import MemoryManager", + contract_import, + "from otheragent.manager import MemoryConflictError, MemoryCorruptionError, MemoryManager", ) deer_mem_file.write_text(text, encoding="utf-8") @@ -229,7 +336,7 @@ def test_per_user_memory_path_matches_host_safe_user_id(deermem_data_dir): user_id = "test-user-123@example.com" # storage_path mirrors what the host factory injects (runtime_home / base_dir) dm = DeerMem(backend_config={"storage_path": str(deermem_data_dir)}) - dm.create_fact("User prefers concise answers", category="preference", user_id=user_id) + dm.create_fact("User prefers concise answers", category="preference", agent_name="default", user_id=user_id) expected_safe = make_safe_user_id(user_id) expected_file = deermem_data_dir / "users" / expected_safe / "memory.json" diff --git a/backend/tests/test_memory_manager_pluggable.py b/backend/tests/test_memory_manager_pluggable.py index 8431536e1..2025b0e74 100644 --- a/backend/tests/test_memory_manager_pluggable.py +++ b/backend/tests/test_memory_manager_pluggable.py @@ -143,7 +143,7 @@ def test_empty_storage_path_factory_injects_runtime_home(tmp_path, monkeypatch) set_memory_config(MemoryConfig(manager_class="deermem")) # no storage_path manager = get_memory_manager() assert Path(manager._config.storage_path) == tmp_path - manager.create_fact("hello", user_id="u1") + manager.create_fact("hello", user_id="u1", agent_name="test-agent") # per-user dir created under the injected runtime_home root user_dirs = [p.name for p in (tmp_path / "users").iterdir() if p.is_dir()] assert len(user_dirs) == 1 diff --git a/backend/tests/test_memory_markdown_migration_cli.py b/backend/tests/test_memory_markdown_migration_cli.py new file mode 100644 index 000000000..ff1a4fe97 --- /dev/null +++ b/backend/tests/test_memory_markdown_migration_cli.py @@ -0,0 +1,99 @@ +"""Tests for the proactive JSON-to-Markdown memory migration CLI.""" + +import json +from pathlib import Path + +from deerflow.agents.memory.backends.deermem.deermem.core.paths import fact_file_path + + +def _legacy_memory(content: str) -> dict: + return { + "version": "1.0", + "revision": 0, + "lastUpdated": "2026-01-01T00:00:00Z", + "user": {"workContext": {"summary": "keep me", "updatedAt": "2026-01-01T00:00:00Z"}}, + "history": {}, + "facts": [ + { + "id": "fact_legacy", + "content": content, + "category": "context", + "confidence": 0.9, + "createdAt": "2026-01-01T00:00:00Z", + "source": "manual", + } + ], + } + + +def _seed_user(root: Path, user_bucket: str, content: str) -> Path: + path = root / "users" / user_bucket / "memory.json" + path.parent.mkdir(parents=True) + path.write_text(json.dumps(_legacy_memory(content)), encoding="utf-8") + return path + + +def test_cli_migrates_one_explicit_user(tmp_path: Path) -> None: + from scripts.migrate_memory_markdown import main + + memory_path = _seed_user(tmp_path, "alice", "legacy alice fact") + + exit_code = main(["--storage-path", str(tmp_path), "--user-id", "alice"]) + + assert exit_code == 0 + persisted = json.loads(memory_path.read_text(encoding="utf-8")) + assert "facts" not in persisted + assert persisted["user"]["workContext"]["summary"] == "keep me" + assert fact_file_path(memory_path, "fact_legacy", agent_name="__default__").exists() + + +def test_cli_dry_run_reports_without_writing(tmp_path: Path, capsys) -> None: + from scripts.migrate_memory_markdown import main + + memory_path = _seed_user(tmp_path, "alice", "legacy alice fact") + original = memory_path.read_bytes() + + exit_code = main(["--storage-path", str(tmp_path), "--user-id", "alice", "--dry-run"]) + + assert exit_code == 0 + assert memory_path.read_bytes() == original + assert not (memory_path.parent / "agents" / "__default__" / "facts").exists() + assert "would migrate" in capsys.readouterr().out + + +def test_cli_all_users_migrates_each_discovered_bucket(tmp_path: Path) -> None: + from scripts.migrate_memory_markdown import main + + alice_path = _seed_user(tmp_path, "alice", "legacy alice fact") + bob_path = _seed_user(tmp_path, "bob", "legacy bob fact") + + exit_code = main(["--storage-path", str(tmp_path), "--all-users"]) + + assert exit_code == 0 + for memory_path in (alice_path, bob_path): + assert "facts" not in json.loads(memory_path.read_text(encoding="utf-8")) + assert fact_file_path(memory_path, "fact_legacy", agent_name="__default__").exists() + + +def test_cli_is_idempotent_and_reports_current_user(tmp_path: Path, capsys) -> None: + from scripts.migrate_memory_markdown import main + + _seed_user(tmp_path, "alice", "legacy alice fact") + arguments = ["--storage-path", str(tmp_path), "--user-id", "alice"] + + assert main(arguments) == 0 + capsys.readouterr() + assert main(arguments) == 0 + + assert "already current" in capsys.readouterr().out + + +def test_cli_requires_user_selection(tmp_path: Path) -> None: + from scripts.migrate_memory_markdown import main + + try: + main(["--storage-path", str(tmp_path)]) + except SystemExit as exc: + assert exc.code == 2 + else: + raise AssertionError("CLI must require --all-users or --user-id") diff --git a/backend/tests/test_memory_router.py b/backend/tests/test_memory_router.py index 168efad0c..f9c548e49 100644 --- a/backend/tests/test_memory_router.py +++ b/backend/tests/test_memory_router.py @@ -1,4 +1,5 @@ import asyncio +import json from types import SimpleNamespace from unittest.mock import MagicMock, patch @@ -6,6 +7,8 @@ from fastapi import FastAPI from fastapi.testclient import TestClient from app.gateway.routers import memory +from deerflow.agents.memory import MemoryConflictError, MemoryCorruptionError +from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem def _sample_memory(facts: list[dict] | None = None) -> dict: @@ -86,6 +89,36 @@ def test_import_memory_route_returns_imported_memory() -> None: assert response.json()["facts"] == imported_memory["facts"] +def test_import_route_without_agent_name_persists_default_bucket_markdown(tmp_path) -> None: + app = FastAPI() + app.include_router(memory.router) + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + imported_memory = _sample_memory( + facts=[ + { + "id": "fact_gateway_import", + "content": "Gateway imports use the default agent bucket.", + "category": "context", + "confidence": 0.9, + "createdAt": "2026-07-21T00:00:00Z", + "source": "import", + } + ] + ) + + with ( + patch("app.gateway.routers.memory.get_memory_manager", return_value=manager), + patch("app.gateway.routers.memory.get_effective_user_id", return_value="alice"), + TestClient(app) as client, + ): + response = client.post("/api/memory/import", json=imported_memory) + + assert response.status_code == 200 + assert [fact["id"] for fact in response.json()["facts"]] == ["fact_gateway_import"] + facts_root = tmp_path / "users" / "alice" / "agents" / "__default__" / "facts" + assert [path.stem for path in facts_root.glob("**/*.md")] == ["fact_gateway_import"] + + def test_import_memory_route_preserves_source_error() -> None: app = FastAPI() app.include_router(memory.router) @@ -144,6 +177,34 @@ def test_create_memory_fact_route_returns_updated_memory() -> None: assert response.json()["facts"] == updated_memory["facts"] +def test_create_memory_fact_route_maps_conflict_to_409() -> None: + app = FastAPI() + app.include_router(memory.router) + mock_mgr = MagicMock() + mock_mgr.create_fact.side_effect = MemoryConflictError("stale write") + + with patch("app.gateway.routers.memory.get_memory_manager", return_value=mock_mgr): + with TestClient(app) as client: + response = client.post("/api/memory/facts", json={"content": "fact"}) + + assert response.status_code == 409 + assert response.json()["detail"] == "Memory changed concurrently; reload and retry." + + +def test_get_memory_route_maps_corruption_to_stable_500() -> None: + app = FastAPI() + app.include_router(memory.router) + mock_mgr = MagicMock() + mock_mgr.get_memory.side_effect = MemoryCorruptionError("private path and parser detail") + + with patch("app.gateway.routers.memory.get_memory_manager", return_value=mock_mgr): + with TestClient(app) as client: + response = client.get("/api/memory") + + assert response.status_code == 500 + assert response.json()["detail"] == "Stored memory data is corrupted." + + def test_delete_memory_fact_route_returns_updated_memory() -> None: app = FastAPI() app.include_router(memory.router) @@ -184,6 +245,63 @@ def test_update_memory_fact_route_returns_updated_memory() -> None: assert response.json()["facts"] == updated_memory["facts"] +def test_settings_fact_crud_without_agent_name_uses_default_agent(tmp_path) -> None: + """The current Settings API sends no agent_name; it must remain usable.""" + app = FastAPI() + app.include_router(memory.router) + memory_path = tmp_path / "users" / "alice" / "memory.json" + memory_path.parent.mkdir(parents=True) + legacy = _sample_memory( + facts=[ + { + "id": "fact_legacy", + "content": "Legacy global fact", + "category": "context", + "confidence": 0.9, + "createdAt": "2026-03-20T00:00:00Z", + "source": "manual", + } + ] + ) + memory_path.write_text(json.dumps(legacy), encoding="utf-8") + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + + with ( + patch("app.gateway.routers.memory.get_memory_manager", return_value=manager), + patch("app.gateway.routers.memory.get_effective_user_id", return_value="alice"), + TestClient(app) as client, + ): + fetched = client.get("/api/memory") + assert fetched.status_code == 200 + assert [fact["content"] for fact in fetched.json()["facts"]] == ["Legacy global fact"] + + exported = client.get("/api/memory/export") + assert exported.status_code == 200 + assert [fact["content"] for fact in exported.json()["facts"]] == ["Legacy global fact"] + + created = client.post("/api/memory/facts", json={"content": "Project uses Python", "category": "context", "confidence": 0.8}) + assert created.status_code == 200 + assert all(isinstance(fact["source"], str) for fact in created.json()["facts"]) + fact_id = next(fact["id"] for fact in created.json()["facts"] if fact["content"] == "Project uses Python") + + updated = client.patch(f"/api/memory/facts/{fact_id}", json={"content": "Project uses Python 3.12"}) + assert updated.status_code == 200 + assert updated.json()["facts"][0]["content"] == "Project uses Python 3.12" + + deleted = client.delete(f"/api/memory/facts/{fact_id}") + assert deleted.status_code == 200 + assert [fact["id"] for fact in deleted.json()["facts"]] == ["fact_legacy"] + + deleted_legacy = client.delete("/api/memory/facts/fact_legacy") + assert deleted_legacy.status_code == 200 + assert deleted_legacy.json()["facts"] == [] + + facts_root = tmp_path / "users" / "alice" / "agents" / "__default__" / "facts" + assert facts_root.exists() + assert not list(facts_root.glob("**/*.md")) + assert "facts" not in json.loads(memory_path.read_text(encoding="utf-8")) + + def test_update_memory_fact_route_preserves_omitted_fields() -> None: app = FastAPI() app.include_router(memory.router) diff --git a/backend/tests/test_memory_storage.py b/backend/tests/test_memory_storage.py index 009ca30a2..f7a4a83fa 100644 --- a/backend/tests/test_memory_storage.py +++ b/backend/tests/test_memory_storage.py @@ -55,11 +55,11 @@ class TestFileMemoryStorage: assert storage._get_memory_file_path(None) == tmp_path / "memory.json" def test_get_memory_file_path_agent(self, tmp_path, monkeypatch): - """Legacy per-agent path lives under the DeerMem root ($DEERMEM_DATA_DIR).""" + """Agent facts share the user's global summary JSON path.""" monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) storage = FileMemoryStorage(DeerMemConfig()) path = storage._get_memory_file_path("test-agent") - assert path == tmp_path / "agents" / "test-agent" / "memory.json" + assert path == tmp_path / "memory.json" @pytest.mark.parametrize("invalid_name", ["", "../etc/passwd", "agent/name", "agent\\name", "agent name", "agent@123", "agent_name"]) def test_validate_agent_name_invalid(self, invalid_name): @@ -78,9 +78,10 @@ class TestFileMemoryStorage: monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) memory_file = tmp_path / "memory.json" storage = FileMemoryStorage(DeerMemConfig()) - result = storage.save({"version": "1.0", "facts": [{"content": "test fact"}]}) + result = storage.save({"version": "1.0", "facts": []}) assert result is True assert memory_file.exists() + assert "facts" not in memory_file.read_text(encoding="utf-8") def test_save_does_not_mutate_caller_dict(self, tmp_path, monkeypatch): monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) @@ -98,16 +99,18 @@ class TestFileMemoryStorage: memory_file.parent.mkdir(parents=True, exist_ok=True) import json as _json - memory_file.write_text(_json.dumps({"version": "1.0", "facts": [{"content": "original"}]})) + memory_file.write_text(_json.dumps({"version": "2.0", "user": {"workContext": {"summary": "original"}}, "history": {}})) storage = FileMemoryStorage(DeerMemConfig()) cached = storage.load() - assert cached["facts"][0]["content"] == "original" + assert cached["user"]["workContext"]["summary"] == "original" with patch("builtins.open", side_effect=OSError("disk full")): - result = storage.save({"version": "1.0", "facts": [{"content": "mutated"}]}) + changed = create_empty_memory() + changed["user"]["workContext"]["summary"] = "mutated" + result = storage.save(changed) assert result is False after = storage.load() - assert after["facts"][0]["content"] == "original" + assert after["user"]["workContext"]["summary"] == "original" def test_cache_thread_safety(self, tmp_path, monkeypatch): monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) @@ -137,13 +140,13 @@ class TestFileMemoryStorage: monkeypatch.setenv("DEERMEM_DATA_DIR", str(tmp_path)) memory_file = tmp_path / "memory.json" memory_file.parent.mkdir(parents=True, exist_ok=True) - memory_file.write_text('{"version": "1.0", "facts": [{"content": "initial fact"}]}') + memory_file.write_text('{"version": "2.0", "user": {"workContext": {"summary": "initial"}}, "history": {}}') storage = FileMemoryStorage(DeerMemConfig()) memory1 = storage.load() - assert memory1["facts"][0]["content"] == "initial fact" - memory_file.write_text('{"version": "1.0", "facts": [{"content": "updated fact"}]}') + assert memory1["user"]["workContext"]["summary"] == "initial" + memory_file.write_text('{"version": "2.0", "user": {"workContext": {"summary": "updated"}}, "history": {}}') memory2 = storage.reload() - assert memory2["facts"][0]["content"] == "updated fact" + assert memory2["user"]["workContext"]["summary"] == "updated" class TestCreateStorage: diff --git a/backend/tests/test_memory_storage_markdown.py b/backend/tests/test_memory_storage_markdown.py new file mode 100644 index 000000000..35589cfff --- /dev/null +++ b/backend/tests/test_memory_storage_markdown.py @@ -0,0 +1,1189 @@ +"""File/JSON + single-fact Markdown storage contract tests.""" + +import copy +import gc +import hashlib +import json +import os +import shutil +import weakref +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +from deerflow.agents.memory import MemoryCorruptionError +from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem +from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig +from deerflow.agents.memory.backends.deermem.deermem.core import storage as storage_module +from deerflow.agents.memory.backends.deermem.deermem.core.paths import fact_file_path +from deerflow.agents.memory.backends.deermem.deermem.core.storage import ( + FileMemoryStorage, + MemoryFactRevisionConflict, + MemoryManifestRevisionConflict, + MemoryRevisionConflict, + MemoryStorageCorruption, + create_empty_memory, +) + + +@pytest.fixture +def storage(tmp_path: Path) -> FileMemoryStorage: + return FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path))) + + +def _memory_with_fact(content: str = "Project uses Python 3.12") -> dict: + memory = create_empty_memory() + memory["facts"] = [ + { + "id": "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", + "content": content, + "category": "constraint", + "topics": ["python", "runtime"], + "confidence": 0.95, + "createdAt": "2026-07-17T00:00:00Z", + "source": {"type": "manual", "threadId": "thread-1"}, + "revision": 1, + } + ] + return memory + + +def test_agent_scope_uses_fact_directories_but_one_user_memory_file(storage: FileMemoryStorage, tmp_path: Path) -> None: + assert storage.save(_memory_with_fact("A"), "agent-a", user_id="alice") + assert storage.save(_memory_with_fact("B"), "agent-b", user_id="alice") + + assert storage.load("agent-a", user_id="alice")["facts"][0]["content"] == "A" + assert storage.load("agent-b", user_id="alice")["facts"][0]["content"] == "B" + assert (tmp_path / "users" / "alice" / "memory.json").exists() + assert not (tmp_path / "users" / "alice" / "agents" / "agent-a" / "memory.json").exists() + assert list((tmp_path / "users" / "alice" / "agents" / "agent-a" / "facts").glob("**/*.md")) + assert list((tmp_path / "users" / "alice" / "agents" / "agent-b" / "facts").glob("**/*.md")) + + +def test_thread_id_is_source_only_not_storage_bucket(storage: FileMemoryStorage) -> None: + fact = _memory_with_fact()["facts"][0] + assert fact["source"]["threadId"] == "thread-1" + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + assert "thread-1" not in str(memory_path) + + +def test_memory_json_contains_only_global_summaries_and_agent_fact_is_markdown(storage: FileMemoryStorage) -> None: + memory = _memory_with_fact() + memory["user"]["workContext"] = {"summary": "global profile", "updatedAt": "now"} + assert storage.save(memory, "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + persisted = json.loads(memory_path.read_text(encoding="utf-8")) + + assert persisted["version"] == "2.0" + assert "facts" not in persisted + assert set(persisted) == {"version", "revision", "lastUpdated", "user", "history"} + fact_path = fact_file_path(memory_path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a") + text = fact_path.read_text(encoding="utf-8") + assert text.startswith("---\n") + assert "user_id: alice" in text + assert "agent_name: agent-a" in text + assert "# Project uses Python 3.12" in text + + +def test_load_keeps_frontend_shape_but_only_agent_load_returns_facts(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + + global_memory = storage.load(user_id="alice") + agent_memory = storage.load("agent-a", user_id="alice") + + assert global_memory["facts"] == [] + assert agent_memory["facts"][0]["topics"] == ["python", "runtime"] + assert agent_memory["facts"][0]["scope"] == {"userId": "alice", "agentName": "agent-a"} + + +def test_agent_save_does_not_overwrite_global_summaries(storage: FileMemoryStorage) -> None: + global_memory = create_empty_memory() + global_memory["user"]["workContext"] = {"summary": "works remotely", "updatedAt": "global"} + assert storage.save(global_memory, user_id="alice") + + agent_memory = storage.load("agent-a", user_id="alice") + agent_memory["user"]["workContext"] = {"summary": "project secret", "updatedAt": "agent"} + agent_memory["facts"] = _memory_with_fact()["facts"] + assert storage.save(agent_memory, "agent-a", user_id="alice", expected_revision=1) + + assert storage.load(user_id="alice")["user"]["workContext"]["summary"] == "works remotely" + + +def test_removed_fact_is_physically_deleted(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + fact_path = fact_file_path(memory_path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a") + assert fact_path.exists() + + empty = create_empty_memory() + assert storage.save(empty, "agent-a", user_id="alice") + assert not fact_path.exists() + + +def test_cached_document_is_not_mutable_by_caller(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + first = storage.load("agent-a", user_id="alice") + first["facts"][0]["content"] = "mutated outside storage" + second = storage.load("agent-a", user_id="alice") + assert second["facts"][0]["content"] == "Project uses Python 3.12" + + +def test_cached_load_does_not_scan_all_fact_files(storage: FileMemoryStorage, monkeypatch: pytest.MonkeyPatch) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + storage.load("agent-a", user_id="alice") + + def fail_fact_scan(*args, **kwargs): + raise AssertionError("cached load scanned the agent fact directory") + + monkeypatch.setattr(storage_module, "agent_facts_directory", fail_fact_scan) + + assert storage.load("agent-a", user_id="alice")["facts"][0]["content"] == "Project uses Python 3.12" + + +def test_shared_json_signature_invalidates_cache_after_other_storage_writes(tmp_path: Path) -> None: + config = DeerMemConfig(storage_path=str(tmp_path)) + first = FileMemoryStorage(config) + second = FileMemoryStorage(config) + assert first.save(_memory_with_fact(), "agent-a", user_id="alice") + cached = first.load("agent-a", user_id="alice") + changed = copy.deepcopy(cached["facts"][0]) + changed["content"] = "changed by another storage instance" + + second.apply_changes( + {"upserts": [changed], "upsertRevisions": {changed["id"]: changed["revision"]}}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=cached["revision"], + allow_manifest_rebase=True, + ) + + assert first.load("agent-a", user_id="alice")["facts"][0]["content"] == "changed by another storage instance" + + +def test_revision_invalidates_cache_when_file_metadata_signature_collides(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A coarse filesystem may report the same mtime/size for two writes.""" + monkeypatch.setattr(storage_module, "_file_signature", lambda path: (1, 1)) + config = DeerMemConfig(storage_path=str(tmp_path)) + first = FileMemoryStorage(config) + second = FileMemoryStorage(config) + assert first.save(_memory_with_fact(), "agent-a", user_id="alice") + cached = first.load("agent-a", user_id="alice") + changed = copy.deepcopy(cached["facts"][0]) + changed["content"] = "same metadata, newer repository revision" + + second.apply_changes( + {"upserts": [changed], "upsertRevisions": {changed["id"]: changed["revision"]}}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=cached["revision"], + allow_manifest_rebase=True, + ) + + assert first.load("agent-a", user_id="alice")["facts"][0]["content"] == "same metadata, newer repository revision" + + +def test_fact_paths_shard_by_sha256_id_digest(tmp_path: Path) -> None: + memory_path = tmp_path / "users" / "alice" / "memory.json" + fact_ids = ["fact_00000000", "fact_11111111", "fact_22222222", "external-123"] + + paths = [fact_file_path(memory_path, fact_id, agent_name="agent-a") for fact_id in fact_ids] + + for fact_id, path in zip(fact_ids, paths, strict=True): + assert path.parent.name == hashlib.sha256(fact_id.encode("utf-8")).hexdigest()[:2] + assert len({path.parent.name for path in paths}) > 1 + + +def test_corrupt_manifest_raises_and_is_not_treated_as_empty(storage: FileMemoryStorage) -> None: + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("{broken", encoding="utf-8") + with pytest.raises(MemoryStorageCorruption): + storage.load(user_id="alice") + assert path.read_text(encoding="utf-8") == "{broken" + + +def test_manifest_revision_conflict_rejects_stale_write(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + current = storage.load("agent-a", user_id="alice") + assert current["revision"] == 1 + assert storage.save(_memory_with_fact("new"), "agent-a", user_id="alice", expected_revision=1) + with pytest.raises(MemoryRevisionConflict): + storage.save(_memory_with_fact("stale"), "agent-a", user_id="alice", expected_revision=1) + + +def test_partial_summary_patch_preserves_omitted_sibling_sections(storage: FileMemoryStorage) -> None: + memory = create_empty_memory() + memory["user"]["workContext"] = {"summary": "old work", "updatedAt": "old"} + memory["user"]["personalContext"] = {"summary": "keep personal", "updatedAt": "old"} + memory["user"]["topOfMind"] = {"summary": "keep focus", "updatedAt": "old"} + assert storage.save(memory, user_id="alice") + + storage.apply_changes( + {"summaries": {"user": {"workContext": {"summary": "new work", "updatedAt": "new"}}}}, + user_id="alice", + expected_manifest_revision=1, + ) + + updated = storage.load(user_id="alice") + assert updated["user"]["workContext"] == {"summary": "new work", "updatedAt": "new"} + assert updated["user"]["personalContext"] == {"summary": "keep personal", "updatedAt": "old"} + assert updated["user"]["topOfMind"] == {"summary": "keep focus", "updatedAt": "old"} + + +def test_storage_delegates_index_lifecycle_and_search_to_retrieval(tmp_path: Path) -> None: + class FakeRetrieval: + def __init__(self) -> None: + self.upserts = [] + self.removes = [] + + def upsert(self, fact, *, scope, path): + self.upserts.append((fact["id"], scope, path)) + + def remove(self, fact_id, *, scope): + self.removes.append((fact_id, scope)) + + def search(self, query, *, scopes, top_k, mode, filters): + return [{"id": "fact-result", "score": 0.9, "query": query}] + + retrieval = FakeRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + assert scoped.save(_memory_with_fact(), "agent-a", user_id="alice") + assert retrieval.upserts[0][1] == {"userId": "alice", "agentName": "agent-a"} + assert scoped.search_facts("python", scopes=[{"userId": "alice", "agentName": "agent-a"}])[0]["score"] == 0.9 + + assert scoped.save(create_empty_memory(), "agent-a", user_id="alice") + assert retrieval.removes == [("fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", {"userId": "alice", "agentName": "agent-a"})] + + +def test_prepared_journal_restores_previous_manifest_and_fact(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact("original"), "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + memory = json.loads(memory_path.read_text(encoding="utf-8")) + fact_id = "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ" + fact_path = fact_file_path(memory_path, fact_id, agent_name="agent-a") + operation_id = "op-recovery-test" + recovery = memory_path.parent / ".recovery" / operation_id + recovery.mkdir(parents=True) + shutil.copy2(memory_path, recovery / "memory.json") + shutil.copy2(fact_path, recovery / f"{fact_id}.md") + relative_fact_path = fact_path.relative_to(memory_path.parent).as_posix() + journal = { + "operationId": operation_id, + "state": "prepared", + "agentName": "agent-a", + "expectedRevision": memory["revision"], + "nextRevision": memory["revision"] + 1, + "factIds": [fact_id], + "oldEntries": {fact_id: {"path": relative_fact_path}}, + } + (memory_path.parent / ".memory.journal.json").write_text(json.dumps(journal), encoding="utf-8") + fact_path.write_text("corrupt in-progress content", encoding="utf-8") + + loaded = storage.load("agent-a", user_id="alice") + + assert loaded["facts"][0]["content"] == "original" + assert not (memory_path.parent / ".memory.journal.json").exists() + + +def test_fact_repository_applies_upsert_and_physical_delete(storage: FileMemoryStorage) -> None: + first = storage.upsert_fact( + _memory_with_fact()["facts"][0], + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + expected_fact_revision=None, + ) + assert first["revision"] == 1 + assert first["complete"] is False + assert storage.get_fact("fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", user_id="alice", agent_name="agent-a")["content"] == "Project uses Python 3.12" + + updated = copy.deepcopy(first["upsertedFacts"][0]) + updated["content"] = "Project uses Python 3.13" + second = storage.apply_changes( + {"upserts": [updated]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + ) + assert second["upsertedFacts"][0]["content"] == "Project uses Python 3.13" + + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + fact_path = fact_file_path(memory_path, updated["id"], agent_name="agent-a") + assert fact_path.exists() + third = storage.delete_fact( + updated["id"], + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=2, + expected_fact_revision=second["upsertedFacts"][0]["revision"], + ) + assert third["deletedFactIds"] == [updated["id"]] + assert not fact_path.exists() + + +def test_upsert_fact_uses_separate_manifest_and_fact_revisions(storage: FileMemoryStorage) -> None: + created = storage.upsert_fact( + _memory_with_fact("first")["facts"][0], + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + expected_fact_revision=None, + ) + changed = copy.deepcopy(created["upsertedFacts"][0]) + changed["content"] = "updated" + + updated = storage.upsert_fact( + changed, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=created["revision"], + expected_fact_revision=changed["revision"], + ) + + assert updated["upsertedFacts"][0]["content"] == "updated" + + +def test_revision_conflicts_have_stable_manifest_and_fact_subtypes(storage: FileMemoryStorage) -> None: + storage.upsert_fact( + _memory_with_fact("first")["facts"][0], + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + expected_fact_revision=None, + ) + + with pytest.raises(MemoryManifestRevisionConflict): + storage.apply_changes( + {"summaries": {"user": {}, "history": {}}}, + user_id="alice", + expected_manifest_revision=0, + ) + + with pytest.raises(MemoryFactRevisionConflict): + storage.upsert_fact( + _memory_with_fact("duplicate")["facts"][0], + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + expected_fact_revision=None, + ) + + +def test_search_facts_declares_and_uses_substring_fallback(storage: FileMemoryStorage) -> None: + storage.upsert_fact(_memory_with_fact()["facts"][0], user_id="alice", agent_name="agent-a", expected_manifest_revision=0) + + results = storage.search_facts( + "python", + scopes=[{"userId": "alice", "agentName": "agent-a"}], + ) + + assert results[0]["fact"]["content"] == "Project uses Python 3.12" + assert results[0]["matchType"] == "substring" + assert storage.retrieval_status()["mode"] == "substring_fallback" + assert "substring-fallback" in storage.capabilities() + + +def test_strict_scope_and_custom_manifest_filename(tmp_path: Path) -> None: + strict = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path), strict_user_scope=True, manifest_filename="index.json")) + with pytest.raises(ValueError, match="user_id"): + strict.load() + + strict.upsert_fact(_memory_with_fact()["facts"][0], user_id="alice", agent_name="agent-a", expected_manifest_revision=0) + assert (tmp_path / "users" / "alice" / "index.json").exists() + + +def test_explicit_migrate_converts_legacy_json(storage: FileMemoryStorage) -> None: + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + path.write_text(json.dumps(_memory_with_fact()), encoding="utf-8") + original = path.read_bytes() + + report = storage.migrate(user_id="alice", agent_name="agent-a") + + assert report["migrated"] is True + assert report["fromVersion"] == "1.0" + assert report["toVersion"] == "2.0" + assert "facts" not in json.loads(path.read_text(encoding="utf-8")) + assert fact_file_path(path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a").exists() + assert path.with_name("memory.json.v1.bak").read_bytes() == original + + +def test_explicit_migration_notifies_retrieval_adapter(tmp_path: Path) -> None: + class RecordingRetrieval: + def __init__(self) -> None: + self.upserts: list[tuple[str, dict, str]] = [] + + def upsert(self, fact, *, scope, path): + self.upserts.append((fact["id"], scope, path)) + + def remove(self, fact_id, *, scope): + pass + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + retrieval = RecordingRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + path = scoped._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + path.write_text(json.dumps(_memory_with_fact("legacy indexed fact")), encoding="utf-8") + + scoped.migrate(user_id="alice", agent_name="agent-a") + scoped.migrate(user_id="alice", agent_name="agent-a") + + assert [(fact_id, scope) for fact_id, scope, _ in retrieval.upserts] == [("fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", {"userId": "alice", "agentName": "agent-a"})] + assert retrieval.upserts[0][2].endswith("fact_01HZZZZZZZZZZZZZZZZZZZZZZZ.md") + + +def test_first_agent_load_removes_legacy_per_agent_memory_json(storage: FileMemoryStorage) -> None: + user_memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = user_memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy_path.write_text(json.dumps(_memory_with_fact("legacy agent fact")), encoding="utf-8") + original = legacy_path.read_bytes() + + loaded = storage.load("agent-a", user_id="alice") + + assert loaded["facts"][0]["content"] == "legacy agent fact" + assert not legacy_path.exists() + assert legacy_path.with_name("memory.json.v1.bak").read_bytes() == original + assert user_memory_path.exists() + assert "facts" not in json.loads(user_memory_path.read_text(encoding="utf-8")) + + +def test_lazy_agent_migration_notifies_retrieval_adapter(tmp_path: Path) -> None: + class RecordingRetrieval: + def __init__(self) -> None: + self.upserts: list[str] = [] + + def upsert(self, fact, *, scope, path): + self.upserts.append(fact["id"]) + + def remove(self, fact_id, *, scope): + pass + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + retrieval = RecordingRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + memory_path = scoped._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy_path.write_text(json.dumps(_memory_with_fact("lazy indexed fact")), encoding="utf-8") + + assert scoped.load("agent-a", user_id="alice")["facts"][0]["content"] == "lazy indexed fact" + assert retrieval.upserts == ["fact_01HZZZZZZZZZZZZZZZZZZZZZZZ"] + + +def test_clear_all_migrates_and_then_removes_unread_legacy_agent_facts(storage: FileMemoryStorage) -> None: + global_memory = create_empty_memory() + global_memory["user"]["workContext"] = {"summary": "canonical summary", "updatedAt": "now"} + assert storage.save(global_memory, user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy = _memory_with_fact("legacy fact must stay cleared") + legacy["user"]["workContext"] = {"summary": "conflicting legacy summary", "updatedAt": "then"} + legacy_path.write_text(json.dumps(legacy), encoding="utf-8") + config_path = legacy_path.parent / "config.yaml" + config_path.write_text("name: agent-a\n", encoding="utf-8") + + cleared = storage.clear_all(user_id="alice") + + assert cleared["facts"] == [] + assert not legacy_path.exists() + assert legacy_path.with_name("memory.json.v1.bak").exists() + assert storage.load("agent-a", user_id="alice")["facts"] == [] + assert config_path.read_text(encoding="utf-8") == "name: agent-a\n" + + +def test_clear_all_legacy_migration_leaves_retrieval_fact_removed(tmp_path: Path) -> None: + class RecordingRetrieval: + def __init__(self) -> None: + self.events: list[tuple[str, str]] = [] + + def upsert(self, fact, *, scope, path): + self.events.append(("upsert", fact["id"])) + + def remove(self, fact_id, *, scope): + self.events.append(("remove", fact_id)) + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + retrieval = RecordingRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + memory_path = scoped._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy_path.write_text(json.dumps(_memory_with_fact("legacy indexed then cleared")), encoding="utf-8") + + scoped.clear_all(user_id="alice") + + assert retrieval.events == [ + ("upsert", "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ"), + ("remove", "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ"), + ] + + +def test_migration_rejects_a_different_existing_v1_backup(storage: FileMemoryStorage) -> None: + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + path.write_text(json.dumps(_memory_with_fact("source v1")), encoding="utf-8") + original = path.read_bytes() + path.with_name("memory.json.v1.bak").write_text(json.dumps(_memory_with_fact("older different v1")), encoding="utf-8") + + with pytest.raises(MemoryStorageCorruption, match="migration backup"): + storage.migrate(user_id="alice", agent_name="agent-a") + + assert path.read_bytes() == original + assert not fact_file_path(path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a").exists() + + +def test_migration_does_not_modify_v1_source_when_backup_write_fails( + storage: FileMemoryStorage, + monkeypatch: pytest.MonkeyPatch, +) -> None: + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + path.write_text(json.dumps(_memory_with_fact("source v1")), encoding="utf-8") + original = path.read_bytes() + real_atomic_write = storage_module._atomic_write + + def fail_backup(path_to_write: Path, raw: bytes) -> None: + if path_to_write.name.endswith(".v1.bak"): + raise OSError("backup disk unavailable") + real_atomic_write(path_to_write, raw) + + monkeypatch.setattr(storage_module, "_atomic_write", fail_backup) + + with pytest.raises(OSError, match="backup disk unavailable"): + storage.migrate(user_id="alice", agent_name="agent-a") + + assert path.read_bytes() == original + assert not fact_file_path(path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a").exists() + + +def test_fact_repository_requires_agent_name(storage: FileMemoryStorage) -> None: + with pytest.raises(ValueError, match="agent_name"): + storage.upsert_fact(_memory_with_fact()["facts"][0], user_id="alice", expected_manifest_revision=0) + + +def test_full_save_rejects_non_object_facts_without_deleting_existing(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact("keep me"), "agent-a", user_id="alice") + invalid = storage.load("agent-a", user_id="alice") + invalid["facts"] = [None] + + assert storage.save(invalid, "agent-a", user_id="alice", expected_revision=1) is False + assert storage.load("agent-a", user_id="alice")["facts"][0]["content"] == "keep me" + + +def test_agent_full_save_requires_facts_field_without_deleting_existing(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact("keep me"), "agent-a", user_id="alice") + + assert storage.save({"user": {}, "history": {}}, "agent-a", user_id="alice", expected_revision=1) is False + assert storage.load("agent-a", user_id="alice")["facts"][0]["content"] == "keep me" + + +def test_legacy_agent_migration_merges_existing_canonical_facts(storage: FileMemoryStorage) -> None: + canonical = _memory_with_fact("canonical") + canonical["facts"][0]["id"] = "fact_canonical" + assert storage.save(canonical, "agent-a", user_id="alice") + + legacy = _memory_with_fact("legacy") + legacy["facts"][0]["id"] = "fact_legacy" + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.write_text(json.dumps(legacy), encoding="utf-8") + + loaded = storage.load("agent-a", user_id="alice") + + assert {fact["content"] for fact in loaded["facts"]} == {"canonical", "legacy"} + + +def test_legacy_agent_migration_rejects_same_id_content_conflict(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact("canonical"), "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.write_text(json.dumps(_memory_with_fact("conflicting legacy")), encoding="utf-8") + + with pytest.raises(MemoryStorageCorruption, match="migration conflict"): + storage.load("agent-a", user_id="alice") + + legacy_path.unlink() + assert storage.reload("agent-a", user_id="alice")["facts"][0]["content"] == "canonical" + + +def test_concurrent_legacy_agent_migration_is_idempotent(tmp_path: Path) -> None: + config = DeerMemConfig(storage_path=str(tmp_path)) + first = FileMemoryStorage(config) + second = FileMemoryStorage(config) + memory_path = first._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy_path.write_text(json.dumps(_memory_with_fact("legacy")), encoding="utf-8") + + with ThreadPoolExecutor(max_workers=2) as pool: + loaded = list(pool.map(lambda store: store.load("agent-a", user_id="alice"), (first, second))) + + assert all([fact["content"] for fact in document["facts"]] == ["legacy"] for document in loaded) + assert not legacy_path.exists() + + +def test_default_manager_read_auto_migrates_global_legacy_facts(storage: FileMemoryStorage, tmp_path: Path) -> None: + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + legacy = _memory_with_fact("old global fact") + legacy["user"]["workContext"] = {"summary": "keep global profile", "updatedAt": "2026-01-01T00:00:00Z"} + path.write_text(json.dumps(legacy), encoding="utf-8") + + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + loaded = manager.get_memory(user_id="alice") + loaded_again = manager.reload_memory(user_id="alice") + + assert [fact["content"] for fact in loaded["facts"]] == ["old global fact"] + assert [fact["content"] for fact in loaded_again["facts"]] == ["old global fact"] + persisted = json.loads(path.read_text(encoding="utf-8")) + assert "facts" not in persisted + assert persisted["user"]["workContext"]["summary"] == "keep global profile" + assert fact_file_path(path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="__default__").exists() + + +@pytest.mark.parametrize("custom_first", [True, False]) +def test_default_bucket_is_isolated_from_custom_lead_agent(tmp_path: Path, custom_first: bool) -> None: + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + custom_dir = tmp_path / "users" / "alice" / "agents" / "lead-agent" + custom_dir.mkdir(parents=True) + (custom_dir / "config.yaml").write_text("name: lead-agent\n", encoding="utf-8") + + if custom_first: + manager.create_fact("custom fact", agent_name="lead-agent", user_id="alice") + manager.create_fact("default fact", user_id="alice") + else: + manager.create_fact("default fact", user_id="alice") + manager.create_fact("custom fact", agent_name="lead-agent", user_id="alice") + + assert [fact["content"] for fact in manager.get_memory(user_id="alice")["facts"]] == ["default fact"] + assert [fact["content"] for fact in manager.get_memory(agent_name="lead-agent", user_id="alice")["facts"]] == ["custom fact"] + assert list((tmp_path / "users" / "alice" / "agents" / "__default__" / "facts").glob("**/*.md")) + assert list((tmp_path / "users" / "alice" / "agents" / "lead-agent" / "facts").glob("**/*.md")) + + +def test_deermem_canonicalizes_agent_names_to_lowercase(tmp_path: Path) -> None: + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + + manager.create_fact("case-insensitive fact", agent_name="Lead-Agent", user_id="alice") + + lower = manager.get_memory(agent_name="lead-agent", user_id="alice") + upper = manager.get_memory(agent_name="LEAD-AGENT", user_id="alice") + stored = manager._storage.load("lead-agent", user_id="alice")["facts"][0] + assert [fact["id"] for fact in lower["facts"]] == [fact["id"] for fact in upper["facts"]] + assert stored["scope"]["agentName"] == "lead-agent" + + +def test_old_implicit_lead_agent_bucket_moves_to_reserved_default(tmp_path: Path) -> None: + config = DeerMemConfig(storage_path=str(tmp_path)) + old_storage = FileMemoryStorage(config) + old_storage.upsert_fact( + _memory_with_fact("fact written by the previous PR version")["facts"][0], + user_id="alice", + agent_name="lead-agent", + expected_manifest_revision=0, + ) + + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + loaded = manager.get_memory(user_id="alice") + + assert [fact["content"] for fact in loaded["facts"]] == ["fact written by the previous PR version"] + assert not (tmp_path / "users" / "alice" / "agents" / "lead-agent").exists() + assert list((tmp_path / "users" / "alice" / "agents" / "__default__" / "facts").glob("**/*.md")) + + +def test_old_implicit_bucket_with_unknown_files_is_preserved(tmp_path: Path) -> None: + legacy_dir = tmp_path / "users" / "alice" / "agents" / "lead-agent" + legacy_dir.mkdir(parents=True) + unknown = legacy_dir / "SOUL.md" + unknown.write_text("possibly a partially created custom agent", encoding="utf-8") + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + + with pytest.raises(MemoryCorruptionError, match="unexpected entries"): + manager.get_memory(user_id="alice") + + assert unknown.read_text(encoding="utf-8") == "possibly a partially created custom agent" + + +def test_create_returns_fresh_full_view_after_disjoint_rebase(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + competing_storage = FileMemoryStorage(manager._config) + real_apply_changes = manager._storage.apply_changes + injected = False + + def apply_with_competing_create(change_set, **scope): + nonlocal injected + if not injected: + injected = True + competing_fact = _memory_with_fact("competing fact")["facts"][0] + competing_fact["id"] = "fact_competing" + competing_storage.apply_changes( + {"upserts": [competing_fact], "upsertRevisions": {"fact_competing": None}}, + agent_name=scope["agent_name"], + user_id=scope["user_id"], + expected_manifest_revision=scope["expected_manifest_revision"], + ) + return real_apply_changes(change_set, **scope) + + monkeypatch.setattr(manager._storage, "apply_changes", apply_with_competing_create) + + returned, created_id = manager.create_fact("requested fact", user_id="alice") + fresh = manager.reload_memory(user_id="alice") + + assert created_id is not None + assert {fact["id"] for fact in returned["facts"]} == {fact["id"] for fact in fresh["facts"]} + assert {fact["content"] for fact in returned["facts"]} == {"competing fact", "requested fact"} + + +def test_scoped_clear_recomputes_after_competing_create(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + manager = DeerMem(backend_config={"storage_path": str(tmp_path)}) + manager.create_fact("existing fact", agent_name="agent-a", user_id="alice") + competing_storage = FileMemoryStorage(manager._config) + real_apply_changes = manager._storage.apply_changes + injected = False + + def apply_with_competing_create(change_set, **scope): + nonlocal injected + if not injected: + injected = True + competing_fact = _memory_with_fact("competing fact")["facts"][0] + competing_fact.update({"id": "fact_competing", "revision": 1}) + competing_storage.apply_changes( + {"upserts": [competing_fact], "upsertRevisions": {"fact_competing": None}}, + agent_name=scope["agent_name"], + user_id=scope["user_id"], + expected_manifest_revision=scope["expected_manifest_revision"], + ) + return real_apply_changes(change_set, **scope) + + monkeypatch.setattr(manager._storage, "apply_changes", apply_with_competing_create) + + returned = manager.clear_memory(agent_name="agent-a", user_id="alice") + fresh = manager.reload_memory(agent_name="agent-a", user_id="alice") + + assert returned["facts"] == [] + assert fresh["facts"] == [] + + +def test_max_facts_is_recomputed_after_competing_create(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + manager = DeerMem(backend_config={"storage_path": str(tmp_path), "max_facts": 10}) + initial = create_empty_memory() + initial["facts"] = [] + for index in range(9): + fact = copy.deepcopy(_memory_with_fact(f"existing {index}")["facts"][0]) + fact.update({"id": f"fact_existing_{index}", "confidence": 0.9}) + initial["facts"].append(fact) + assert manager._storage.save(initial, "agent-a", user_id="alice") + + competing_storage = FileMemoryStorage(manager._config) + real_apply_changes = manager._storage.apply_changes + injected = False + + def apply_with_competing_create(change_set, **scope): + nonlocal injected + if not injected: + injected = True + competing_fact = copy.deepcopy(_memory_with_fact("competing fact")["facts"][0]) + competing_fact.update({"id": "fact_competing", "confidence": 0.9}) + competing_storage.apply_changes( + {"upserts": [competing_fact], "upsertRevisions": {"fact_competing": None}}, + agent_name=scope["agent_name"], + user_id=scope["user_id"], + expected_manifest_revision=scope["expected_manifest_revision"], + ) + return real_apply_changes(change_set, **scope) + + monkeypatch.setattr(manager._storage, "apply_changes", apply_with_competing_create) + + returned, created_id = manager.create_fact( + "requested fact", + confidence=0.95, + agent_name="agent-a", + user_id="alice", + ) + fresh = manager.reload_memory(agent_name="agent-a", user_id="alice") + + assert created_id is not None + assert len(returned["facts"]) <= 10 + assert len(fresh["facts"]) <= 10 + assert {fact["id"] for fact in returned["facts"]} == {fact["id"] for fact in fresh["facts"]} + + +def test_agent_fact_scope_must_match_requested_directory(storage: FileMemoryStorage) -> None: + assert storage.save(_memory_with_fact(), "agent-a", user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + fact_path = fact_file_path(memory_path, "fact_01HZZZZZZZZZZZZZZZZZZZZZZZ", agent_name="agent-a") + fact_path.write_text(fact_path.read_text(encoding="utf-8").replace("agent_name: agent-a", "agent_name: agent-b"), encoding="utf-8") + + with pytest.raises(MemoryStorageCorruption, match="scope mismatch"): + storage.reload("agent-a", user_id="alice") + + +def test_legacy_fact_path_must_remain_below_user_directory(storage: FileMemoryStorage, tmp_path: Path) -> None: + outside = tmp_path / "outside.md" + outside.write_text("do not read", encoding="utf-8") + path = storage._get_memory_file_path(user_id="alice") + path.parent.mkdir(parents=True) + path.write_text( + json.dumps( + { + "version": "2.0", + "revision": 0, + "user": {}, + "history": {}, + "facts": {"fact_escape": {"path": "../../outside.md"}}, + } + ), + encoding="utf-8", + ) + + with pytest.raises(MemoryStorageCorruption, match="escapes"): + storage.migrate(user_id="alice", agent_name="agent-a") + + +def test_fact_schema_rejects_invalid_collection_and_revision_types(storage: FileMemoryStorage) -> None: + invalid_topics = _memory_with_fact()["facts"][0] + invalid_topics["topics"] = "python" + with pytest.raises(ValueError, match="topics"): + storage.upsert_fact(invalid_topics, user_id="alice", agent_name="agent-a", expected_manifest_revision=0) + + invalid_revision = _memory_with_fact()["facts"][0] + invalid_revision["revision"] = [] + with pytest.raises(ValueError, match="revision"): + storage.upsert_fact(invalid_revision, user_id="alice", agent_name="agent-a", expected_manifest_revision=0) + + +def test_changed_fact_increments_revision_and_updated_at(storage: FileMemoryStorage) -> None: + created = storage.upsert_fact(_memory_with_fact()["facts"][0], user_id="alice", agent_name="agent-a", expected_manifest_revision=0) + original = created["upsertedFacts"][0] + updated = copy.deepcopy(original) + updated["content"] = "Project uses Python 3.13" + + result = storage.apply_changes({"upserts": [updated]}, user_id="alice", agent_name="agent-a", expected_manifest_revision=1) + changed = result["upsertedFacts"][0] + + assert changed["revision"] == original["revision"] + 1 + assert changed["updatedAt"] > original["updatedAt"] + assert changed["createdAt"] == original["createdAt"] + + +def test_single_fact_change_writes_and_notifies_only_that_fact(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + class RecordingRetrieval: + def __init__(self) -> None: + self.upserts: list[str] = [] + + def upsert(self, fact, *, scope, path): + self.upserts.append(fact["id"]) + + def remove(self, fact_id, *, scope): + pass + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + retrieval = RecordingRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + memory = create_empty_memory() + first = _memory_with_fact("first")["facts"][0] + second = copy.deepcopy(first) + second.update({"id": "fact_second", "content": "second"}) + memory["facts"] = [first, second] + assert scoped.save(memory, "agent-a", user_id="alice") + retrieval.upserts.clear() + + markdown_writes: list[Path] = [] + real_atomic_write = storage_module._atomic_write + + def record_atomic_write(path: Path, raw: bytes) -> None: + if path.suffix == ".md": + markdown_writes.append(path) + real_atomic_write(path, raw) + + monkeypatch.setattr(storage_module, "_atomic_write", record_atomic_write) + loaded = scoped.load("agent-a", user_id="alice") + changed = copy.deepcopy(next(fact for fact in loaded["facts"] if fact["id"] == first["id"])) + changed["content"] = "first changed" + + scoped.apply_changes({"upserts": [changed]}, user_id="alice", agent_name="agent-a", expected_manifest_revision=1) + + assert [path.stem for path in markdown_writes] == [first["id"]] + assert retrieval.upserts == [first["id"]] + + +def test_incremental_change_does_not_scan_all_fact_files(storage: FileMemoryStorage, monkeypatch: pytest.MonkeyPatch) -> None: + memory = create_empty_memory() + first = _memory_with_fact("first")["facts"][0] + second = copy.deepcopy(first) + second.update({"id": "fact_second", "content": "second"}) + memory["facts"] = [first, second] + assert storage.save(memory, "agent-a", user_id="alice") + changed = copy.deepcopy(storage.get_fact(first["id"], user_id="alice", agent_name="agent-a")) + changed["content"] = "first changed" + + def fail_full_scan(*args, **kwargs): + raise AssertionError("incremental change attempted to scan all facts") + + monkeypatch.setattr(storage, "_load_agent_facts", fail_full_scan) + result = storage.apply_changes( + {"upserts": [changed]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + ) + + assert result["complete"] is False + assert [fact["id"] for fact in result["upsertedFacts"]] == [first["id"]] + assert result["upsertedFacts"][0]["content"] == "first changed" + + +def test_fresh_storage_incremental_result_does_not_hide_untouched_sibling(tmp_path: Path) -> None: + config = DeerMemConfig(storage_path=str(tmp_path)) + writer = FileMemoryStorage(config) + memory = create_empty_memory() + first = _memory_with_fact("first")["facts"][0] + second = copy.deepcopy(first) + second.update({"id": "fact_second", "content": "second"}) + memory["facts"] = [first, second] + assert writer.save(memory, "agent-a", user_id="alice") + + fresh = FileMemoryStorage(config) + changed = copy.deepcopy(fresh.get_fact(first["id"], user_id="alice", agent_name="agent-a")) + changed["content"] = "first changed" + result = fresh.apply_changes( + {"upserts": [changed]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + ) + + assert result["complete"] is False + assert [fact["id"] for fact in result["upsertedFacts"]] == [first["id"]] + reloaded = fresh.load("agent-a", user_id="alice") + assert {fact["content"] for fact in reloaded["facts"]} == {"first changed", "second"} + + +def test_stale_user_revision_rebases_disjoint_fact_change_but_not_same_fact(storage: FileMemoryStorage) -> None: + memory = create_empty_memory() + first = _memory_with_fact("first")["facts"][0] + second = copy.deepcopy(first) + second.update({"id": "fact_second", "content": "second"}) + memory["facts"] = [first, second] + assert storage.save(memory, "agent-a", user_id="alice") + snapshot = storage.load("agent-a", user_id="alice") + first_update = copy.deepcopy(next(fact for fact in snapshot["facts"] if fact["id"] == first["id"])) + second_update = copy.deepcopy(next(fact for fact in snapshot["facts"] if fact["id"] == second["id"])) + first_update["content"] = "first changed" + second_update["content"] = "second changed" + + storage.apply_changes({"upserts": [first_update]}, user_id="alice", agent_name="agent-a", expected_manifest_revision=1) + rebased = storage.apply_changes( + {"upserts": [second_update]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + allow_manifest_rebase=True, + ) + assert [fact["content"] for fact in rebased["upsertedFacts"]] == ["second changed"] + assert {fact["content"] for fact in storage.load("agent-a", user_id="alice")["facts"]} == {"first changed", "second changed"} + + first_update["content"] = "stale overwrite" + with pytest.raises(MemoryRevisionConflict): + storage.apply_changes( + {"upserts": [first_update]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=1, + allow_manifest_rebase=True, + ) + + +def test_rebuild_index_continues_after_adapter_exception(tmp_path: Path) -> None: + class FlakyRetrieval: + def upsert(self, fact, *, scope, path): + if fact["id"] == "fact_bad": + raise RuntimeError("index unavailable") + + def remove(self, fact_id, *, scope): + pass + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=FlakyRetrieval()) + memory = create_empty_memory() + good = _memory_with_fact("good")["facts"][0] + bad = copy.deepcopy(good) + bad.update({"id": "fact_bad", "content": "bad"}) + memory["facts"] = [good, bad] + assert scoped.save(memory, "agent-a", user_id="alice") + + result = scoped.rebuild_index([{"userId": "alice", "agentName": "agent-a"}]) + + assert result == {"supported": True, "indexed": 1, "failed": 1} + + +def test_full_rebuild_index_accepts_original_email_user_scope(tmp_path: Path) -> None: + class RecordingRetrieval: + def __init__(self) -> None: + self.fact_ids: list[str] = [] + + def upsert(self, fact, *, scope, path): + self.fact_ids.append(fact["id"]) + + def remove(self, fact_id, *, scope): + pass + + def search(self, query, *, scopes, top_k, mode, filters): + return [] + + retrieval = RecordingRetrieval() + scoped = FileMemoryStorage(DeerMemConfig(storage_path=str(tmp_path)), retrieval=retrieval) + scoped.upsert_fact( + _memory_with_fact("email user fact")["facts"][0], + user_id="test@example.com", + agent_name="agent-a", + expected_manifest_revision=0, + expected_fact_revision=None, + ) + retrieval.fact_ids.clear() + + result = scoped.rebuild_index() + + assert result == {"supported": True, "indexed": 1, "failed": 0} + assert retrieval.fact_ids == ["fact_01HZZZZZZZZZZZZZZZZZZZZZZZ"] + + +def test_scope_lock_cache_releases_unused_entries(storage: FileMemoryStorage) -> None: + key = storage._cache_key("agent-a", user_id="alice") + lock = storage._scope_lock(key) + lock_ref = weakref.ref(lock) + del lock + gc.collect() + + assert lock_ref() is None + assert key not in storage._scope_locks + + +def test_atomic_write_syncs_parent_directory(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + synced: list[Path] = [] + monkeypatch.setattr(storage_module, "_fsync_parent_directory", lambda path: synced.append(path)) + target = tmp_path / "nested" / "memory.json" + + storage_module._atomic_write(target, b"{}") + + assert synced == [target.parent] + + +def test_migrate_reports_legacy_agent_file(storage: FileMemoryStorage) -> None: + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy_path.write_text(json.dumps(_memory_with_fact("legacy")), encoding="utf-8") + + report = storage.migrate(user_id="alice", agent_name="agent-a") + + assert report["migrated"] is True + assert not legacy_path.exists() + + +def test_migrate_preserves_legacy_summaries_before_deleting_agent_file(storage: FileMemoryStorage) -> None: + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy = _memory_with_fact("legacy") + legacy["user"]["workContext"] = {"summary": "legacy profile", "updatedAt": "2026-01-01T00:00:00Z"} + legacy["history"]["recentMonths"] = {"summary": "legacy history", "updatedAt": "2026-01-01T00:00:00Z"} + legacy_path.write_text(json.dumps(legacy), encoding="utf-8") + + storage.migrate(user_id="alice", agent_name="agent-a") + + global_memory = storage.load(user_id="alice") + assert global_memory["user"]["workContext"]["summary"] == "legacy profile" + assert global_memory["history"]["recentMonths"]["summary"] == "legacy history" + assert not legacy_path.exists() + + +def test_migrate_keeps_legacy_file_when_summary_conflicts(storage: FileMemoryStorage) -> None: + global_memory = create_empty_memory() + global_memory["user"]["workContext"] = {"summary": "canonical profile", "updatedAt": "now"} + assert storage.save(global_memory, user_id="alice") + memory_path = storage._get_memory_file_path("agent-a", user_id="alice") + legacy_path = memory_path.parent / "agents" / "agent-a" / "memory.json" + legacy_path.parent.mkdir(parents=True) + legacy = _memory_with_fact("legacy") + legacy["user"]["workContext"] = {"summary": "different profile", "updatedAt": "then"} + legacy_path.write_text(json.dumps(legacy), encoding="utf-8") + + with pytest.raises(MemoryStorageCorruption, match="summary migration conflict"): + storage.migrate(user_id="alice", agent_name="agent-a") + + assert legacy_path.exists() + assert not fact_file_path(memory_path, legacy["facts"][0]["id"], agent_name="agent-a").exists() + + +def test_two_storage_instances_cannot_create_the_same_fact_id(tmp_path: Path) -> None: + config = DeerMemConfig(storage_path=str(tmp_path)) + first_storage = FileMemoryStorage(config) + second_storage = FileMemoryStorage(config) + new_fact = _memory_with_fact("first writer")["facts"][0] + new_fact.pop("revision") + + first_storage.apply_changes( + {"upserts": [copy.deepcopy(new_fact)]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + ) + competing = copy.deepcopy(new_fact) + competing["content"] = "second writer" + + with pytest.raises(MemoryRevisionConflict, match="must not already exist"): + second_storage.apply_changes( + {"upserts": [competing]}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + allow_manifest_rebase=True, + ) + + stored = first_storage.get_fact(new_fact["id"], user_id="alice", agent_name="agent-a") + assert stored["content"] == "first writer" + + +def test_stale_summary_change_is_not_rebased_over_newer_summary(storage: FileMemoryStorage) -> None: + newer = create_empty_memory() + newer["user"]["workContext"] = {"summary": "newer", "updatedAt": "now"} + assert storage.save(newer, user_id="alice", expected_revision=0) + stale = create_empty_memory() + stale["user"]["workContext"] = {"summary": "stale", "updatedAt": "before"} + fact = _memory_with_fact("should not be committed")["facts"][0] + fact.pop("revision") + + with pytest.raises(MemoryRevisionConflict): + storage.apply_changes( + {"upserts": [fact], "upsertRevisions": {fact["id"]: None}, "summaries": {"user": stale["user"], "history": stale["history"]}}, + user_id="alice", + agent_name="agent-a", + expected_manifest_revision=0, + ) + + assert storage.load(user_id="alice")["user"]["workContext"]["summary"] == "newer" + assert not fact_file_path(storage._get_memory_file_path("agent-a", user_id="alice"), fact["id"], agent_name="agent-a").exists() + + +@pytest.mark.skipif(os.name != "nt", reason="Windows append-mode lock behavior") +def test_windows_lock_file_does_not_grow_per_acquisition(storage: FileMemoryStorage) -> None: + for _ in range(5): + assert storage.save(create_empty_memory(), user_id="alice") + lock_path = storage._get_memory_file_path(user_id="alice").parent / ".memory.lock" + assert lock_path.stat().st_size == 1 diff --git a/backend/tests/test_memory_storage_user_isolation.py b/backend/tests/test_memory_storage_user_isolation.py index f2b616868..9e9f0ff5b 100644 --- a/backend/tests/test_memory_storage_user_isolation.py +++ b/backend/tests/test_memory_storage_user_isolation.py @@ -5,6 +5,7 @@ from pathlib import Path import pytest from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig +from deerflow.agents.memory.backends.deermem.deermem.core.paths import fact_file_path from deerflow.agents.memory.backends.deermem.deermem.core.storage import FileMemoryStorage, create_empty_memory @@ -75,12 +76,15 @@ class TestUserIsolatedStorage: assert s.load(user_id="alice")["user"]["workContext"]["summary"] == "alice" def test_user_agent_memory_file_location(self, base_dir: Path): - """Per-user per-agent memory uses {root}/users/{uid}/agents/{name}/memory.json.""" + """One user JSON stores summaries; agent facts live below the agent directory.""" s = FileMemoryStorage(DeerMemConfig()) memory = create_empty_memory() - memory["user"]["workContext"]["summary"] = "agent scoped" + memory["facts"] = [{"id": "fact_agent", "content": "agent scoped"}] s.save(memory, "test-agent", user_id="alice") - assert (base_dir / "users" / "alice" / "agents" / "test-agent" / "memory.json").exists() + memory_path = base_dir / "users" / "alice" / "memory.json" + assert memory_path.exists() + assert not (base_dir / "users" / "alice" / "agents" / "test-agent" / "memory.json").exists() + assert fact_file_path(memory_path, "fact_agent", agent_name="test-agent").exists() def test_cache_key_is_user_agent_tuple(self, base_dir: Path): """Cache keys must be (user_id, agent_name) tuples.""" diff --git a/config.example.yaml b/config.example.yaml index 078fc0035..0b26bb31e 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -1638,6 +1638,11 @@ memory: # Each backend self-interprets it (DeerMem parses it into DeerMemConfig). backend_config: storage_path: "" # empty = deer-flow base_dir (factory injects absolute runtime_home); a non-empty path is the root DIRECTORY (per-user memory under {storage_path}/users/{uid}/memory.json) + storage_class: file # file (default) or a dotted MemoryStorage class path; invalid persistent backends fail fast + strict_user_scope: false # set true in authenticated deployments after all callers propagate user_id + manifest_filename: memory.json # user-global JSON: version/revision/time + user/history only; no facts or fact index + file_lock_timeout_seconds: 10 # per-scope cross-process advisory lock timeout (single-machine local filesystem) + retrieval_adapter: "" # optional dotted factory(config) supplied by the retrieval module debounce_seconds: 30 # Wait time before processing queued updates model: # LLM for memory extraction; omit all fields = no extraction (non-LLM ops still work; an update raises) # provider: openai diff --git a/docs/plans/STORAGE_REWRITE_CHANGES.md b/docs/plans/STORAGE_REWRITE_CHANGES.md new file mode 100644 index 000000000..381bb761d --- /dev/null +++ b/docs/plans/STORAGE_REWRITE_CHANGES.md @@ -0,0 +1,260 @@ +# DeerMem Storage Rewrite: Implementation Guide + +> This document explains the final PR behavior for readers who are new to the codebase. It complements `STORAGE_REWRITE_PLAN.md`: the plan explains the intended contract; this file explains where the code changed and how a request travels through it. + +## 1. What changed on disk + +Before the rewrite, facts and summaries could live together in a coarse JSON document. The final layout is: + +```text +users/alice/ +├── memory.json +├── memory.json.v1.bak +└── agents/ + ├── __default__/facts/30/fact_a.md + └── research-agent/facts/ee/fact_b.md +``` + +`memory.json` owns project-independent summaries. Each Markdown file owns one atomic fact. No JSON fact index duplicates the Markdown repository. + +## 2. Main files + +### `deer_mem.py` + +This is the DeerMem implementation of the public `MemoryManager` plugin interface. + +It: + +- maps omitted agents to `__default__`; +- canonicalizes explicit agent names to lowercase; +- translates storage-private exceptions to public manager errors; +- converts structured internal source metadata to the legacy public string; +- reloads complete documents where the manager/API contract promises completeness. + +Gateway code does not import DeerMem storage internals. + +### `deermem/core/paths.py` + +This owns user, agent, and fact path validation. The internal `__default__` sentinel is deliberately outside the public custom-agent regular expression. + +Fact paths are sharded by the first two hexadecimal characters of `SHA-256(fact_id)`. This gives deterministic direct lookup while distributing generated IDs that all begin with `fact_`. + +### `deermem/core/storage.py` + +This owns canonical parsing/rendering, locks, revisions, journaling, recovery, migration, cache invalidation, fact repository operations, and retrieval notifications. + +### `deermem/core/updater.py` + +This converts manual CRUD and LLM extraction results into repository change sets. It decides whether an operation is a point mutation or depends on a complete snapshot. + +### `app/gateway/routers/memory.py` + +This keeps the HTTP schema compatible. It catches only backend-neutral MemoryManager exceptions and maps concurrency to 409 and corruption to a stable 500. + +### `scripts/migrate_memory_markdown.py` + +This optional CLI lets operators preview or proactively run the same migration that normal first reads perform lazily. + +## 3. Read trajectory + +For an ordinary request with no explicit agent: + +```text +Gateway / middleware + -> MemoryManager.get_memory(user_id, agent_name=None) + -> DeerMem canonicalizes agent to __default__ + -> FileMemoryStorage.load(__default__) + -> recover journal / migrate legacy data when needed + -> compare memory.json (mtime_ns, file_size, revision) with cache + -> cache hit, or parse selected agent Markdown files + -> DeerMem converts source metadata to public strings + -> Gateway validates and returns the compatibility document +``` + +For a custom agent, the same trajectory reads that lowercase agent bucket and combines it with the user's shared summaries. + +## 4. Single-fact update trajectory + +```text +MemoryManager.update_fact(fact_id) + -> storage.get_fact(fact_id) + -> build one upsert with expected fact revision + -> acquire in-process + cross-process locks + -> recover an interrupted prior journal if present + -> validate shared revision and target fact revision + -> journal only memory.json + the addressed fact + -> atomically replace the Markdown fact and memory.json + -> fsync files and the POSIX parent directory + -> notify retrieval for that fact only + -> reload a complete compatibility response +``` + +Unchanged sibling facts are not backed up, rewritten, or re-indexed. + +## 5. Why there are two revisions + +The shared JSON revision protects the multi-file transaction. The fact revision protects one Markdown object. + +They answer different questions: + +```text +manifest revision: Did anything in this user's memory change? +fact revision: Is the exact fact I read still the same object version? +``` + +Direct `upsert_fact` and `delete_fact` therefore accept separate expected manifest and fact revisions. + +## 6. Safe rebase versus fresh-snapshot retry + +A point update to `fact_a` can safely move from manifest revision 10 to 11 if `fact_a` still has the expected revision. A scoped clear cannot: its meaning depends on every fact that existed at the successful commit. + +The final implementation uses this rule: + +```text +point mutation + valid fact preconditions + -> explicit bounded manifest rebase is allowed + +clear / trim / consolidation / collection-derived set + -> manifest conflict + -> reload full document + -> recompute complete operation + -> bounded retry +``` + +This fixes two reviewed races: + +- a fact created between a scoped-clear snapshot and commit no longer survives behind a successful “empty” response; +- two creators starting from 9/10 facts cannot commit 11 facts. + +## 7. Clear semantics + +The manager distinguishes two calls: + +```text +clear_memory(user_id=alice) + clears shared user/history summaries and every agent fact bucket + preserves custom-agent config.yaml and SOUL.md + +clear_memory(user_id=alice, agent_name=research-agent) + clears only research-agent facts + preserves shared user/history summaries +``` + +The file backend's all-user clear holds the user lock while enumerating buckets. It first migrates facts from any unread legacy per-agent `memory.json` without adopting legacy summaries that are about to be cleared, then deletes the resulting canonical fact files; retained `.v1.bak` files are inactive migration evidence and are never read back, while custom-agent configuration remains untouched. + +## 8. Source compatibility + +On disk: + +```yaml +source: + type: conversation + threadId: thread_123 +``` + +Public API: + +```json +{"source": "thread_123"} +``` + +Manual/import/consolidation sources similarly return their type string. The richer internal form is ready for retrieval metadata without breaking the frontend. + +## 9. Cache behavior + +The earlier cache signature scanned and statted every fact file on every `load()`. The final cache observes the shared `memory.json` only. + +The token is `(mtime_ns, file_size, revision)`. Every supported storage mutation replaces the JSON and increments its revision, so another process's write invalidates cached agent documents even when a coarse-mtime filesystem reports identical metadata for same-size writes. Validation reads one shared JSON and does not scale with the number of fact files. This may invalidate more agents than strictly necessary, but it avoids an O(n) fact-directory walk without introducing a second manifest. + +Manual out-of-band Markdown edits are not part of the supported write API and require `reload()`. + +Per-scope in-process locks live in a weak-value dictionary, so inactive users do not permanently grow the lock map. + +## 10. Migration trajectory + +When a v1 `memory.json` still contains facts: + +```text +first load/reload + -> detect legacy facts/version + -> acquire normal locks + -> durably retain every source as memory.json.v1.bak + -> normalize facts into __default__ scope + -> commit Markdown files and preserved summaries through the journal + -> rewrite memory.json without facts + -> release storage locks and notify the configured retrieval adapter + -> continue the original read +``` + +The explicit CLI path emits the same retrieval upserts. Re-running an idempotent migration emits no duplicate notification because no fact is rewritten. + +Operators may run: + +```bash +cd backend +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users +``` + +The command is idempotent, accepts repeated `--user-id`, supports `--storage-path`, continues reporting after one user fails, and exits non-zero if any user failed. It is not required for startup. + +This is a one-way application migration: pre-PR code cannot read Markdown facts. Before upgrading a persistent deployment, stop DeerFlow and take a filesystem snapshot or full backup of the configured storage root. Storage also retains each destructive v1 JSON source beside its original path as `{source_filename}.v1.bak` before committing v2 data. Existing backups are immutable; a mismatch or backup-write failure aborts before changing the source. The local backup contains pre-migration data only and does not replace the full snapshot requirement. + +## 11. Configuration changes + +`manifest_filename` and lock timeout remain configurable. Markdown format and journaling are required invariants, not modes. + +`fact_format` and `journal_enabled` are not `DeerMemConfig` fields. If supplied in `backend_config`, they follow the normal unknown-key behavior: DeerMem logs a warning and ignores them. + +## 12. Agent naming + +The Gateway already stores custom-agent names in lowercase. DeerMem now makes the same rule explicit at its manager boundary: + +```text +Lead-Agent == lead-agent == LEAD-AGENT +``` + +This gives Linux, macOS, and Windows the same behavior. A cross-layer contract test pins that both public naming patterns reject `__default__`, while storage may use the sentinel internally. + +## 13. Failure behavior + +- Stale same-fact write: conflict, no overwrite. +- Same-ID concurrent create: conflict, no conversion into update. +- Continuing collection contention: conflict after bounded retries. +- Corrupt JSON/Markdown: stable public corruption error; original files retained. +- Conflicting legacy summaries/facts: migration fails loudly and preserves the legacy source. +- Missing, unreadable, or mismatched persistent v1 backup: migration stops before changing the source. +- Retrieval adapter failure: persistence remains committed; the failure is logged/reported per fact. + +## 14. Review-driven fixes retained in the implementation + +- Default bucket changed from colliding `lead-agent` to reserved `__default__`. +- Custom-agent deletion requires a genuine config file. +- Legacy global facts migrate on first read and through an optional CLI. +- Public responses retain string `source`. +- Conflicts map to HTTP 409 through MemoryManager-neutral errors. +- Clear All and scoped clear have distinct semantics. +- Full index rebuild validates original email-style user IDs correctly. +- Lock cache is reclaimable. +- Manifest and fact revisions are separate. +- Retry dispatch uses exception subtypes, not message text. +- POSIX atomic replacement fsyncs the parent directory. +- Stale default-bucket test now asserts `__default__`. +- Snapshot-derived set operations recompute instead of replaying stale intent. +- Cache validation no longer scans every fact on every read. +- Cache tokens include the persisted revision to survive coarse-mtime same-size writes. +- Fact paths shard by `SHA-256(fact_id)[:2]` rather than the constant `fa` prefix. +- Destructive migration writes immutable `.v1.bak` sources before v2 data. +- Fixed storage invariants are no longer presented as configurable features. + +## 15. Verification map + +Primary coverage lives in: + +- `tests/test_memory_storage_markdown.py` +- `tests/test_deermem_self_contained.py` +- `tests/test_memory_router.py` +- `tests/test_memory_markdown_migration_cli.py` +- `tests/test_custom_agent.py` + +Tests cover layout, compatibility, incremental writes, concurrency, cap enforcement, clear semantics, corruption, migration, retrieval delegation, cache invalidation, path containment, portability, Windows lock behavior, and POSIX directory sync. diff --git a/docs/plans/STORAGE_REWRITE_PLAN.md b/docs/plans/STORAGE_REWRITE_PLAN.md new file mode 100644 index 000000000..816a355af --- /dev/null +++ b/docs/plans/STORAGE_REWRITE_PLAN.md @@ -0,0 +1,224 @@ +# DeerMem Storage Rewrite Plan + +> Status: implemented in PR #4279 and retained as the design record requested during review. +> Scope: file-backed storage only. Retrieval ranking, embeddings, project scope, and recall policy remain separate work. + +## 1. Objective + +Separate durable memory into two ownership layers: + +```text +user_id +├── memory.json # project-independent user/history summaries +└── agents/{agent_name}/facts/ # agent-related atomic facts + └── {sha256-prefix}/{fact_id}.md +``` + +The rewrite must: + +1. keep one global summary JSON per user; +2. store every fact as one canonical Markdown document; +3. prevent custom agents from sharing a fact repository accidentally; +4. make single-fact writes genuinely incremental; +5. preserve the existing MemoryManager and HTTP response shape; +6. give future retrieval implementations a stable fact repository API; +7. survive stale writers, process concurrency, and interrupted multi-file writes; +8. upgrade existing JSON facts without requiring application downtime. + +## 2. Non-goals + +- No `project` or `project_id` storage scope in this PR. +- No vector database, BM25, embedding, MMR, or reranking implementation. +- No change to when facts are recalled or injected into prompts. +- No SQLite/Postgres backend. +- No new frontend memory-management experience. +- No support for disabling the safety journal or selecting another fact format. + +## 3. Canonical layout + +```text +{storage_root}/users/{safe_user_id}/ +├── memory.json +├── memory.json.v1.bak # retained only after migrating this v1 source +├── .memory.lock +├── .memory.journal.json # exists only during/recovering a transaction +├── .recovery/ # transaction backups +└── agents/ + ├── __default__/ + │ └── facts/{sha256-prefix}/{fact_id}.md + └── {custom-agent}/ + ├── config.yaml # owned by the custom-agent subsystem + ├── SOUL.md + └── facts/{sha256-prefix}/{fact_id}.md +``` + +`memory.json` contains only: + +```json +{ + "version": "2.0", + "revision": 12, + "lastUpdated": "2026-07-19T00:00:00Z", + "user": {}, + "history": {} +} +``` + +It never contains facts, fact paths, hashes, embeddings, or a fact manifest. + +`sha256-prefix` is the first two hexadecimal characters of `SHA-256(fact_id)`, giving a deterministic 256-way shard that also distributes generated `fact_*` IDs. + +## 4. Scope rules + +The storage scope in this PR is: + +```text +user_id + agent_name +``` + +- `thread_id` is source metadata only. +- Omitted `agent_name` resolves inside DeerMem to the reserved `__default__` bucket. +- `__default__` is outside the public custom-agent grammar. +- Public agent identifiers are case-insensitive and canonicalized to lowercase. +- A custom agent named `lead-agent` is distinct from `__default__`. +- The custom-agent delete route must require `config.yaml`; a memory-only directory is preserved. + +## 5. Fact document + +Each Markdown file has YAML front matter plus one human-readable body: + +```markdown +--- +id: fact_ab12 +schemaVersion: 2 +category: constraint +topics: [python, runtime] +confidence: 0.95 +status: active +user_id: alice +agent_name: research-agent +source: + type: conversation + threadId: thread_123 +createdAt: 2026-07-19T00:00:00Z +updatedAt: 2026-07-19T00:00:00Z +revision: 1 +consolidatedFrom: [] +--- + +# Runtime constraint + +Project uses Python 3.12. +``` + +Storage validates IDs, content, category, confidence, timestamps, lifecycle status, scope, source, revision, and consolidation metadata before committing. + +## 6. Compatibility boundary + +Internally, `source` is structured metadata. Public MemoryManager/API documents preserve the historical string field: + +```text +{type: conversation, threadId: thread_123} -> "thread_123" +{type: manual, threadId: null} -> "manual" +``` + +Manager/API reads materialize a compatibility `facts` array for the selected/default agent. The frontend never reads facts from `memory.json`. + +Storage-specific conflict and corruption errors are translated at the MemoryManager boundary. The Gateway maps conflicts to HTTP 409 and returns a stable, non-sensitive HTTP 500 for corruption. + +## 7. Repository contract + +The file backend exposes: + +- `get_fact` / `list_facts` +- `upsert_fact` / `delete_fact` +- `apply_changes` +- summary reads/updates +- migration +- index lifecycle and scoped search + +`apply_changes()` returns an explicit incomplete delta. A caller that promises a full compatibility document must reload after committing. + +Direct fact CRUD uses separate preconditions: + +- expected shared `memory.json` revision; +- expected target fact revision or expected absence. + +## 8. Concurrency model + +Every transaction holds an in-process scope lock and a cross-process user file lock. Fact files and the shared JSON are protected by a recoverable journal. + +Operations are divided by intent: + +### Point operations + +An update/delete of named facts depends only on addressed fact preconditions. It may explicitly rebase after a manifest conflict if all expected absence/revision checks still hold. + +### Snapshot-derived operations + +Clear, max-fact trimming, consolidation, and other collection-derived set operations depend on the complete fact snapshot. They must not replay an old delete/trim set against a newer manifest. + +On conflict they: + +1. reload the complete selected-agent document; +2. recompute the complete operation; +3. retry at most three times; +4. return a conflict if contention continues. + +This preserves Clear All/scoped-clear meaning and the `max_facts` invariant. + +Summary change sets are patches at the `user`/`history` child-key level: omitted siblings remain persisted. Import remains replacement-oriented by normalizing incoming sections against the complete empty compatibility schema before it calls storage. + +## 9. Cache model + +Every supported fact mutation advances and atomically replaces the shared `memory.json`. Cache validation uses `(mtime_ns, file_size, revision)` from that file. The persisted revision prevents a stale hit when a coarse-mtime filesystem reports identical metadata for same-size writes. + +The read path does not glob/stat every Markdown fact merely to validate a cache hit, so validation cost does not grow with the number of fact files. It reads the shared JSON revision on every check. Direct out-of-band edits to Markdown require an explicit `reload()` or restart. + +Inactive per-scope locks are weakly cached and may be garbage-collected. + +## 10. Migration + +The first normal default read detects legacy facts in `memory.json`, acquires the normal locks, and migrates them into `__default__`. User/history summaries are preserved and the JSON is rewritten without `facts`. Explicit and lazy migrations return their committed fact deltas through the call chain and notify a configured retrieval adapter after releasing storage locks. + +Clear All enumerates every agent while holding the user lock. Facts from any unread legacy per-agent JSON are migrated first without adopting legacy summaries that are about to be cleared, and the resulting canonical facts are then deleted with the rest of the bucket. This prevents both summary conflicts from blocking an explicit clear and a later read from resurrecting skipped facts. The immutable `.v1.bak` remains inactive and agent configuration files remain untouched. + +The v1-to-v2 migration is one-way for the running application because pre-PR code does not read Markdown facts. Operators must stop DeerFlow and create a filesystem snapshot or full backup of the configured storage root before upgrading a persistent deployment. Before the first destructive v2 write, storage atomically and durably retains every migrated JSON source as `{source_filename}.v1.bak`. An existing backup is never overwritten: if it differs from the source, or if the backup cannot be written, migration stops before changing v1 data. These local backups preserve pre-migration data only and do not replace the required full snapshot. + +An optional idempotent operator CLI supports preflight audits and proactive migration: + +```bash +cd backend +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run +PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users +``` + +Repeated `--user-id` and custom `--storage-path` are supported. The CLI is optional; lazy first-read migration remains the zero-administration path. + +Legacy per-agent JSON is deleted only after its summaries are safely adopted or confirmed identical. Conflicting non-empty summaries and different same-ID facts fail loudly and preserve the source. + +## 11. Fixed storage invariants + +This release has one canonical fact format and requires crash recovery: + +```text +fact format = Markdown +journal = enabled +``` + +These are implementation invariants, not public `DeerMemConfig` fields. If `fact_format` or `journal_enabled` is supplied in `backend_config`, it follows the normal unknown-key behavior: DeerMem logs a warning and ignores it. + +## 12. Acceptance criteria + +- Global JSON contains no facts or manifest. +- Default and custom-agent fact sets remain isolated. +- One fact mutation writes/notifies only addressed facts. +- Same-ID creates and stale same-fact writes fail. +- Snapshot-derived operations recompute after conflicts. +- Persisted fact count never exceeds `max_facts` after concurrent creates. +- Scoped clear either clears facts committed before its successful revision or returns conflict. +- Legacy global facts migrate through normal reads and the CLI. +- Public source remains a string. +- Cache validation does not scale with the number of fact files and includes the persisted revision. +- Every destructive v1 JSON source is durably backed up before migration. +- Tests cover Windows locking, POSIX directory sync, recovery, migration, concurrency, API compatibility, and portability.