deer-flow/docs/plans/STORAGE_REWRITE_CHANGES.md

DeerMem Storage Rewrite: Implementation Guide

This document explains the final PR behavior for readers who are new to the codebase. It complements STORAGE_REWRITE_PLAN.md: the plan explains the intended contract; this file explains where the code changed and how a request travels through it.

1. What changed on disk

Before the rewrite, facts and summaries could live together in a coarse JSON document. The final layout is:

users/alice/
├── memory.json
├── memory.json.v1.bak
└── agents/
    ├── __default__/facts/30/fact_a.md
    └── research-agent/facts/ee/fact_b.md

memory.json owns project-independent summaries. Each Markdown file owns one atomic fact. No JSON fact index duplicates the Markdown repository.

2. Main files

deer_mem.py

This is the DeerMem implementation of the public MemoryManager plugin interface.

It:

• maps omitted agents to __default__;
• canonicalizes explicit agent names to lowercase;
• translates storage-private exceptions to public manager errors;
• converts structured internal source metadata to the legacy public string;
• reloads complete documents where the manager/API contract promises completeness.

Gateway code does not import DeerMem storage internals.

deermem/core/paths.py

This owns user, agent, and fact path validation. The internal __default__ sentinel is deliberately outside the public custom-agent regular expression.

Fact paths are sharded by the first two hexadecimal characters of SHA-256(fact_id). This gives deterministic direct lookup while distributing generated IDs that all begin with fact_.

deermem/core/storage.py

This owns canonical parsing/rendering, locks, revisions, journaling, recovery, migration, cache invalidation, fact repository operations, and retrieval notifications.

deermem/core/updater.py

This converts manual CRUD and LLM extraction results into repository change sets. It decides whether an operation is a point mutation or depends on a complete snapshot.

app/gateway/routers/memory.py

This keeps the HTTP schema compatible. It catches only backend-neutral MemoryManager exceptions and maps concurrency to 409 and corruption to a stable 500.

scripts/migrate_memory_markdown.py

This optional CLI lets operators preview or proactively run the same migration that normal first reads perform lazily.

3. Read trajectory

For an ordinary request with no explicit agent:

Gateway / middleware
  -> MemoryManager.get_memory(user_id, agent_name=None)
  -> DeerMem canonicalizes agent to __default__
  -> FileMemoryStorage.load(__default__)
  -> recover journal / migrate legacy data when needed
  -> compare memory.json (mtime_ns, file_size, revision) with cache
  -> cache hit, or parse selected agent Markdown files
  -> DeerMem converts source metadata to public strings
  -> Gateway validates and returns the compatibility document

For a custom agent, the same trajectory reads that lowercase agent bucket and combines it with the user's shared summaries.

4. Single-fact update trajectory

MemoryManager.update_fact(fact_id)
  -> storage.get_fact(fact_id)
  -> build one upsert with expected fact revision
  -> acquire in-process + cross-process locks
  -> recover an interrupted prior journal if present
  -> validate shared revision and target fact revision
  -> journal only memory.json + the addressed fact
  -> atomically replace the Markdown fact and memory.json
  -> fsync files and the POSIX parent directory
  -> notify retrieval for that fact only
  -> reload a complete compatibility response

Unchanged sibling facts are not backed up, rewritten, or re-indexed.

5. Why there are two revisions

The shared JSON revision protects the multi-file transaction. The fact revision protects one Markdown object.

They answer different questions:

manifest revision: Did anything in this user's memory change?
fact revision:     Is the exact fact I read still the same object version?

Direct upsert_fact and delete_fact therefore accept separate expected manifest and fact revisions.

6. Safe rebase versus fresh-snapshot retry

A point update to fact_a can safely move from manifest revision 10 to 11 if fact_a still has the expected revision. A scoped clear cannot: its meaning depends on every fact that existed at the successful commit.

The final implementation uses this rule:

point mutation + valid fact preconditions
    -> explicit bounded manifest rebase is allowed

clear / trim / consolidation / collection-derived set
    -> manifest conflict
    -> reload full document
    -> recompute complete operation
    -> bounded retry

This fixes two reviewed races:

• a fact created between a scoped-clear snapshot and commit no longer survives behind a successful “empty” response;
• two creators starting from 9/10 facts cannot commit 11 facts.

7. Clear semantics

The manager distinguishes two calls:

clear_memory(user_id=alice)
    clears shared user/history summaries and every agent fact bucket
    preserves custom-agent config.yaml and SOUL.md

clear_memory(user_id=alice, agent_name=research-agent)
    clears only research-agent facts
    preserves shared user/history summaries

The file backend's all-user clear holds the user lock while enumerating buckets. It first migrates facts from any unread legacy per-agent memory.json without adopting legacy summaries that are about to be cleared, then deletes the resulting canonical fact files; retained .v1.bak files are inactive migration evidence and are never read back, while custom-agent configuration remains untouched.

8. Source compatibility

On disk:

source:
  type: conversation
  threadId: thread_123

Public API:

{"source": "thread_123"}

Manual/import/consolidation sources similarly return their type string. The richer internal form is ready for retrieval metadata without breaking the frontend.

9. Cache behavior

The earlier cache signature scanned and statted every fact file on every load(). The final cache observes the shared memory.json only.

The token is (mtime_ns, file_size, revision). Every supported storage mutation replaces the JSON and increments its revision, so another process's write invalidates cached agent documents even when a coarse-mtime filesystem reports identical metadata for same-size writes. Validation reads one shared JSON and does not scale with the number of fact files. This may invalidate more agents than strictly necessary, but it avoids an O(n) fact-directory walk without introducing a second manifest.

Manual out-of-band Markdown edits are not part of the supported write API and require reload().

Per-scope in-process locks live in a weak-value dictionary, so inactive users do not permanently grow the lock map.

10. Migration trajectory

When a v1 memory.json still contains facts:

first load/reload
  -> detect legacy facts/version
  -> acquire normal locks
  -> durably retain every source as memory.json.v1.bak
  -> normalize facts into __default__ scope
  -> commit Markdown files and preserved summaries through the journal
  -> rewrite memory.json without facts
  -> release storage locks and notify the configured retrieval adapter
  -> continue the original read

The explicit CLI path emits the same retrieval upserts. Re-running an idempotent migration emits no duplicate notification because no fact is rewritten.

Operators may run:

cd backend
PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run
PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users

The command is idempotent, accepts repeated --user-id, supports --storage-path, continues reporting after one user fails, and exits non-zero if any user failed. It is not required for startup.

This is a one-way application migration: pre-PR code cannot read Markdown facts. Before upgrading a persistent deployment, stop DeerFlow and take a filesystem snapshot or full backup of the configured storage root. Storage also retains each destructive v1 JSON source beside its original path as {source_filename}.v1.bak before committing v2 data. Existing backups are immutable; a mismatch or backup-write failure aborts before changing the source. The local backup contains pre-migration data only and does not replace the full snapshot requirement.

11. Configuration changes

manifest_filename and lock timeout remain configurable. Markdown format and journaling are required invariants, not modes.

fact_format and journal_enabled are not DeerMemConfig fields. If supplied in backend_config, they follow the normal unknown-key behavior: DeerMem logs a warning and ignores them.

12. Agent naming

The Gateway already stores custom-agent names in lowercase. DeerMem now makes the same rule explicit at its manager boundary:

Lead-Agent == lead-agent == LEAD-AGENT

This gives Linux, macOS, and Windows the same behavior. A cross-layer contract test pins that both public naming patterns reject __default__, while storage may use the sentinel internally.

13. Failure behavior

• Stale same-fact write: conflict, no overwrite.
• Same-ID concurrent create: conflict, no conversion into update.
• Continuing collection contention: conflict after bounded retries.
• Corrupt JSON/Markdown: stable public corruption error; original files retained.
• Conflicting legacy summaries/facts: migration fails loudly and preserves the legacy source.
• Missing, unreadable, or mismatched persistent v1 backup: migration stops before changing the source.
• Retrieval adapter failure: persistence remains committed; the failure is logged/reported per fact.

14. Review-driven fixes retained in the implementation

• Default bucket changed from colliding lead-agent to reserved __default__.
• Custom-agent deletion requires a genuine config file.
• Legacy global facts migrate on first read and through an optional CLI.
• Public responses retain string source.
• Conflicts map to HTTP 409 through MemoryManager-neutral errors.
• Clear All and scoped clear have distinct semantics.
• Full index rebuild validates original email-style user IDs correctly.
• Lock cache is reclaimable.
• Manifest and fact revisions are separate.
• Retry dispatch uses exception subtypes, not message text.
• POSIX atomic replacement fsyncs the parent directory.
• Stale default-bucket test now asserts __default__.
• Snapshot-derived set operations recompute instead of replaying stale intent.
• Cache validation no longer scans every fact on every read.
• Cache tokens include the persisted revision to survive coarse-mtime same-size writes.
• Fact paths shard by SHA-256(fact_id)[:2] rather than the constant fa prefix.
• Destructive migration writes immutable .v1.bak sources before v2 data.
• Fixed storage invariants are no longer presented as configurable features.

15. Verification map

Primary coverage lives in:

• tests/test_memory_storage_markdown.py
• tests/test_deermem_self_contained.py
• tests/test_memory_router.py
• tests/test_memory_markdown_migration_cli.py
• tests/test_custom_agent.py

Tests cover layout, compatibility, incremental writes, concurrency, cap enforcement, clear semantics, corruption, migration, retrieval delegation, cache invalidation, path containment, portability, Windows lock behavior, and POSIX directory sync.