deer-flow/backend/packages/harness/deerflow/agents/memory/AGENTS.md

Memory System (packages/harness/deerflow/agents/memory/)

Components:

• updater.py - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication, optimistic revision checks, and repository change sets
• queue.py - Debounced update queue (per-thread deduplication, configurable wait time); captures user_id at enqueue time so it survives the threading.Timer boundary
• prompt.py - Prompt templates for memory updates
• storage.py - File repository with one user-global summary JSON, agent-owned single-fact Markdown, target-only journaled changes, strict fact validation, shared-user plus per-fact optimistic revisions, lock-protected migration, deep-copy caching, and a RetrievalPort adapter boundary
• retrieval.py - Built-in scope-aware SQLite FTS5/BM25 adapter; it stores only rebuildable derived data and can be disabled with an empty retrieval_adapter. Chinese jieba tokenization is optional via the backend memory-zh extra; without it the adapter uses SQLite unicode tokenization and the substring fallback. A corrupt persistent derived database is deleted and recreated once before falling back to substring retrieval. The Gateway closes the derived SQLite connection after its shutdown flush; reads and writes remain serialized by the adapter lock, with connection pooling deferred as a performance follow-up.
• tools.py - Tool-driven memory mode (memory_search, memory_add, memory_update, memory_delete) using the same storage/update primitives

Per-User Isolation:

• Memory is stored per-user at {base_dir}/users/{user_id}/memory.json
• Per-agent facts at {base_dir}/users/{user_id}/agents/{agent_name}/facts/{sha256-prefix}/{fact-id}.md, where the prefix is the first two hexadecimal characters of SHA-256(fact_id); there is no per-agent memory.json
• Custom agent definitions (SOUL.md + config.yaml) are also per-user at {base_dir}/users/{user_id}/agents/{agent_name}/. The legacy shared layout {base_dir}/agents/{agent_name}/ remains read-only fallback for unmigrated installations
• Middleware mode captures user_id via resolve_runtime_user_id(runtime) at enqueue time; tool mode resolves user_id and agent_name from ToolRuntime.context via the same helper so both Gateway and standalone LangGraph Server runs stay scoped to the authenticated user and active custom agent
• The /api/memory* endpoints resolve the owner through _resolve_memory_user_id(request): trusted internal callers (IM channel workers carrying the X-DeerFlow-Owner-User-Id header, e.g. a bound /memory command) act for the connection owner; browser/API callers fall back to get_effective_user_id(). The header is only honored after AuthMiddleware validated the internal token, mirroring get_trusted_internal_owner_user_id used by the threads router
• In no-auth mode, user_id defaults to "default" (constant DEFAULT_USER_ID)
• Absolute storage_path in config opts out of per-user isolation
• Migration: Run PYTHONPATH=. python scripts/migrate_user_isolation.py to move legacy memory.json, threads/, and agents/ into per-user layout. Supports --dry-run (preview changes) and --user-id USER_ID (assign unowned legacy data to a user, defaults to default).

Data Structure:

• User Context: workContext, personalContext, topOfMind (1-3 sentence summaries)
• History: recentMonths, earlierContext, longTermBackground
• Global JSON: {base_dir}/users/{user_id}/memory.json stores only version, shared revision/time, user, and history; it never stores facts or a fact index
• Facts: Schema-v2 Markdown documents under agents/{agent_name}/facts/{sha256-prefix}/{fact-id}.md; YAML front matter contains structure and the body contains the atomic fact
• Default agent compatibility: DeerMem resolves an omitted agent_name to the reserved __default__ fact bucket at the manager boundary. The sentinel is accepted only by DeerMem storage and is outside the custom-agent name grammar, so a real custom lead-agent remains isolated. Public agent identifiers are case-insensitive and canonicalized to lowercase before storage
• Compatibility view: direct global storage reads return facts: [], while DeerMem Manager/API reads select the explicit agent or reserved default and return its facts, so existing Settings and embedded-client schemas remain stable. Markdown keeps structured source metadata internally; the manager projects it to the historical string field before returning a public document
• Incremental result contract: FileMemoryStorage.apply_changes() returns complete: false plus upsertedFacts/deletedFactIds; it never presents a partial cache as a complete memory document. Public compatibility callers explicitly reload a fresh complete view only where their response contract requires it, including after successful disjoint-create rebases
• Repository: get/list/upsert/delete_fact, apply_changes, summary operations, migration, index lifecycle/status, and scoped search. apply_changes and direct fact CRUD touch only target Markdown files; direct fact CRUD accepts separate expected user-memory and fact revisions. Supplied summary child keys merge over their persisted section, while import normalizes complete replacement sections first. Whole-document load/save remains for compatibility but validates the complete facts list and diffs it before persistence. An unscoped manager clear first migrates facts from unread legacy agent JSON without adopting potentially conflicting summaries, then removes the global summaries and every agent's canonical facts while preserving agent configuration; an explicit agent clear removes only that bucket's facts and preserves the shared summaries

Workflow:

• memory.mode: middleware (default) keeps the passive path: MemoryMiddleware filters messages (user inputs + final AI responses), captures user_id via resolve_runtime_user_id(runtime), queues conversation with the captured user_id, and the debounced background thread invokes the LLM to extract context updates and facts using the stored user_id. DynamicContextMiddleware passes the same resolved identity to the memory read path. Both ordinary and bootstrap custom-agent construction pass agent_name into the middleware factory, keeping setup facts in the custom agent's bucket instead of __default__. On standalone Agent Server runs, server-owned auth identity is also resolved during lead-agent construction, normalized through make_safe_user_id for DeerFlow storage, and explicitly reused for custom-agent config/SOUL, user skills, skill policy, and prompt assembly; ordinary client user_id values cannot override langgraph_auth_user_id. On the embedded Gateway path, inject_authenticated_user_context removes client-supplied langgraph_auth_user / langgraph_auth_user_id from both RunnableConfig sections before graph construction, so those reserved fields cannot impersonate Agent Server auth.
• The optional openviking backend under packages/harness/deerflow/agents/memory/backends/openviking/ is a remote-only adapter built on the maintained langchain-openviking package. Select it with memory.manager_class: openviking and keep memory.mode: middleware. It uses one OpenViking USER API key bound to the configured DeerFlow owner_user_id; another DeerFlow user is rejected before remote access. DeerFlow owns the existing recall/capture timing, fixed injection query and full-transcript suffix cursor. langchain-openviking owns SDK transport, message conversion, tool-call preservation, batching, partial-write progress and Session commits. One DeerFlow thread maps to one stable OpenViking Session, with the default or named agent represented as its actor peer. Bounded hash-only cursors live below {storage_path}/openviking/sessions/; session locks are weakly cached, async entrypoints offload synchronous SDK and file IO, and graceful shutdown drains active operations before closing the recorder-owned client. The recorder receives an explicit empty extra_headers mapping so ovcli.conf cannot add arbitrary transport headers. Do not reintroduce a backend-local HTTP client, explicitly configured trusted identity headers, root-key data access, or imports of the OpenViking embedded runtime. Multi-user provisioning, query-aware refresh policy and new lifecycle scheduling are separate changes, not part of this backend.
• The optional honcho backend under packages/harness/deerflow/agents/memory/backends/honcho/ is a remote-only HTTP adapter for user-model memory (RFC #1898's user-dimension option). Select with memory.manager_class: honcho, keep memory.mode: middleware (tool mode also supported — it implements search). It writes filtered turns as Honcho messages (no local LLM calls; Honcho's deriver builds representations server-side), resolves one workspace per user_id (workspace_overrides else workspace_prefix + collision-resistant sanitized id; missing user fails closed to no memory), offloads sync HTTP in its a* overrides via asyncio.to_thread, and tool mode retains passive writes via MemoryMiddleware, mirroring mem0. failure_policy.read: fail_closed rethrows recall failures; default is log-and-empty.
• Honcho configuration objects reject non-finite or non-positive timeout values and non-positive character budgets during construction, including direct dataclass construction, before an HTTP client can use them.
• memory.mode: tool skips MemoryMiddleware and registers memory_search, memory_add, memory_update, and memory_delete on the agent. The model decides when to search, add, update, or delete facts; this is opt-in/experimental and should not be described as better than middleware mode without eval evidence.
• Both modes share FileMemoryStorage, per-user/per-agent isolation, manual CRUD primitives, and the updater backend. Injection is mode-aware: middleware mode injects global user/history summaries plus the selected agent's facts, while tool mode injects only the global summaries and leaves every agent fact behind memory_search to avoid duplicating automatically injected and retrieval-returned context. memory.injection_enabled: false suppresses the complete block in either mode.
• Middleware extraction classifies proposed facts with extraction-only scope/durability/authority labels. _apply_updates accepts only user + durable + descriptive new/consolidated facts, accepts only wholly user-scoped summary prose with authority=descriptive, and rejects missing labels per item without aborting unrelated updates. Contradiction removals use object entries with id, scope, reason, and optional zero-based replacementFactIndex; task/project removals fail closed, and a paired removal runs only when the referenced replacement survives the scope/confidence gates, deduplication, and max-fact trim under another fact ID. The labels are not persisted, so no storage migration is required. Staleness removals retain their independent candidate/cap guardrails, while tool-mode CRUD remains outside this extraction gate. Custom memory.backend_config.prompts_dir templates (including per-agent overrides) must carry the same classification fields; an un-migrated template makes the fail-closed gate reject every extraction-driven write, observable only through rejected_by_scope_gate and the >60% fact-rejection warning.
• Capacity eviction is centralized in deermem/core/eviction.py for automatic extraction, manual/tool fact creation, and import. confidence remains the default policy. Opt-in hybrid-v1 uses bounded 0.65 confidence + 0.25 explicit-confirmation freshness + 0.10 query-access heat, with configurable half-lives and a bounded minimum correction reserve. Confirmation/access metadata is collected only when hybrid-v1 or shadow mode is active. The existing update LLM may return factsToReinforce, but _apply_updates updates lastConfirmedAt/confirmationCount only when deterministic message processing also detected reinforcement; a valid lastConfirmedAt also resets the staleness-review clock. That deterministic gate is batch-level: it matches a human message among the last six filtered messages in the current extraction batch, while the LLM-provided ID supplies fact binding without an independent signal-to-fact correspondence check. Duplicate extraction, prompt injection, and search alone never confirm. Only facts actually returned by DeerMem.search() increment the decaying usage sidecar; get_context() never does, and confidence-only capacity selection does not read the usage sidecar. Sidecars live under the agent .metadata/ directory so usage does not mutate canonical Markdown timestamps/revisions. Capacity audits are bounded and metadata-only, are written only after canonical persistence succeeds, and user delete/clear removes matching usage/audit data. Shadow mode computes hybrid disagreement while continuing to execute confidence-only.
• Middleware mode queue debounces (30s default), batches updates, and commits global summaries plus the selected/default agent's fact delta through a user-level lock, optimistic user-memory revisions, per-fact revisions, and a recoverable target-file journal. Only explicitly marked point operations may rebase a stale shared revision, and only while every addressed fact still satisfies its original absent/revision precondition. Snapshot-derived clear/trim/consolidation operations instead reload the complete document and recompute their intent on a manifest conflict, with a bounded retry. Typed manifest/fact conflict subclasses keep that decision independent of exception text, and same-ID creates and stale same-fact writes fail. Scope-lock objects are weakly cached so inactive users do not grow a process-lifetime map. Cache validation does not scale with the fact-file count: its token combines the shared JSON's (mtime_ns, size, revision), so the persisted revision invalidates stale caches even when a coarse-mtime filesystem reports identical metadata for same-size writes; direct out-of-band Markdown edits require reload(). Atomic replacement also syncs the parent directory on POSIX so the rename is durable. DeerMem translates private storage conflict/corruption exceptions to the backend-neutral MemoryManager contract; the Gateway maps them to HTTP 409 and a stable HTTP 500 response respectively. A normal default-manager read automatically migrates legacy facts from the global JSON into __default__; it also adopts the earlier implicit lead-agent fact bucket only when that directory has no custom-agent config.yaml, and rejects unexpected files instead of deleting them. The v1-to-v2 migration is one-way for the running application: operators must stop DeerFlow and snapshot the configured storage root before upgrade. Before any destructive v2 write, every migrated JSON source is durably retained as {manifest_filename}.v1.bak; a missing-write or mismatched existing backup aborts without modifying v1 data. Legacy per-agent JSON is deleted only after its non-empty summaries are safely adopted or confirmed identical; summary conflicts keep the source file and fail loudly.
• Proactive Markdown migration CLI: from backend/, run PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run to audit and omit --dry-run to migrate before serving traffic. Use repeated --user-id values when selecting exact original identities, especially standalone raw IDs containing @ or other characters that are normalized in directory names; --storage-path selects a non-default DeerMem root. The CLI reuses FileMemoryStorage.migrate, is idempotent, continues across per-user failures, and exits non-zero if any user fails. It is optional because the first normal read still performs the same migration automatically.
• retrieval_adapter owns indexing and retrieval. fts5 is the DeerMem default and uses a persistent derived SQLite index under .retrieval/; an empty value disables the adapter and selects substring_fallback. File storage sends upsert/remove notifications for normal writes and both explicit and lazy migrations after releasing durable storage locks, then delegates search. Gateway startup schedules DeerMem.warm_retrieval() as a background full rebuild so readiness is not delayed, while a first search lazily rebuilds its exact scope until warm-up completes. Individual malformed facts are logged and skipped without triggering repeated full scans; only a fatal adapter rebuild failure keeps lazy retry enabled. During shutdown, the Gateway waits at most one second for this derived rebuild and leaves the full configured timeout to the canonical memory flush; if the rebuild is still active, its adapter remains open until process exit. Adapter failures mark the scope dirty and fall back to canonical substring search until rebuilding succeeds. FileMemoryStorage owns and closes the adapter so higher layers do not reach into private storage state.
• Staleness pass (same LLM invocation as the regular updater, no extra API call): when staleness_review_enabled is true and at least staleness_min_candidates aged facts exist, _select_stale_candidates selects facts older than their individual review window (expected_valid_days, or the global staleness_age_days fallback) that are not in staleness_protected_categories (default: correction), surfaces them in the prompt with a valid:Nd annotation, and the LLM judges each as KEEP, REMOVE, or EXTEND. REMOVE entries go in staleFactsToRemove; EXTEND entries go in staleFactsToExtend with an extend_by_days value, which sets the fact's expected_valid_days to min(days_since_created + extend_by_days, staleness_max_extension_days). The LLM assigns expected_valid_days when creating a fact; it is clamped at write time to staleness_age_days × staleness_max_lifetime_multiplier (creation cap). _apply_updates enforces the guardrail unconditionally at apply time: it intersects both the removal and extension sets with _select_stale_candidates output before applying the per-cycle cap (staleness_max_removals_per_cycle), so protected and non-aged facts can never be targeted regardless of model behavior or the feature flag setting. Facts the LLM proposed for removal are excluded from extension even if the per-cycle cap prevented their actual deletion that cycle. Extensions use an absolute ceiling (staleness_max_extension_days) rather than the creation multiplier so a deliberate review decision can advance the window beyond the initial cap while preventing timedelta overflow from a malformed extend_by_days.
• Consolidation pass (same LLM invocation as the regular updater, no extra API call): when consolidation_enabled is true and at least one category holds consolidation_min_facts or more facts, _select_consolidation_candidates identifies fragmented categories and surfaces at most consolidation_max_groups_per_cycle of them (largest first) in the prompt. The LLM decides which groups to merge and proposes a synthesised fact per group. _apply_updates enforces guardrails: source IDs must exist and must not overlap across groups, group size is capped at consolidation_max_sources, the merged fact's confidence cannot exceed the source maximum, and facts below fact_confidence_threshold are not written. The merged fact carries the newest source's createdAt (so the staleness clock reflects the underlying information, not synthesis time) and inherits expected_valid_days set so the merged fact is re-reviewed at the earliest source review deadline (min(createdAt + effective_lifetime) across sources, where a source's effective lifetime is its expected_valid_days or the global staleness_age_days fallback for legacy facts without one - so a legacy source's default window is not swallowed by a long-lived sibling), relative to the merged createdAt, clamped to a minimal positive window if a source is already past its deadline, then capped at the creation-time staleness_max_lifetime_multiplier; this keeps a volatile or legacy sub-detail from inheriting a stable source's long window and escaping staleness review for years, while a merge of uniformly stable sources does not re-enter review prematurely.
• Next interaction injects selected facts + context into <memory> tags in the system prompt when injection_enabled is true.

Run-level memory identity:

• Every Gateway run with an effective hidden memory block hashes the exact HumanMessage.content, including the <memory> wrapper, and records one context:memory event through its run-scoped RunJournal. Later runs and checkpoint-based branches reuse the frozen message without reloading memory; goal continuations are deduplicated to one event per run.
• A first-run block is trusted only when it comes from DynamicContextMiddleware's current update. A reused block must have existed in the checkpoint before the run, and the Gateway strips dynamic-context markers from untrusted input so a caller cannot forge the identity event by reusing a known message ID.
• The production consumer is the existing debug/audit endpoint GET /api/threads/{thread_id}/runs/{run_id}/events?event_types=context:memory. Event content has exactly one field, content_sha256, which operators use to compare the effective memory identity across runs. The full memory text stays in checkpoint state and is not duplicated into run_events.

Token counting (packages/harness/deerflow/agents/memory/prompt.py):

• _count_tokens budgets the injection. In default tiktoken mode, the encoding is loaded lazily and cached.
• Failed tiktoken loads are cached with a timestamp. During the fixed cooldown (_TIKTOKEN_RETRY_COOLDOWN_S, 600s), callers fall back to char estimation immediately instead of re-triggering the blocking BPE download; after the cooldown, transient outages can self-heal without a restart.
• In-flight loads are cached as a LOADING sentinel so concurrent callers fall back instead of spawning more blocking threads.
• Set memory.token_counting: char to skip tiktoken entirely and use the network-free CJK-aware char estimate.

Focused regression coverage for the updater lives in backend/tests/test_memory_updater.py.

Configuration (config.yaml → memory):

• enabled / injection_enabled - Master switches
• mode - Operation mode: middleware (default passive background extraction) or tool (experimental model-driven memory tools). Modes are mutually exclusive.
• storage_path - DeerMem storage root; one global summary JSON lives under each user and Markdown facts remain under agent buckets
• storage_class - file or a dotted MemoryStorage class; invalid persistent backends fail fast
• strict_user_scope - Require user_id for all storage access (default false for no-auth/legacy compatibility)
• manifest_filename - User-global summary JSON filename (kept for configuration compatibility)
• file_lock_timeout_seconds - Scope-lock wait; Markdown facts and the recovery journal are required storage invariants rather than configurable modes
• retrieval_adapter - fts5 by default, empty to disable, or a dotted factory receiving DeerMemConfig and returning a retrieval-port implementation
• debounce_seconds - Wait time before processing (default: 30)
• shutdown_flush_timeout_seconds - Hard budget (seconds) reserved for draining the memory backend's pending-update buffer on Gateway graceful shutdown (default: 30; 1–300). Each pending item does one LLM call, so large IM batches may need more. The Gateway lifespan calls MemoryManager.shutdown_flush(timeout) after channels/scheduler stop and after waiting at most one additional second for the derived retrieval warm-up; the backend short-circuits on an idle buffer, so the host calls it unconditionally (no pending/processing gate). The retrieval wait does not reduce this canonical flush budget. The combined shutdown hooks, brief retrieval wait, flush budget, and scheduling margin must fit inside the pod's K8s terminationGracePeriodSeconds (gateway Helm chart default: 45s) or K8s SIGKILLs the drain mid-flight.
• model_name - LLM for updates (null = default model)
• max_facts / fact_confidence_threshold - Fact storage limits (100 / 0.7)
• fact_eviction_policy / fact_eviction_shadow_enabled - Capacity policy (confidence default; opt-in hybrid-v1) and non-enforcing hybrid comparison audit
• eviction_confidence_weight / eviction_confirmation_weight / eviction_access_weight - Hybrid weights (0.65 / 0.25 / 0.10; must sum to 1.0)
• eviction_confirmation_half_life_days / eviction_access_half_life_days - Confirmation and query-heat decay windows (90 / 30 days)
• eviction_correction_reserved_fraction / eviction_correction_reserved_max - Bounded minimum correction capacity (0.10 / 10; unused slots are released)
• eviction_audit_max_entries - Metadata-only capacity audit bound per user/agent scope (200; 0 disables)
• max_injection_tokens - Token limit for prompt injection (2000)
• token_counting - Token counting strategy for the injection budget: tiktoken (default, accurate but may download BPE data from a public endpoint on first use — can block for a long time in network-restricted environments, see issues #3402/#3429) or char (network-free CJK-aware char estimate, never touches tiktoken)
• staleness_review_enabled - Enable proactive staleness pruning of aged facts (default: true; only triggers when aged candidates exist)
• staleness_age_days - Age in days before a fact becomes a staleness candidate (default: 90; range: 30–365)
• staleness_min_candidates - Minimum aged candidates required to trigger a review cycle (default: 3; range: 1–50)
• staleness_max_removals_per_cycle - Maximum facts removed in a single cycle; lowest-confidence entries are kept when the LLM requests more (default: 10; range: 1–50)
• staleness_protected_categories - Fact categories that are never pruned by staleness review (default: ["correction"])
• staleness_max_lifetime_multiplier - Creation-time cap multiplier for a fact's LLM-assigned expected_valid_days: stored value is clamped to staleness_age_days × multiplier so the model cannot defer first review indefinitely (default: 20.0; range: 1.0–100.0). Default 20.0 (90 × 20 = 1800 d ≈ 5 years) is generous enough to support the very-stable prompt tier without needing multiple review cycles to escape the cap.
• staleness_max_extension_days - Absolute upper bound (in days) on expected_valid_days after a lifetime extension (staleFactsToExtend). Applied at write time as min(days_since + extend_by, staleness_max_extension_days). Uses an absolute ceiling rather than the multiplier because extensions are deliberate review decisions; prevents timedelta overflow and LLM misfire from permanently deferring a fact (default: 3650 = 10 years; range: 90–36500).
• consolidation_enabled - Enable memory consolidation (default: true; no extra API call — runs in the same LLM invocation as the normal memory update)
• consolidation_min_facts - Minimum facts in a category to trigger consolidation review (default: 8; range: 3–30)
• consolidation_max_groups_per_cycle - Maximum categories the LLM can merge in one cycle (default: 3; range: 1–10; also controls the LLM's prompt instruction)
• consolidation_max_sources - Maximum source facts per merge group; prevents over-merging (default: 8; range: 2–20)
• watermark_max_keys - Soft cap on the in-memory conversation-watermark cache (one entry per distinct thread/user/agent). A bounded LRU: when over capacity the least-recently-used entry is dropped, and a dropped key re-extracts one batch on that thread's next turn (same as a restart). Bounds memory in long-lived gateways handling many threads (default: 4096; 0 = unbounded)