github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-07-28 17:06:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,844 Commits 45 Branches 8 Tags
Commit Graph

6 Commits

Author SHA1 Message Date
lllyfff
8145d66a33
feat(memory): memory message processing (#4447) 
* feat(memory): signals-based update pipeline + always-on watermark/trivial filter

Refactor the DeerMem memory update pipeline (message_processing -> queue ->
updater) around a signals frozenset seam, replacing the
(filtered, correction_detected, reinforcement_detected) 3-tuple with
(filtered, signals: frozenset[str]) end to end.

message_processing:
- Externalize signal-detection patterns to YAML (message_patterns/*.yaml).
- Extend signals from correction/reinforcement to a 6-class set
  (correction/reinforcement/preference/identity/goal/decision); detect_signals
  returns a frozenset aligned with the fact category enum.
- Pure-acknowledgment turns ("ok"/"好的"/...) are always filtered out before
  enqueue (whole-message fullmatch), saving an extraction LLM call.

queue (core/queue.py):
- In-memory list + debounce timer, with flush_sync (graceful-shutdown drain
  that joins an in-flight worker under a hard timeout) and queue_max_depth
  backpressure (signal-bearing updates always admitted; QueueFull otherwise).
- Same-key updates coalesce with a signal union; per-batch success/fail summary.

updater (core/updater.py):
- head500+tail500 message truncation (replaces the 1000-char head chop).
- Always-on per-thread watermark: feed only messages added since the last
  extraction. The watermark is in-memory and is not advanced on failure, so a
  failed/lost update is re-fed on the next conversation turn.
- [MANUAL] prompt marker for user-authored facts (source.type="manual").
- Post-invoke extraction_callback (host-injected) emitting facts_extracted /
  facts_accepted / rejected_low_confidence; the host default logs metrics and
  flags >60% rejection.

Confidence filtering remains in _apply_updates (the existing
fact_confidence_threshold check); there is no separate write gate.
Consolidation stays opt-in (lossy). The ABC add/add_nowait signature is
unchanged, so the summarization flush hook and host are unaffected.

Tests: add test_message_processing_signals, test_updater_truncation,
test_updater_watermark; update queue/updater/consolidation/staleness/pluggable
tests for the signals seam.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): harden update pipeline per PR review

- Catch QueueFull in DeerMem.add/add_nowait so backpressure degrades to
  'update skipped' instead of propagating into after_agent /
  summarization_hook and breaking the agent run (peer middlewares
  self-guard; MemoryMiddleware was the lone exception). Emergency
  (add_nowait) always admits under backpressure -- its data cannot be
  re-fed next turn.
- Rewrite the watermark from index-based to content/identity-based
  (_message_identity + _feed_after_watermark) so it stays correct when
  summarization removes the conversation front -- an index watermark
  pointed at the wrong message and silently skipped un-extracted tail
  turns. The emergency flush bypasses the watermark (bypass_watermark on
  ConversationContext, threaded through update_memory) and coexists with
  (does not replace) a pending normal update, so a flush cannot drop a
  pending update's un-extracted tail.
- Populate facts_accepted / rejected_low_confidence inside _apply_updates
  at the real confidence-filter site (passed_threshold) instead of
  re-deriving the threshold in _finalize_update -- eliminates metric drift.
- Emit extraction metrics in a finally with an 'attempted' flag so
  exception failures (parse error, apply_changes raise after retry) are
  observable, not only the happy path.
- Re-detect signals on the post-watermark feed for the extraction hint so
  it no longer references turns the LLM cannot see; admission-time signals
  still drive backpressure.
- Move the post-batch reschedule inside the queue lock to close a
  non-atomic self._timer race with a concurrent add.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address follow-up review nits (LRU, metric name, docstring)

- Bound the in-memory watermark cache with a configurable LRU
  (watermark_max_keys, default 4096, 0=unbounded). A dropped key re-extracts
  one batch on that thread's next turn (the documented restart behavior), so
  eviction is safe and preserves the content-identity watermark's
  front-removal guarantee. Adds _watermark_get/_watermark_set helpers and a
  bounded-LRU regression test.
- Rename the extraction metric facts_accepted -> facts_passed_confidence so
  the name matches what the >60% rejection-rate warning assumes (a
  confidence-gate signal, not a persisted-fact count); drop the stale
  "historical semantics" justification. Brand-new callback, one consumer.
- Fix the stale test_message_processing_signals module docstring: the signals
  seam is already swapped to frozenset, and a stale stage-numbering prefix is
  removed.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-26 21:16:36 +08:00
Tianye Song
8511fa6aa3
fix(memory): consolidated facts inherit expected_valid_days from sources (#4225) 
* fix(memory): consolidated facts inherit expected_valid_days from sources

Consolidation (#3996) and per-fact expected_valid_days (#4143) were both
authored by the same contributor but never connected: the consolidated
new_fact carried the newest source's createdAt but no expected_valid_days,
so _effective_fact_staleness_age fell back to the global staleness_age_days.
A merge of several 200-day-old stable facts (each evd=3650) would land with
no evd, read as a 90-day window, and re-enter the staleness candidate set on
the very next cycle - the merge discarded the lifetime signal of the
underlying information and contradicted consolidation's premise (these are
stable, related facts worth synthesising).

Fix: the merged fact inherits expected_valid_days set so it is re-reviewed at
the EARLIEST source review deadline (min(createdAt + expected_valid_days)
across sources, relative to the merged fact's createdAt = the newest source's).
A merge combines details from every source, so a volatile sub-detail (evd=7)
must not inherit a stable source's 3650-day window and escape staleness review
for years - staleness KEEP/REMOVE is the only path that re-validates a merged
fact, so biasing toward the soonest deadline keeps uncertain merges re-checked
sooner. A source already past its deadline yields a minimal positive window
(review next cycle) rather than the global fallback, which would defer an
overdue review. Capped at the creation-time staleness_max_lifetime_multiplier
like any new fact. Omitted when no source carries a valid evd (legacy facts
fall back to the global age at read time, matching pre-feature behaviour).

DRY: extract _read_expected_valid_days(fact) -> int | None, the shared type
rule (int/float, reject bool, coerce to int BEFORE the > 0 guard) previously
inlined in four places - _normalize_memory_update_fact,
_effective_fact_staleness_age, the newFact creation cap, and consolidation
inheritance. All four call the single helper. Coercing before the guard
matters for values in (0, 1): 0.5 passes a raw > 0 check but truncates to 0,
which would violate the helper's "positive int or None" contract; the order
now matches the original _normalize_memory_update_fact rule.

No prompt/schema change: consolidation's prompt does not surface source evd to
the LLM, so asking the model to assign a merged lifetime would be guessing
without signal. Source inheritance is deterministic and always available.

Tests: consolidation evd cases now use time-stable createdAt (relative to now
via a _days_ago helper) covering - earliest-deadline selection, creation-cap
clamp, omit when no source evd, volatile source governs the deadline (and
re-enters staleness next cycle), overdue source clamps to a minimal window,
float coercion. Plus TestReadExpectedValidDays / TestEffectiveFactStalenessAge
regression cases for the (0, 1) coercion-order fix.

* fix(memory): reject non-finite expected_valid_days before int coercion

The shared _read_expected_valid_days helper (introduced when consolidating the
evd type rule across four call sites) coerces with int(raw) before the > 0
guard - reversing the original _normalize_memory_update_fact order so that a
fractional 0.5 does not leak as 0. But int(raw) raises for non-finite floats:
int(nan) raises ValueError and int(inf)/int(-inf) raise OverflowError. Python's
JSON decoder accepts NaN / Infinity as floats by default, so a single malformed
expected_valid_days in a hand-edited memory.json would abort staleness selection
or consolidation instead of falling back to the global lifetime.

On main, _effective_fact_staleness_age checked raw > 0 first, so NaN fell back
safely (nan > 0 is false) - but inf did NOT (inf > 0 is true, so main also
crashed on inf). This helper is now the persisted-fact read path for both
staleness and consolidation, so the regression (and the pre-existing inf crash)
must be closed here.

Fix: require math.isfinite(float(raw)) before the int() coercion, then keep
the existing positivity check and fallback. NaN / +/-inf all return None, so
callers fall back to the global staleness_age_days. Normal int/float values
(including large ones) are unaffected - isfinite is a no-op for them.

Tests:
- TestReadExpectedValidDays.test_rejects_non_finite_values - NaN, inf, -inf
  return None (not raise).
- TestEffectiveFactStalenessAge.test_falls_back_for_non_finite_values - the
  persisted-fact read path returns the global age for each, no raise.
- test_consolidation_with_non_finite_source_evd_does_not_raise - end-to-end:
  a NaN-evd source merged with a stable source does not abort consolidation;
  the NaN source's effective lifetime falls back to the global 90 and its
  deadline participates in the earliest-deadline computation.

* test(memory): hoist _select_stale_candidates import + tidy deadline docstring

Two review nits from the latest pass:

- `_select_stale_candidates` was imported inline inside three test methods;
  hoisted to the module-level import block so the dependency is declared once.
- `test_consolidated_evd_volatile_source_with_equal_created_at_future_deadline`
  had an abandoned calculation in its comment ("3 + 7 = 10 ... minus 3 already
  elapsed = 7? No:") that could mislead future readers into thinking
  elapsed-since-creation factors into the inherited window. Collapsed to a
  single clear line stating the window is relative to the merged createdAt,
  regardless of the source's current age.

* fix(memory): reject huge-int expected_valid_days above timedelta.max.days

_read_expected_valid_days routed every numeric value through float(raw) for the
math.isfinite guard, but Python's JSON decoder parses an integer literal with
no decimal point as an arbitrary-precision int (unlike 1e400, which decodes to
float inf). So a hand-edited memory.json carrying "expected_valid_days": 10**400
raised OverflowError in float(raw) before math.isfinite was ever called -
exactly the malformed-field-aborts-everything scenario the helper's docstring
claims to prevent.

The earlier non-finite fix only closed the float cases (NaN / +/-inf / 1e400).
A huge int below the float limit but above timedelta.max.days (e.g. 10**12)
would pass the helper and raise OverflowError downstream in
timedelta(days=evd) during staleness selection or consolidation - the same
crash fancyboi999 flagged for extend_by_days, just reached via a stored evd.

Fix: branch on type so an int never passes through float() (matching the
reviewer's suggestion), AND cap the returned int at timedelta.max.days
(999999999) so the downstream timedelta(days=evd) call cannot overflow either.
The float branch keeps the isfinite + int() coercion. Both branches share the
0 < evd <= timedelta.max.days positivity/range check.

Normal values are unaffected - any legitimate expected_valid_days is far below
the cap (the config ceiling staleness_max_extension_days tops out at 36500).

Tests (all three layers):
- TestReadExpectedValidDays.test_rejects_huge_int_above_timedelta_max - 10**400,
  10**12, 10**9, timedelta.max.days+1 return None; timedelta.max.days itself
  is accepted.
- TestEffectiveFactStalenessAge.test_falls_back_for_huge_int_above_timedelta_max
  - the persisted-fact read path returns the global age, no raise.
- test_consolidation_with_huge_int_source_evd_does_not_raise[1e400|1e12|1e9] -
  parametrized end-to-end: a huge-int-evd source merged with a stable source
  does not abort consolidation; the bad source falls back to the global 90.

* fix(memory): guard datetime arithmetic, not just timedelta construction

The huge-int fix capped _read_expected_valid_days at timedelta.max.days, but
that only proves timedelta(days=evd) can be constructed - adding it to a real
fact timestamp still overflows datetime.max. @fancyboi999 reproduced it: a
source with expected_valid_days=timedelta.max.days raises
"OverflowError: date value out of range" at dt + timedelta(...) in the new
consolidation deadline calculation. capping at timedelta.max.days was another
patch chasing the next overflow boundary, not a real close.

Root cause: the helper was doing datetime-range validation, but the safe bound
depends on the datetime the evd is added to, not on the evd alone. So the
responsibility moves to the arithmetic site, with try/except as the terminal
guard - no concrete upper bound to be wrong about.

Changes:
- _read_expected_valid_days returns any positive int (huge ints included, not
  routed through float). Its job is type/positivity validation only.
- New _safe_add_days(dt, days) -> datetime | None wraps dt + timedelta(days),
  returning None on OverflowError/ValueError. This is the terminal guard -
  there is no further boundary to overflow because try/except catches any
  datetime-range failure regardless of magnitude.
- _select_stale_candidates uses _safe_add_days(now, -effective_age); a None
  result means the window is unrepresentably large, so the fact cannot yet be
  stale and is skipped (not selected).
- Consolidation computes each source's deadline via _safe_add_days; a source
  whose deadline overflows falls back to the global staleness_age_days deadline
  (same treatment as a legacy no-evd source), so one malformed field cannot
  abort the merge.

Normal values are unaffected - any legitimate expected_valid_days is far below
the overflow boundary (the config ceiling staleness_max_extension_days tops
out at 36500).

Tests:
- TestSafeAddDays: normal/negative shifts; 10**400/10**12/10**9 return None;
  timedelta.max.days (the exact reproduced value) returns None, not raises.
- TestSelectStaleCandidates.test_huge_evd_does_not_abort_selection: a fact with
  a huge evd is skipped, not selected, and selection does not raise.
- test_consolidation_with_huge_int_source_evd_does_not_raise now parametrized
  over [1e400, 1e12, 1e9, timedelta.max.days] - the last is the value that
  constructs a valid timedelta but overflows datetime arithmetic.
- Helper/read-path tests updated to assert huge ints are returned as-is (the
  overflow guard is no longer in the helper).
 2026-07-21 09:36:09 +08:00
lllyfff
de024646a7
feat(memory): memory template externalization (#4286) 
* feat(memory): template externalization — externalized prompts + signal patterns

Externalize two categories of hardcoded content in the DeerMem memory
backend so operators can customise them without touching Python code.

Prompt externalization (plugin 06):
- 4 memory extraction prompts moved from Python string constants to YAML
  files under core/prompts/ (consolidation / fact_extraction /
  memory_update.chat / staleness_review).
- load_prompt(name) and load_prompt_messages(name, variables) loaders.
- memory_update uses the chat format (system / user message split) for
  prompt-caching-friendly system prefixes; other prompts stay text.
- The extraction callback + per-agent prompt directories + Jinja2
  dependency are intentionally not included (minimal surface).
- Compatible with the upstream prompt additions from #4143
  (expected_valid_days, staleFactsToExtend, KEEP/REMOVE/EXTEND in
  staleness review).

Signal-pattern externalization (plugin 07, regex only):
- Correction and reinforcement detection patterns externalised to
  core/message_patterns/{correction,reinforcement}.yaml.
- load_patterns(name, patterns_dir) loader with bundled defaults.
- detect_correction / detect_reinforcement accept a keyword-only
  patterns= parameter; signatures are backward-compatible.
- patterns_dir config field added to DeerMemConfig.
- Importance scorer (importance.py, build_importance_scorer,
  _prepare_update changes) is NOT included — deferred to a later PR.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): fail loudly on invalid yaml in prompt/pattern loaders

Replace silent error handling in load_prompt / load_prompt_messages /
load_patterns so malformed yaml or missing required keys raise ValueError
with the file path rather than a raw YAMLError traceback or silent
empty-string / empty-list return.

Changes:
- load_prompt: YAMLError -> ValueError(path); missing/empty 'template'
  key -> ValueError
- load_prompt_messages: YAMLError -> ValueError(path); missing/empty
  'messages' key -> ValueError; KeyError from .format (placeholder
  mismatch) -> ValueError
- load_patterns: YAMLError -> ValueError (was silent warn+return []).
  OSError still degrades (permission, not format). Not-a-list yaml
  -> ValueError (was silent warn+return []).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): wire prompts_dir through updater, fail-loud patterns_dir, bump config_version

Addresses PR feedback on the template-externalization PR:

P1 — Wire prompts_dir into the DeerMem update path:
- Add prompts_dir field to DeerMemConfig (default None = bundled defaults).
- Thread it through DeerMem -> MemoryUpdater.__init__(prompts_dir=).
- _build_staleness_section / _build_consolidation_section accept
  prompts_dir= keyword and call load_prompt() instead of relying on
  module-level shim constants.
- _prepare_update_prompt passes prompts_dir to load_prompt_messages
  and to both section builders.
- Add _PROMPT_CACHE to load_prompt so repeated lookups (once per
  memory-update cycle) do not re-read yaml from disk.

P2 — Explicit patterns_dir must find its files:
- When patterns_dir is explicitly set and a YAML file is missing,
  load_patterns() raises FileNotFoundError instead of silently caching
  an empty list. Bundled defaults (patterns_dir=None) still log a
  WARNING and return [] for a missing bundled file (packaging bug).

P3 — Bump config_version and sync Helm:
- config.example.yaml: 26 -> 27
- deploy/helm/deer-flow/values.yaml: 26 -> 27
- deploy/helm/deer-flow/README.md: 26 -> 27
- scripts/check_config_version.sh confirms parity.

Tests: 224 passed (218 memory + 6 config_version).
Lint: ruff check + ruff format --check pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): cache load_prompt_messages, validate format fields, fail-loud load_patterns

Review feedback from willem-bd:

Cache for load_prompt_messages:
  Add _CHAT_TEMPLATE_CACHE + _render_messages() helper so the parsed chat
  templates are cached per (name, agent, prompts_dir). On cache hit only
  .format() rendering runs; the yaml file is read once per key.

Validate format field in both loaders:
  load_prompt rejects format='chat' (redirects to load_prompt_messages);
  load_prompt_messages rejects format='text' (redirects to load_prompt).
  Prevents operators from loading a chat yaml via the text loader (or vice
  versa) without a clear error.

Load_patterns fail-loud for explicit directories:
  - OSError (permission, etc.) now raises for explicit patterns_dir
    instead of silently disabling detection.
  - Malformed entries (missing/empty pattern key, wrong type) are skipped
    with a WARNING instead of silent skip.
  - Unknown flag names are warned instead of silently dropped.
  - re.error from compile() raises ValueError with file path and entry
    index instead of a bare re.error traceback.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): thread agent_name through section builders, validate explicit prompts at construction, bump config_version to 28

P1 — Thread agent_name through staleness/consolidation section builders:
  _build_staleness_section and _build_consolidation_section now accept
  agent_name= and pass it to load_prompt(), so per-agent prompt overrides
  work for section templates too (not just memory_update). Previously
  only prompts_dir was threaded; agent_name was ignored for section
  builders.

P1 — Validate explicit prompts at DeerMem construction:
  When DeerMemConfig.prompts_dir is explicitly set, DeerMem.__init__ now
  pre-loads all four prompt templates (staleness_review, consolidation,
  fact_extraction, memory_update with dummy variables) at construction
  time. A missing file, malformed YAML, or invalid placeholder raises
  immediately at startup instead of being caught by the updater's
  generic error handler and silently dropped as a failed update.

P2 — Advance config_version to 28:
  Upstream main already consumed the previous 26→27 bump with a
  different schema change. Bump config.example.yaml, Helm values.yaml,
  and Helm README.md to 28 to version this PR's patterns_dir and
  prompts_dir additions.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): render text templates at construction, propagate PromptConfigurationError

Keep prompt-configuration failures out of the recoverable update catch:

- Define PromptConfigurationError(ValueError) in prompt.py. Raised by
  load_prompt, load_prompt_messages, and _render_messages for bad yaml,
  missing keys, and invalid placeholders.

- _do_update_memory_sync_impl, update_memory's executor path, and
  _process_queue all re-raise (PromptConfigurationError, FileNotFoundError,
  OSError) before the generic except Exception, so a bad explicit prompt
  is never silently returned as False.

- DeerMem.__init__ prompt validation now renders text templates with
  dummy variables (.format(stale_facts="") etc.) so an unknown
  placeholder in staleness_review.yaml, consolidation.yaml, or
  fact_extraction.yaml is caught at construction.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): drop re-raise guards, soften validation comment, exclude dormant fact_extraction

- Remove (PromptConfigurationError, FileNotFoundError, OSError) re-raise
  guards from _do_update_memory_sync_impl, update_memory executor path,
  and _process_queue. The existing except Exception: logger.exception(...)
  already logs prompt-config errors at ERROR with full traceback; the
  re-raise was killing the entire batch and only surfacing via stderr.
  Per-agent override errors now log at ERROR per-item without aborting
  co-tenant updates.

- Soften the construction validation comment to clarify it only covers
  global templates. Per-agent overrides are validated lazily at first use
  and logged at ERROR by the updater's exception handler.

- Drop fact_extraction from construction validation (dormant prompt with
  no runtime caller; the shim + yaml remain for backward compat).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-20 07:24:34 +08:00
qin-chenghan
ad45f59d66
feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122) 
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>
 2026-07-15 11:21:04 +08:00
Tianye Song
54f3c43fe3
fix(security): html-escape fact content in memory prompt sections (#4028) 
* fix(security): html-escape fact content in memory prompt sections

Raw memory fact content was injected verbatim into prompt XML — a fact
containing a literal `"` could break the `"..."` delimiter, and a
closing tag like `</consolidation_candidates>` could prematurely end
the XML block, both potentially confusing the model.

Apply `html.escape()` to `content` in `_build_staleness_section` and
`_build_consolidation_section`, and to `cat` in the consolidation
section's XML attribute. Tests added for both sections covering special
characters, XML tag injection, and attribute injection.

Follow-up to #3996 as noted by reviewer willem-bd.

* fix(security): address reviewer follow-ups on html-escaping PR

- Escape `cat` in _build_staleness_section for symmetry with the
  consolidation section (both sections now consistently html-escape
  all LLM-derived category values that appear in the prompt)
- Add comment at current_memory=json.dumps() documenting the conscious
  accept: json.dumps leaves < > & unescaped; lower-risk than
  staleness/consolidation (read-only context, not delete/merge
  instructions); fix at fact-content insert time if revisited
- Add test for category escaping in the staleness section

* fix(security): reference tracking issue #4044 in conscious-accept comment

* style: compress conscious-accept comment to two lines
 2026-07-11 08:54:45 +08:00
Tianye Song
9097642658
feat(memory): add memory consolidation to synthesize fragmented facts (#3996) 
* feat(memory): add memory consolidation to synthesize fragmented facts

When a fact category accumulates many individual entries, the LLM
reviews them during the normal memory-update call (same invocation,
no extra API cost) and decides whether groups of related facts can be
synthesized into a single richer fact. This completes the memory
lifecycle: extraction → guaranteed injection → staleness review →
consolidation.

- Select fragmented categories by min-facts threshold, surface the most
  fragmented groups first; prompt-layer caps aligned with apply-layer
  guardrails so the LLM never sees groups it cannot act on
- Cap consolidated confidence at source maximum to prevent inflation;
  reject results below fact_confidence_threshold
- Double-consume protection prevents a fact from being merged into
  multiple consolidation targets
- Feature-gated at both prompt and apply time with per-cycle safety caps
- Add 26 tests covering candidate selection, normalization, apply
  guardrails, and prompt integration

* fix(memory): address consolidation correctness issues from PR review

Six fixes based on maintainer review of #3996:

1. Deduplicate sourceIds in normalization — ["f1","f1"] previously
   bypassed the ≥2-distinct-sources check; dict.fromkeys collapses it
   to ["f1"] which is correctly rejected.

2. Run consolidation after max_facts trim — previously, sources were
   deleted then the merged fact could be evicted by the trim, leaving
   no record of either. Moving consolidation last ensures source facts
   exist in the post-trim index before removal.

3. Fix count= attribute in consolidation prompt — advertised the full
   category size but listed only max_sources IDs; now uses
   min(len(group), max_sources) to match what the LLM can act on.

4. Exempt staleness_protected_categories from consolidation candidates
   — mirrors the existing staleness-review contract so correction facts
   are never surfaced for merging.

5. Strip and default category in consolidation normalization — "  " or
   " preference " are now normalised, matching _normalize_memory_update_fact.

6. Propagate sourceError from source facts into consolidated fact —
   correction context is no longer silently lost on merge.

* fix(memory): add apply-time guardrails and tests for consolidation

P1: mirror the staleness-pass defense-in-depth pattern — build
allowed_source_ids from _select_consolidation_candidates at apply time
so a protected-category or below-threshold fact proposed by the LLM is
rejected regardless of model behavior.

P2a: test that LLM-returned confidence is capped at max source confidence
and that a capped result below fact_confidence_threshold is rejected.

P2b: test that factsToConsolidate with consolidation_enabled=False is a
no-op at apply time (35 tests, all pass).

* fix(memory): address three correctness issues from second review round

1. Default consolidation_enabled=False — consolidation is lossy (source
   content is permanently replaced, only consolidatedFrom IDs preserved);
   new lossy features default to off. config.example.yaml updated to match.

2. Unify confidence coercion between prompt and apply — _build_consolidation_section
   now calls _coerce_source_confidence(fact) instead of an inline 0.0-default
   coercion, so a null-confidence fact renders with 0.50 in the LLM prompt and
   is capped at 0.50 at apply time (same value, same function).

3. Preserve staleness clock on merge — consolidated fact now carries the
   newest source's createdAt (not now) so aged information does not gain a
   fresh staleness-review window just by being consolidated; consolidatedAt
   is added as an explicit audit field.

Three regression tests added (default=false, null-confidence consistency,
createdAt policy); all guardrail tests now set consolidation_enabled=True
explicitly so they test the guardrail, not the feature flag. 38 tests pass.

* fix(memory): harden createdAt comparison and confidence handling

1. createdAt max via _parse_fact_datetime — replaces string max() which
   crashes on non-string createdAt (numeric unix timestamps) and sorts
   Z/+00:00 mixed formats incorrectly. Mirrors how staleness computes age.

2. Remove dead min(..., 1.0) — _coerce_source_confidence already clamps
   each source confidence to [0, 1], so max(source_confidences) ≤ 1.0
   by contract; the outer min could never bind.

3. Clamp raw_llm_conf to [0, 1] before applying the source cap — out-of-
   range values like 1.5 are safe today (pinned by the cap) but defensively
   clamped first so the invariant holds even if the cap is ever loosened.

4. Doc: expand the apply-time guardrails comment to call out the protected-
   category exclusion via allowed_source_ids — this is the central safety
   property ("explicit user feedback is never silently merged away").

5. Test: add test_confidence_fallback_to_max_source_when_llm_omits_field
   covering the else-branch (LLM omits confidence → uses max_source_conf).

6. Fix lint: reorder imports in test file (stdlib before third-party).

39 tests, all pass.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-07-10 11:31:08 +08:00