mirror of https://github.com/bytedance/deer-flow.git synced 2026-08-15 01:08:53 +00:00

History

fix(memory): consolidated facts inherit expected_valid_days from sources (#4225 )

* fix(memory): consolidated facts inherit expected_valid_days from sources

Consolidation (#3996) and per-fact expected_valid_days (#4143) were both
authored by the same contributor but never connected: the consolidated
new_fact carried the newest source's createdAt but no expected_valid_days,
so _effective_fact_staleness_age fell back to the global staleness_age_days.
A merge of several 200-day-old stable facts (each evd=3650) would land with
no evd, read as a 90-day window, and re-enter the staleness candidate set on
the very next cycle - the merge discarded the lifetime signal of the
underlying information and contradicted consolidation's premise (these are
stable, related facts worth synthesising).

Fix: the merged fact inherits expected_valid_days set so it is re-reviewed at
the EARLIEST source review deadline (min(createdAt + expected_valid_days)
across sources, relative to the merged fact's createdAt = the newest source's).
A merge combines details from every source, so a volatile sub-detail (evd=7)
must not inherit a stable source's 3650-day window and escape staleness review
for years - staleness KEEP/REMOVE is the only path that re-validates a merged
fact, so biasing toward the soonest deadline keeps uncertain merges re-checked
sooner. A source already past its deadline yields a minimal positive window
(review next cycle) rather than the global fallback, which would defer an
overdue review. Capped at the creation-time staleness_max_lifetime_multiplier
like any new fact. Omitted when no source carries a valid evd (legacy facts
fall back to the global age at read time, matching pre-feature behaviour).

DRY: extract _read_expected_valid_days(fact) -> int | None, the shared type
rule (int/float, reject bool, coerce to int BEFORE the > 0 guard) previously
inlined in four places - _normalize_memory_update_fact,
_effective_fact_staleness_age, the newFact creation cap, and consolidation
inheritance. All four call the single helper. Coercing before the guard
matters for values in (0, 1): 0.5 passes a raw > 0 check but truncates to 0,
which would violate the helper's "positive int or None" contract; the order
now matches the original _normalize_memory_update_fact rule.

No prompt/schema change: consolidation's prompt does not surface source evd to
the LLM, so asking the model to assign a merged lifetime would be guessing
without signal. Source inheritance is deterministic and always available.

Tests: consolidation evd cases now use time-stable createdAt (relative to now
via a _days_ago helper) covering - earliest-deadline selection, creation-cap
clamp, omit when no source evd, volatile source governs the deadline (and
re-enters staleness next cycle), overdue source clamps to a minimal window,
float coercion. Plus TestReadExpectedValidDays / TestEffectiveFactStalenessAge
regression cases for the (0, 1) coercion-order fix.

* fix(memory): reject non-finite expected_valid_days before int coercion

The shared _read_expected_valid_days helper (introduced when consolidating the
evd type rule across four call sites) coerces with int(raw) before the > 0
guard - reversing the original _normalize_memory_update_fact order so that a
fractional 0.5 does not leak as 0. But int(raw) raises for non-finite floats:
int(nan) raises ValueError and int(inf)/int(-inf) raise OverflowError. Python's
JSON decoder accepts NaN / Infinity as floats by default, so a single malformed
expected_valid_days in a hand-edited memory.json would abort staleness selection
or consolidation instead of falling back to the global lifetime.

On main, _effective_fact_staleness_age checked raw > 0 first, so NaN fell back
safely (nan > 0 is false) - but inf did NOT (inf > 0 is true, so main also
crashed on inf). This helper is now the persisted-fact read path for both
staleness and consolidation, so the regression (and the pre-existing inf crash)
must be closed here.

Fix: require math.isfinite(float(raw)) before the int() coercion, then keep
the existing positivity check and fallback. NaN / +/-inf all return None, so
callers fall back to the global staleness_age_days. Normal int/float values
(including large ones) are unaffected - isfinite is a no-op for them.

Tests:
- TestReadExpectedValidDays.test_rejects_non_finite_values - NaN, inf, -inf
  return None (not raise).
- TestEffectiveFactStalenessAge.test_falls_back_for_non_finite_values - the
  persisted-fact read path returns the global age for each, no raise.
- test_consolidation_with_non_finite_source_evd_does_not_raise - end-to-end:
  a NaN-evd source merged with a stable source does not abort consolidation;
  the NaN source's effective lifetime falls back to the global 90 and its
  deadline participates in the earliest-deadline computation.

* test(memory): hoist _select_stale_candidates import + tidy deadline docstring

Two review nits from the latest pass:

- `_select_stale_candidates` was imported inline inside three test methods;
  hoisted to the module-level import block so the dependency is declared once.
- `test_consolidated_evd_volatile_source_with_equal_created_at_future_deadline`
  had an abandoned calculation in its comment ("3 + 7 = 10 ... minus 3 already
  elapsed = 7? No:") that could mislead future readers into thinking
  elapsed-since-creation factors into the inherited window. Collapsed to a
  single clear line stating the window is relative to the merged createdAt,
  regardless of the source's current age.

* fix(memory): reject huge-int expected_valid_days above timedelta.max.days

_read_expected_valid_days routed every numeric value through float(raw) for the
math.isfinite guard, but Python's JSON decoder parses an integer literal with
no decimal point as an arbitrary-precision int (unlike 1e400, which decodes to
float inf). So a hand-edited memory.json carrying "expected_valid_days": 10**400
raised OverflowError in float(raw) before math.isfinite was ever called -
exactly the malformed-field-aborts-everything scenario the helper's docstring
claims to prevent.

The earlier non-finite fix only closed the float cases (NaN / +/-inf / 1e400).
A huge int below the float limit but above timedelta.max.days (e.g. 10**12)
would pass the helper and raise OverflowError downstream in
timedelta(days=evd) during staleness selection or consolidation - the same
crash fancyboi999 flagged for extend_by_days, just reached via a stored evd.

Fix: branch on type so an int never passes through float() (matching the
reviewer's suggestion), AND cap the returned int at timedelta.max.days
(999999999) so the downstream timedelta(days=evd) call cannot overflow either.
The float branch keeps the isfinite + int() coercion. Both branches share the
0 < evd <= timedelta.max.days positivity/range check.

Normal values are unaffected - any legitimate expected_valid_days is far below
the cap (the config ceiling staleness_max_extension_days tops out at 36500).

Tests (all three layers):
- TestReadExpectedValidDays.test_rejects_huge_int_above_timedelta_max - 10**400,
  10**12, 10**9, timedelta.max.days+1 return None; timedelta.max.days itself
  is accepted.
- TestEffectiveFactStalenessAge.test_falls_back_for_huge_int_above_timedelta_max
  - the persisted-fact read path returns the global age, no raise.
- test_consolidation_with_huge_int_source_evd_does_not_raise[1e400|1e12|1e9] -
  parametrized end-to-end: a huge-int-evd source merged with a stable source
  does not abort consolidation; the bad source falls back to the global 90.

* fix(memory): guard datetime arithmetic, not just timedelta construction

The huge-int fix capped _read_expected_valid_days at timedelta.max.days, but
that only proves timedelta(days=evd) can be constructed - adding it to a real
fact timestamp still overflows datetime.max. @fancyboi999 reproduced it: a
source with expected_valid_days=timedelta.max.days raises
"OverflowError: date value out of range" at dt + timedelta(...) in the new
consolidation deadline calculation. capping at timedelta.max.days was another
patch chasing the next overflow boundary, not a real close.

Root cause: the helper was doing datetime-range validation, but the safe bound
depends on the datetime the evd is added to, not on the evd alone. So the
responsibility moves to the arithmetic site, with try/except as the terminal
guard - no concrete upper bound to be wrong about.

Changes:
- _read_expected_valid_days returns any positive int (huge ints included, not
  routed through float). Its job is type/positivity validation only.
- New _safe_add_days(dt, days) -> datetime | None wraps dt + timedelta(days),
  returning None on OverflowError/ValueError. This is the terminal guard -
  there is no further boundary to overflow because try/except catches any
  datetime-range failure regardless of magnitude.
- _select_stale_candidates uses _safe_add_days(now, -effective_age); a None
  result means the window is unrepresentably large, so the fact cannot yet be
  stale and is skipped (not selected).
- Consolidation computes each source's deadline via _safe_add_days; a source
  whose deadline overflows falls back to the global staleness_age_days deadline
  (same treatment as a legacy no-evd source), so one malformed field cannot
  abort the merge.

Normal values are unaffected - any legitimate expected_valid_days is far below
the overflow boundary (the config ceiling staleness_max_extension_days tops
out at 36500).

Tests:
- TestSafeAddDays: normal/negative shifts; 10**400/10**12/10**9 return None;
  timedelta.max.days (the exact reproduced value) returns None, not raises.
- TestSelectStaleCandidates.test_huge_evd_does_not_abort_selection: a fact with
  a huge evd is skipped, not selected, and selection does not raise.
- test_consolidation_with_huge_int_source_evd_does_not_raise now parametrized
  over [1e400, 1e12, 1e9, timedelta.max.days] - the last is the value that
  constructs a valid timedelta but overflows datetime arithmetic.
- Helper/read-path tests updated to assert huge ints are returned as-is (the
  overflow guard is no longer in the helper).

2026-07-21 09:36:09 +08:00

deermem

fix(memory): consolidated facts inherit expected_valid_days from sources (#4225 )

2026-07-21 09:36:09 +08:00

noop

fix(memory): flush memory queue on graceful shutdown to prevent loss (#4181 )

2026-07-15 20:25:41 +08:00

__init__.py

feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122 )

2026-07-15 11:21:04 +08:00

README.md

feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122 )

2026-07-15 11:21:04 +08:00

README.md

Memory Backends

Each subfolder under agents/memory/backends/ is a pluggable memory backend. Swap the active one by changing one line in config.yaml - no deer-flow core changes required.

deermem/ - the default backend (deer-flow's own: structured facts + JSON storage).
noop/ - an empty backend and the template to copy when adding a new one.

This guide tells you which files to touch when you change, swap, or add a memory system. Paths are relative to backend/ unless noted.

Add a New Backend
Switch the Active Backend
Backend Contract
Do Not Modify
Common Pitfalls
Reference

Add a New Backend

Copy noop/ to backends/<yourname>/ and edit three files in this folder plus two outside it.

File	What to change
`backends/<yourname>/config.py`	Declare your config fields + `from_backend_config` (parse `backend_config`; read `storage_path` from it - do not import deer-flow path helpers)
`backends/<yourname>/<yourname>_manager.py`	Rename the class; parse config in `__init__`; implement the 9 ABC methods; optionally implement fact CRUD (see Backend Contract)
`backends/<yourname>/__init__.py`	`MANAGER_CLASS = YourManager` (relative import)
`config.yaml` (repo root, parent of `backend/`)	`memory.manager_class: <yourname>` + your knobs under `memory.backend_config`
`packages/harness/pyproject.toml`	Only if the backend needs external libs: declare the dependency; add `[tool.uv.sources]` for vendored source. Otherwise `uv sync` purges it (see Common Pitfalls)

See the docstring at the top of noop/noop_manager.py for the full 6-step walkthrough.

Switch the Active Backend

Edit config.yaml (repo root) only:

memory:
  manager_class: <name>        # deermem / noop / <yourname>
  backend_config: { ... }      # that backend's private config

Then restart deer-flow - the memory manager is a process-level singleton; a running process does not hot-reload config or backend code.

Backend Contract

1. The 9 ABC methods

Implement every method on MemoryManager in packages/harness/deerflow/agents/memory/manager.py. Signatures must match (parameter names, keyword-only args). noop is the empty-implementation reference.

2. Return shape (critical, easy to get wrong)

get_memory / export_memory / clear_memory / import_memory return a dict that the gateway casts to the DeerMem shape (MemoryResponse: version / lastUpdated / user / history / facts[]). Your backend must return a dict this shape accepts, or:

the data is silently dropped (pydantic ignores unknown fields);
the frontend gets empty defaults and lastUpdated="" crashes the date formatter.

A non-DeerMem backend maps its native records (e.g. {"results": [...]}) into this shape via a small adapter helper.

3. Optional capabilities (DeerMem-internal, not on the ABC)

The gateway probes these with hasattr(manager, "<name>") and returns 501 when absent:

create_fact / delete_fact / update_fact - the frontend's add/delete/edit-fact buttons. Signatures are in the commented block at the bottom of noop/noop_manager.py.
reload_memory - the frontend's reload button (delegate to get_memory if your backend has no cache).
warm - one-time warm-up at gateway startup (skipped if absent).

Implement the ones you support; leave the rest as 501.

4. Portability (the golden rule)

Important

A backend talks to the host through exactly two channels: (1) the ABC method arguments (manager.py), and (2) the backend_config dict. The only from deerflow import allowed anywhere in your backend folder is the ABC contract line in <name>_manager.py:

from deerflow.agents.memory.manager import MemoryManager

Change that one line (and only that line) to port the backend to another agent. Do not import deer-flow path helpers, config singletons, or models - get storage_path and everything else from backend_config.

5. What the host injects into `backend_config`

The factory (manager.py::get_memory_manager) injects these for every backend:

storage_path (str) - a writable state dir (the host's runtime_home by default, or whatever config.yaml sets). Use this as your storage root.
tracing_callback (Callable | None) - trace your LLM calls (langfuse). Ignore if you don't trace.
should_keep_hidden_message (Callable | None) - filter hide_from_ui messages. Ignore if not relevant.
Plus whatever the user puts under config.yaml::memory.backend_config (your backend's own knobs).

Do Not Modify

These are backend-agnostic. Don't touch them when swapping backends (unless you're changing the shared contract, which affects every backend):

File	Role
`packages/harness/deerflow/agents/memory/manager.py`	ABC + factory + scanner
`packages/harness/deerflow/agents/middlewares/memory_middleware.py`	`after_agent` -> `manager.add`
`packages/harness/deerflow/agents/memory/summarization_hook.py`	summarization -> `manager.add_nowait`
`packages/harness/deerflow/agents/lead_agent/prompt.py`	`_get_memory_context` -> `manager.get_context`
`app/gateway/routers/memory.py`	HTTP endpoints -> `manager.*` (hasattr-probed)
`packages/harness/deerflow/config/memory_config.py`	shared 4 fields (`enabled` / `injection_enabled` / `manager_class` / `backend_config`)
`frontend/src/components/workspace/settings/memory-settings-page.tsx`	frontend memory page (assumes DeerMem shape)

Note

The gateway and frontend are currently hard-coded to the DeerMem shape - that's why backends must return DeerMem-shape data (contract #2). Making them fully backend-agnostic is a larger refactor; see E:\deerflow\memory\plugin\00-插件兼容性矩阵.md.

Common Pitfalls

Lessons from integrating external backends:

External deps must be declared in pyproject.toml. A bare uv pip install is purged on the next uv sync / langgraph dev. Declare the dep (and [tool.uv.sources] for vendored source).
Return the DeerMem shape. Otherwise the frontend crashes with Invalid time value and your data is silently dropped. Build a small adapter helper to map your native records into it.
Fact CRUD returns 501 if not implemented. The frontend's delete-fact button reports Operation 'delete fact' not supported. Implement delete_fact (and friends) to fix it.
Don't import runtime_home. Read storage_path from backend_config. (The noop template shows the correct pattern; importing deer-flow path helpers breaks portability - contract #4.)
Restart deer-flow after changes. The manager is a process-level singleton; a running process does not hot-reload config or backend code.
Cap get_context length yourself. The host applies no token budget; the backend must truncate (DeerMem has max_injection_tokens; noop does not).

Reference

Template: noop/ - empty implementation with full docstrings; copy and go.
Design proposal: E:\deerflow\memory\记忆系统方案.md.
Plugin plans + compatibility matrix: E:\deerflow\memory\plugin\ (00-插件兼容性矩阵.md is the spine; defines the 9 shared contracts S1-S9).

README.md

Memory Backends

Table of Contents

Add a New Backend

Switch the Active Backend

Backend Contract

1. The 9 ABC methods

2. Return shape (critical, easy to get wrong)

3. Optional capabilities (DeerMem-internal, not on the ABC)

4. Portability (the golden rule)

5. What the host injects into backend_config

Do Not Modify

Common Pitfalls

Reference

5. What the host injects into `backend_config`