feat(memory): LLM-assigned per-fact expected_valid_days and staleFactsToExtend (#4143)

Re-ports this feature onto the pluggable-memory backend introduced in #4122
(the original #4143 was force-pushed clean by accident and auto-closed). The
#4122 refactor moved the staleness logic into the self-contained DeerMem
backend (backends/deermem/deermem/core/) and reverted it to the pre-feature
global-threshold version, so the per-fact lifetime work is re-applied here
against the new module layout + DI (MemoryUpdater is now (config, storage,
llm)-injected; config lives on DeerMemConfig, not host MemoryConfig).

**expected_valid_days (creation)**
The LLM assigns a per-fact review window when storing each new fact. The
prompt exposes five tiers (<=14 d transient -> >365 d very stable). The value
is capped at write time by staleness_age_days x staleness_max_lifetime_multiplier
(default 20.0 -> 1800 d ~= 5 years; range 1.0-100.0) so the model cannot set
an initial lifetime so long the fact is never re-evaluated. The default 20.0
makes the "> 365 d very stable" tier achievable out of the box (3.0 silently
clamped it to 270 d).

**staleFactsToExtend (review)**
During staleness review the LLM can emit extension entries for kept facts
whose window seems miscalibrated. new_evd = min(days_since_created +
extend_by_days, staleness_max_extension_days). Extensions use an absolute
ceiling (default 3650 d ~= 10 years; range 90-36500) rather than the creation
multiplier - they are deliberate review decisions that must be able to advance
the window beyond the initial cap, but the absolute bound prevents timedelta
overflow (a model-supplied extend_by_days of 10**9 previously crashed every
later candidate-selection pass with OverflowError) and LLM misfire.

**Invariant correctness**
- Read-time cap removed from _effective_fact_staleness_age; cap is write-time
  only so extensions actually advance the review window.
- proposed_remove_ids hoisted out of the removals sub-block and used to exclude
  from extension, so a cap-surviving proposed-removal fact is never extended.
- extend_by coerced to int before the > 0 guard (a fractional 0.9 would pass
  the float check then int() to 0, silently writing a zero-delta extension).
- days_since uses total_seconds() // 86400 (not .days truncation).
- staleness-section html.escape uses quote=False to match the prompt.py
  convention; only <, >, & break element-text structure.

**Tests**
test_memory_staleness_review.py was module-level skipped by #4122 ("full
unit-test migration is a follow-up"). This PR performs that migration: DI
construction via (DeerMemConfig, _FakeStorage), _build_staleness_section back
to the (candidates, config) signature, plus new coverage for per-fact
selection, EXTEND with the absolute cap, the overflow next-cycle regression,
the proposed-removal-not-extendable case, fractional extend_by skipping, and
the creation-time cap. 67 tests, all green.
This commit is contained in:
Tianye Song 2026-07-16 09:34:21 +08:00 committed by GitHub
parent 259f51ca4f
commit 8da7cbf028
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 884 additions and 323 deletions

View File

@ -580,7 +580,7 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_
- `memory.mode: tool` skips `MemoryMiddleware` and registers `memory_search`, `memory_add`, `memory_update`, and `memory_delete` on the agent. The model decides when to search, add, update, or delete facts; this is opt-in/experimental and should not be described as better than middleware mode without eval evidence.
- Both modes share `FileMemoryStorage`, per-user/per-agent isolation, prompt injection, manual CRUD primitives, and the updater backend.
- Middleware mode queue debounces (30s default), batches updates, deduplicates per-thread, applies updates atomically (temp file + rename) with cache invalidation, and skips duplicate fact content before append.
- Staleness pass (same LLM invocation as the regular updater, no extra API call): when `staleness_review_enabled` is `true` and at least `staleness_min_candidates` aged facts exist, `_select_stale_candidates` selects facts older than `staleness_age_days` that are not in `staleness_protected_categories` (default: `correction`), surfaces them in the prompt, and the LLM judges each as KEEP or REMOVE. `_apply_updates` enforces the guardrail unconditionally at apply time: it intersects the LLM-returned removal set with `_select_stale_candidates` output before applying the per-cycle cap (`staleness_max_removals_per_cycle`), so protected and non-aged facts can never be deleted regardless of model behavior or the feature flag setting.
- Staleness pass (same LLM invocation as the regular updater, no extra API call): when `staleness_review_enabled` is `true` and at least `staleness_min_candidates` aged facts exist, `_select_stale_candidates` selects facts older than their individual review window (`expected_valid_days`, or the global `staleness_age_days` fallback) that are not in `staleness_protected_categories` (default: `correction`), surfaces them in the prompt with a `valid:Nd` annotation, and the LLM judges each as KEEP, REMOVE, or EXTEND. REMOVE entries go in `staleFactsToRemove`; EXTEND entries go in `staleFactsToExtend` with an `extend_by_days` value, which sets the fact's `expected_valid_days` to `min(days_since_created + extend_by_days, staleness_max_extension_days)`. The LLM assigns `expected_valid_days` when creating a fact; it is clamped at write time to `staleness_age_days × staleness_max_lifetime_multiplier` (creation cap). `_apply_updates` enforces the guardrail unconditionally at apply time: it intersects both the removal and extension sets with `_select_stale_candidates` output before applying the per-cycle cap (`staleness_max_removals_per_cycle`), so protected and non-aged facts can never be targeted regardless of model behavior or the feature flag setting. Facts the LLM proposed for removal are excluded from extension even if the per-cycle cap prevented their actual deletion that cycle. Extensions use an absolute ceiling (`staleness_max_extension_days`) rather than the creation multiplier so a deliberate review decision can advance the window beyond the initial cap while preventing `timedelta` overflow from a malformed `extend_by_days`.
- Consolidation pass (same LLM invocation as the regular updater, no extra API call): when `consolidation_enabled` is `true` and at least one category holds `consolidation_min_facts` or more facts, `_select_consolidation_candidates` identifies fragmented categories and surfaces at most `consolidation_max_groups_per_cycle` of them (largest first) in the prompt. The LLM decides which groups to merge and proposes a synthesised fact per group. `_apply_updates` enforces guardrails: source IDs must exist and must not overlap across groups, group size is capped at `consolidation_max_sources`, the merged fact's confidence cannot exceed the source maximum, and facts below `fact_confidence_threshold` are not written.
- Next interaction injects selected facts + context into `<memory>` tags in the system prompt when `injection_enabled` is true.
@ -607,6 +607,8 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
- `staleness_min_candidates` - Minimum aged candidates required to trigger a review cycle (default: 3; range: 150)
- `staleness_max_removals_per_cycle` - Maximum facts removed in a single cycle; lowest-confidence entries are kept when the LLM requests more (default: 10; range: 150)
- `staleness_protected_categories` - Fact categories that are never pruned by staleness review (default: `["correction"]`)
- `staleness_max_lifetime_multiplier` - Creation-time cap multiplier for a fact's LLM-assigned `expected_valid_days`: stored value is clamped to `staleness_age_days × multiplier` so the model cannot defer first review indefinitely (default: 20.0; range: 1.0100.0). Default 20.0 (90 × 20 = 1800 d ≈ 5 years) is generous enough to support the very-stable prompt tier without needing multiple review cycles to escape the cap.
- `staleness_max_extension_days` - Absolute upper bound (in days) on `expected_valid_days` after a lifetime extension (`staleFactsToExtend`). Applied at write time as `min(days_since + extend_by, staleness_max_extension_days)`. Uses an absolute ceiling rather than the multiplier because extensions are deliberate review decisions; prevents `timedelta` overflow and LLM misfire from permanently deferring a fact (default: 3650 = 10 years; range: 9036500).
- `consolidation_enabled` - Enable memory consolidation (default: `true`; no extra API call — runs in the same LLM invocation as the normal memory update)
- `consolidation_min_facts` - Minimum facts in a category to trigger consolidation review (default: 8; range: 330)
- `consolidation_max_groups_per_cycle` - Maximum categories the LLM can merge in one cycle (default: 3; range: 110; also controls the LLM's prompt instruction)
@ -734,7 +736,7 @@ Config is env-driven like the others — `MonocleTracingConfig`, built in `get_t
- `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path)
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
- `subagents.enabled` - Master switch for subagent delegation
- `memory` - Memory system (enabled, storage_path, debounce_seconds, shutdown_flush_timeout_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens, staleness_review_enabled, staleness_age_days, staleness_min_candidates, staleness_max_removals_per_cycle, staleness_protected_categories)
- `memory` - Memory system (enabled, storage_path, debounce_seconds, shutdown_flush_timeout_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens, staleness_review_enabled, staleness_age_days, staleness_min_candidates, staleness_max_removals_per_cycle, staleness_protected_categories, staleness_max_lifetime_multiplier, staleness_max_extension_days)
**`extensions_config.json`**:
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description, `routing`, `tools`, `tool_call_timeout`). `routing.mode="prefer"` emits `<mcp_routing_hints>` prompt guidance; if `tool_search` defers the hinted tool, `McpRoutingMiddleware` can also auto-promote matching deferred schemas before the model call. It does not hard-disable other tools.

View File

@ -115,6 +115,39 @@ class DeerMemConfig(BaseModel):
default_factory=lambda: ["correction"],
description="Fact categories exempt from staleness review.",
)
staleness_max_lifetime_multiplier: float = Field(
default=20.0,
ge=1.0,
le=100.0,
description=(
"Creation-time cap multiplier for a fact's LLM-assigned "
"expected_valid_days. When a new fact is stored, its "
"expected_valid_days is clamped to "
"staleness_age_days * staleness_max_lifetime_multiplier so the "
"model cannot set an initial lifetime so long that the fact is "
"never re-evaluated. Default 20.0 (90 x 20 = 1800 d ~= 5 years) "
"is generous enough to support the 'very stable' prompt tier "
"(core skills, native language) without needing multiple review "
"cycles to escape the cap. Lifetime extensions (staleFactsToExtend) "
"are subject to staleness_max_extension_days instead."
),
)
staleness_max_extension_days: int = Field(
default=3650,
ge=90,
le=36500,
description=(
"Absolute upper bound (in days) on expected_valid_days after a "
"lifetime extension (staleFactsToExtend). Applied at write time "
"during staleness review: new_evd = min(days_since + extend_by, "
"staleness_max_extension_days). Separate from the creation-time "
"multiplier cap because extensions are deliberate recalibration "
"decisions and are not subject to the staleness_age_days scale. "
"The ceiling prevents a single LLM misfire from permanently "
"deferring a fact or causing timedelta overflow on the next "
"candidate-selection pass. Default 3650 (10 years)."
),
)
# ── Memory consolidation ────────────────────────────────────────────
consolidation_enabled: bool = Field(
default=False,

View File

@ -82,6 +82,16 @@ Memory Section Guidelines:
* behavior: Working patterns, communication habits, problem-solving approaches
* goal: Stated objectives, learning targets, project ambitions
* correction: Explicit agent mistakes or user corrections, including the correct approach
- Fact lifetime (``expected_valid_days``, optional integer):
How many days before this fact should be reviewed for possible removal.
The system schedules review automatically; omit when uncertain.
* <= 14: highly transient - active bugs, immediate tasks, today's focus
* 15-60: short-term - current experiments, in-progress side projects, near-term goals
* 60-180: medium-term - current role, active tech stack, ongoing preferences
* 180-365: stable - professional background, established working patterns
* > 365: very stable - core skills, native language, personality traits
Assign the value that semantically fits; values above the server-configured
ceiling are silently reduced on storage.
- Confidence levels:
* 0.9-1.0: Explicitly stated facts ("I work on X", "My role is Y")
* 0.7-0.8: Strongly implied from actions/discussions
@ -114,10 +124,11 @@ Output Format (JSON):
"longTermBackground": {{ "summary": "...", "shouldUpdate": true/false }}
}},
"newFacts": [
{{ "content": "...", "category": "preference|knowledge|context|behavior|goal|correction", "confidence": 0.0-1.0 }}
{{ "content": "...", "category": "preference|knowledge|context|behavior|goal|correction", "confidence": 0.0-1.0, "expected_valid_days": 90 }}
],
"factsToRemove": ["fact_id_1", "fact_id_2"],
"staleFactsToRemove": [{{ "id": "fact_id", "reason": "brief explanation" }}],
"staleFactsToExtend": [{{ "id": "fact_id", "extend_by_days": 365, "reason": "brief explanation" }}],
"factsToConsolidate": [
{{
"sourceIds": ["fact_id_1", "fact_id_2"],
@ -155,27 +166,37 @@ Return ONLY valid JSON, no explanation or markdown."""
# rather than relying on passive contradiction from the current conversation.
STALENESS_REVIEW_PROMPT = """## Staleness Review
The following facts were created more than {age_days} days ago and may no longer
accurately reflect the user's current situation. Review each one against the full
conversation context and your understanding of the user.
The following facts have reached their individual review window and may no longer
accurately reflect the user's current situation. Each entry shows a ``valid:Nd``
annotation - the number of days this fact was expected to remain valid before
re-evaluation. Use it to calibrate conservatism: a ``valid:30d`` fact was
considered volatile at creation; a ``valid:365d`` fact was considered stable.
<stale_facts>
{stale_facts}
</stale_facts>
For each fact, decide KEEP or REMOVE:
- KEEP: Still likely valid even if not mentioned in this conversation.
For each fact, decide KEEP, REMOVE, or EXTEND:
- KEEP: Still likely valid - even if not mentioned in this conversation.
Stable attributes (native language, core expertise, personality traits) often
remain true indefinitely.
- REMOVE: Outdated, contradicted by recent context, or no longer relevant.
Examples: tech-stack migrations, job changes, relocated offices, abandoned projects.
- EXTEND: Keep but recalibrate the review window (see below).
Add REMOVE decisions to "staleFactsToRemove" in your output JSON.
Each entry must be {{"id": "fact_id", "reason": "brief explanation"}}.
The reason should cite what signal in the conversation (or absence thereof)
supports the removal.
Be conservative when in doubt, KEEP. Removing a valid fact is worse than
Optionally, for facts you KEEP and wish to recalibrate, add them to
"staleFactsToExtend" with the number of days from now before the next review:
{{"id": "fact_id", "extend_by_days": 365, "reason": "brief explanation"}}
Use this when the current window seems miscalibrated - e.g. a core skill marked
``valid:30d`` that is clearly stable, or a goal nearing completion whose window
should shrink. Omit facts whose current window already seems appropriate.
Be conservative - when in doubt, KEEP. Removing a valid fact is worse than
keeping a slightly stale one, because the next review cycle will re-evaluate it."""

View File

@ -171,6 +171,15 @@ def _normalize_memory_update_fact(fact: Any) -> dict[str, Any] | None:
if normalized_source_error:
normalized_fact["sourceError"] = normalized_source_error
# Fact lifetime (expected_valid_days): optional LLM-assigned review window.
# Accept int/float (reject bool which subclasses int), coerce to int, keep
# only positive values; the creation-time cap is applied in _apply_updates.
raw_evd = fact.get("expected_valid_days")
if isinstance(raw_evd, (int, float)) and not isinstance(raw_evd, bool):
evd = int(raw_evd)
if evd > 0:
normalized_fact["expected_valid_days"] = evd
return normalized_fact
@ -216,6 +225,32 @@ def _normalize_memory_update_data(update_data: dict[str, Any]) -> dict[str, Any]
}
)
# ── Normalize staleness review lifetime extensions ──
stale_extensions_raw = update_data.get("staleFactsToExtend")
normalized_stale_extensions: list[dict[str, Any]] = []
if isinstance(stale_extensions_raw, list):
for entry in stale_extensions_raw:
if not isinstance(entry, dict):
continue
fact_id = entry.get("id")
if not isinstance(fact_id, str) or not fact_id:
continue
# extend_by_days: accept int/float (reject bool), coerce to int, keep > 0.
# A fractional value in (0, 1) coerces to 0 and is dropped here so the
# apply path never silently writes a zero-delta extension.
raw_extend = entry.get("extend_by_days")
if isinstance(raw_extend, (int, float)) and not isinstance(raw_extend, bool):
extend_by = int(raw_extend)
if extend_by > 0:
reason = entry.get("reason", "")
normalized_stale_extensions.append(
{
"id": fact_id,
"extend_by_days": extend_by,
"reason": reason if isinstance(reason, str) else "",
}
)
# ── Normalize consolidation decisions ──
consolidation_raw = update_data.get("factsToConsolidate")
normalized_consolidation: list[dict[str, Any]] = []
@ -265,6 +300,7 @@ def _normalize_memory_update_data(update_data: dict[str, Any]) -> dict[str, Any]
"newFacts": normalized_new_facts,
"factsToRemove": normalized_facts_to_remove,
"staleFactsToRemove": normalized_stale_removals,
"staleFactsToExtend": normalized_stale_extensions,
"factsToConsolidate": normalized_consolidation,
}
@ -357,16 +393,38 @@ def _parse_fact_datetime(raw: str) -> datetime | None:
return None
def _effective_fact_staleness_age(fact: dict[str, Any], config: Any) -> int:
"""Return the effective staleness review age in days for *fact*.
Returns the stored ``expected_valid_days`` value directly when present and
valid. The ``staleness_max_lifetime_multiplier`` cap is applied once at
*write time* (when a fact is first created) so the review window is bounded
from the start. Re-applying it here would prevent lifetime-extension
operations from ever moving the review window beyond that initial cap,
defeating the purpose of ``staleFactsToExtend``. Falls back to the global
``staleness_age_days`` for facts that pre-date this feature or where the
LLM did not provide an estimate.
"""
raw = fact.get("expected_valid_days")
if isinstance(raw, (int, float)) and not isinstance(raw, bool) and raw > 0:
return int(raw)
return config.staleness_age_days
def _select_stale_candidates(
current_memory: dict[str, Any],
config: Any,
) -> list[dict[str, Any]]:
"""Return facts that are older than ``staleness_age_days`` and not protected.
"""Return facts that have exceeded their individual review window.
Protected categories (default: ``correction``) are excluded because they
represent explicit user feedback that should not be auto-pruned by age.
Each fact's effective review age is determined by
``_effective_fact_staleness_age``: facts with an LLM-assigned
``expected_valid_days`` use that value directly; facts without it fall back
to the global ``staleness_age_days``. Protected categories (default:
``correction``) are excluded because they represent explicit user feedback
that should not be auto-pruned by age.
"""
cutoff = datetime.now(UTC) - timedelta(days=config.staleness_age_days)
now = datetime.now(UTC)
protected = frozenset(config.staleness_protected_categories)
candidates: list[dict[str, Any]] = []
for fact in current_memory.get("facts", []):
@ -376,31 +434,41 @@ def _select_stale_candidates(
if isinstance(category, str) and category in protected:
continue
created_at = _parse_fact_datetime(fact.get("createdAt", ""))
if created_at is not None and created_at < cutoff:
if created_at is None:
continue
effective_age = _effective_fact_staleness_age(fact, config)
if created_at < now - timedelta(days=effective_age):
candidates.append(fact)
return candidates
def _build_staleness_section(
stale_candidates: list[dict[str, Any]],
age_days: int,
config: Any,
) -> str:
"""Format the staleness review prompt section from candidate facts."""
"""Format the staleness review prompt section from candidate facts.
Each fact line includes a ``valid:Nd`` annotation - the effective review
window for that fact - so the LLM can calibrate its conservatism: a fact
reviewed after 30 days was considered volatile at creation; one reviewed
after 365 days was considered stable.
"""
if not stale_candidates:
return ""
lines: list[str] = []
for fact in stale_candidates:
fid = fact.get("id", "?")
cat = html.escape(str(fact.get("category", "context")).strip() or "context")
cat = html.escape(str(fact.get("category", "context")).strip() or "context", quote=False)
conf = _coerce_source_confidence(fact)
created_raw = fact.get("createdAt", "")
created_short = created_raw[:10] if isinstance(created_raw, str) and len(created_raw) >= 10 else created_raw
content = html.escape(str(fact.get("content", "")))
lines.append(f'- [{fid} | {cat} | {conf:.2f} | {created_short}] "{content}"')
return STALENESS_REVIEW_PROMPT.format(
stale_facts="\n".join(lines),
age_days=age_days,
)
# quote=False: content is in element-text position (inside <stale_facts>
# tags, never an attribute value), so only <, >, & can break structure -
# leave ' and " untouched. Mirrors the convention in prompt.py #4028.
content = html.escape(str(fact.get("content", "")), quote=False)
effective_age = _effective_fact_staleness_age(fact, config)
lines.append(f'- [{fid} | {cat} | {conf:.2f} | {created_short} | valid:{effective_age}d] "{content}"')
return STALENESS_REVIEW_PROMPT.format(stale_facts="\n".join(lines))
# ── Consolidation helpers ───────────────────────────────────────────────
@ -670,10 +738,7 @@ class MemoryUpdater:
if config.staleness_review_enabled:
stale_candidates = _select_stale_candidates(current_memory, config)
if len(stale_candidates) >= config.staleness_min_candidates:
staleness_section = _build_staleness_section(
stale_candidates,
config.staleness_age_days,
)
staleness_section = _build_staleness_section(stale_candidates, config)
# ── Build consolidation section ──
consolidation_section = ""
@ -953,48 +1018,101 @@ class MemoryUpdater:
if facts_to_remove:
current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in facts_to_remove]
# ── Staleness review removals ──
# ── Staleness review: removals + lifetime extensions ──
# Both operations share one staleness-candidate guardrail pass and one
# candidate_ids set. proposed_remove_ids is hoisted out of the removals
# sub-block so it covers ALL LLM-proposed removals, not just the ones
# the per-cycle cap actually deleted: a fact the LLM wanted to remove
# must never be silently extended even when the cap spares it.
stale_removals = update_data.get("staleFactsToRemove", [])
if isinstance(stale_removals, list) and stale_removals:
stale_ids_to_remove = {entry["id"] for entry in stale_removals if isinstance(entry, dict) and "id" in entry}
# Deterministic guardrail: intersect with actual staleness
# candidates so an LLM slip that emits a protected-category or
# non-aged fact id is silently rejected. Runs unconditionally
# so the apply-layer protection is independent of model behavior
# AND of the staleness_review_enabled flag.
# Guard against legacy / hand-edited facts that predate the id
# field: an aged, non-protected fact with no "id" is a valid
# staleness candidate but has no id to intersect against, so skip
# it here instead of raising KeyError (id-less facts can never be
# targeted by the id-based removal set anyway).
stale_extensions = update_data.get("staleFactsToExtend", [])
has_staleness_ops = (isinstance(stale_removals, list) and stale_removals) or (isinstance(stale_extensions, list) and stale_extensions)
if has_staleness_ops:
# Deterministic guardrail: intersect with actual staleness candidates
# so an LLM slip that emits a protected-category or non-aged fact id
# is silently rejected. Runs unconditionally so the apply-layer
# protection is independent of model behavior AND of the
# staleness_review_enabled flag. Guard against legacy / hand-edited
# facts that predate the id field: an aged, non-protected fact with
# no "id" is a valid staleness candidate but has no id to intersect
# against, so skip it here instead of raising KeyError.
candidate_ids = {f["id"] for f in _select_stale_candidates(current_memory, config) if f.get("id") is not None}
stale_ids_to_remove &= candidate_ids
if not stale_ids_to_remove:
# After intersection with candidate set, nothing to remove.
stale_removals = []
else:
# Safety cap: limit max staleness removals per cycle.
# When the LLM returns more than the cap, keep only the
# lowest-confidence entries up to the limit so the most
# questionable facts are removed first.
max_stale = config.staleness_max_removals_per_cycle
if len(stale_ids_to_remove) > max_stale:
stale_facts = [f for f in current_memory.get("facts", []) if f.get("id") in stale_ids_to_remove]
stale_facts.sort(key=_coerce_source_confidence)
stale_ids_to_remove = {f["id"] for f in stale_facts[:max_stale]}
# ── Removals ──
proposed_remove_ids: set[str] = set()
if isinstance(stale_removals, list) and stale_removals:
proposed_remove_ids = {entry["id"] for entry in stale_removals if isinstance(entry, dict) and "id" in entry}
stale_ids_to_remove = proposed_remove_ids & candidate_ids
current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in stale_ids_to_remove]
if not stale_ids_to_remove:
stale_removals = []
else:
# Safety cap: limit max staleness removals per cycle. When
# the LLM returns more than the cap, keep only the
# lowest-confidence entries up to the limit so the most
# questionable facts are removed first.
max_stale = config.staleness_max_removals_per_cycle
if len(stale_ids_to_remove) > max_stale:
stale_facts = [f for f in current_memory.get("facts", []) if f.get("id") in stale_ids_to_remove]
stale_facts.sort(key=_coerce_source_confidence)
stale_ids_to_remove = {f["id"] for f in stale_facts[:max_stale]}
# Log removals for observability
for entry in stale_removals:
if isinstance(entry, dict) and entry.get("id") in stale_ids_to_remove:
logger.info(
"Staleness review removed fact %s: %s",
entry["id"],
entry.get("reason", "no reason provided"),
)
current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in stale_ids_to_remove]
# Log removals for observability
for entry in stale_removals:
if isinstance(entry, dict) and entry.get("id") in stale_ids_to_remove:
logger.info(
"Staleness review removed fact %s: %s",
entry["id"],
entry.get("reason", "no reason provided"),
)
# ── Lifetime extensions ──
# Recalibrate expected_valid_days for facts the LLM chose to keep.
# Eligible facts are stale candidates that the LLM did NOT propose
# for removal - including those that survived only because the
# per-cycle cap prevented their deletion. The new window is
# min(days_since + extend_by_days, staleness_max_extension_days).
# Extensions use an absolute ceiling rather than the creation-time
# multiplier cap: they are deliberate review decisions and must be
# able to advance the window beyond the original creation cap, but
# an absolute bound prevents timedelta overflow and LLM misfire.
if isinstance(stale_extensions, list) and stale_extensions:
# Exclude all LLM-proposed removals, not just the trimmed set,
# so a cap-surviving proposed-removal fact is never extended.
extendable_ids = candidate_ids - proposed_remove_ids
ext_by_id = {e["id"]: e for e in stale_extensions if isinstance(e, dict) and isinstance(e.get("id"), str) and e["id"] in extendable_ids}
if ext_by_id:
now_utc = datetime.now(UTC)
max_ext = config.staleness_max_extension_days
updated_facts: list[dict[str, Any]] = []
for fact in current_memory.get("facts", []):
fid = fact.get("id")
ext = ext_by_id.get(fid) if fid else None
if ext is not None:
extend_by = ext.get("extend_by_days")
if isinstance(extend_by, (int, float)) and not isinstance(extend_by, bool):
extend_by_int = int(extend_by) # coerce before guard
if extend_by_int > 0:
created = _parse_fact_datetime(fact.get("createdAt", ""))
if created is None:
# Unreachable: _select_stale_candidates already
# excludes facts with unparseable createdAt.
updated_facts.append(fact)
continue
days_since = int((now_utc - created).total_seconds() // 86400)
new_evd = min(days_since + extend_by_int, max_ext)
fact = {**fact, "expected_valid_days": new_evd}
logger.info(
"Staleness review extended fact %s by %d days (new expected_valid_days: %d): %s",
fid,
extend_by_int,
new_evd,
ext.get("reason", "no reason provided"),
)
updated_facts.append(fact)
current_memory["facts"] = updated_facts
# Add new facts
existing_fact_keys = {fact_key for fact_key in (_fact_content_key(fact.get("content")) for fact in current_memory.get("facts", [])) if fact_key is not None}
@ -1028,6 +1146,15 @@ class MemoryUpdater:
normalized_source_error = source_error.strip()
if normalized_source_error:
fact_entry["sourceError"] = normalized_source_error
evd = fact.get("expected_valid_days")
if isinstance(evd, int) and not isinstance(evd, bool) and evd > 0:
# Apply the creation-time cap so the LLM cannot assign an
# unbounded lifetime that defers staleness review indefinitely.
# Extensions (staleFactsToExtend) bypass this cap via their own
# staleness_max_extension_days ceiling because they represent a
# deliberate review decision, not an unchecked initial assignment.
creation_cap = int(config.staleness_age_days * config.staleness_max_lifetime_multiplier)
fact_entry["expected_valid_days"] = min(evd, creation_cap)
current_memory["facts"].append(fact_entry)
if fact_key is not None:
existing_fact_keys.add(fact_key)

View File

@ -41,6 +41,8 @@ _LEGACY_DEERMEM_FIELDS = frozenset(
"staleness_min_candidates",
"staleness_max_removals_per_cycle",
"staleness_protected_categories",
"staleness_max_lifetime_multiplier",
"staleness_max_extension_days",
"consolidation_enabled",
"consolidation_min_facts",
"consolidation_max_groups_per_cycle",

File diff suppressed because it is too large Load Diff

View File

@ -1474,6 +1474,15 @@ summarization:
# staleness_min_candidates - Minimum stale facts to trigger review (default: 3)
# staleness_max_removals_per_cycle - Safety cap on removals per cycle (default: 10)
# staleness_protected_categories - Categories exempt from staleness review (default: ["correction"])
# staleness_max_lifetime_multiplier - Creation-time cap multiplier for a fact LLM-assigned
# expected_valid_days; new facts clamped to
# staleness_age_days x multiplier. Default 20.0
# (90 x 20 = 1800d ~ 5 years) supports the very-stable
# prompt tier. (default: 20.0, range: 1.0-100.0)
# staleness_max_extension_days - Absolute ceiling (in days) on expected_valid_days after a
# lifetime extension (staleFactsToExtend). Prevents timedelta
# overflow and LLM misfire from permanently deferring a fact.
# (default: 3650, range: 90-36500)
memory:
enabled: true
injection_enabled: true # Whether to inject memory into system prompt
@ -1527,14 +1536,19 @@ memory:
guaranteed_token_budget: 500
# Staleness review: periodically prune aged facts that may no longer reflect
# the user current situation. When triggered, the LLM reviews facts older
# than staleness_age_days during the normal memory-update call (same LLM
# invocation - no extra API call) and decides KEEP or REMOVE for each.
# than their individual review window (expected_valid_days, or
# staleness_age_days as fallback) during the normal memory-update call (same
# LLM invocation - no extra API call) and decides KEEP, REMOVE, or EXTEND
# for each. The LLM assigns expected_valid_days when creating a fact; EXTEND
# (staleFactsToExtend) recalibrates that window at review time.
staleness_review_enabled: true
staleness_age_days: 90
staleness_min_candidates: 3
staleness_max_removals_per_cycle: 10
staleness_protected_categories:
- correction
staleness_max_lifetime_multiplier: 20.0 # creation cap = staleness_age_days x multiplier (90 x 20 = 1800d)
staleness_max_extension_days: 3650 # absolute ceiling on extended expected_valid_days (~10 years)
# Memory consolidation (opt-in, lossy: source facts are replaced by a
# synthesized one; only consolidatedFrom IDs are kept). Runs in the same
# memory-update LLM call as extraction/staleness - no extra API cost.