github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-10-10 05:16:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,976 Commits 42 Branches 10 Tags
Commit Graph

9 Commits

Author SHA1 Message Date
Nan Gao
7389331e65
feat(extensions): observe task lifecycle and system model calls (#4684) 
* feat(extensions): observe task lifecycle and system model calls

PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.

Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).

`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.

Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.

Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.

System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.

The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.

Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.

Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.

* fix(extensions): decide notification fail-open by origin, observe cancellation

`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.

`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.

DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.

* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown

Review response on #4684:

- The memory observation bridge caught BaseException, which would swallow
  a teardown signal raised while dispatching; it now catches Exception,
  matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
  hook-failure path and logged an asyncio-internal traceback; it now logs
  a warning like the pre-hook budget skip, while a TimeoutError a
  contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
  so log lines said "task goal/title/..."; they now carry the task
  scope id alongside the kind.
 2026-08-11 16:33:22 +08:00
Aari
cccda35cc5
fix(memory): prevent task-scoped data from entering long-term memory (#4604) 
* fix(memory): gate long-term updates by scope

* docs(memory): note custom prompts_dir migration; fix stale accept-filter comment

* fix(memory): harden scope-gate review paths
 2026-08-01 08:39:28 +08:00
Vanzeren
352f247a81
feat(memory): add mem0 HTTP memory backend (#4528) 
* feat(memory): add mem0 HTTP memory backend

* fix(memory): address mem0 review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-07-29 07:11:20 +08:00
qin-chenghan
795af20a6b
feat(memory): built-in FTS5/BM25 retrieval adapter (#4360) 
* feat(memory): integrate FTS5 retrieval adapter

* deps: add jieba as default dependency for Chinese tokenization

Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.

* fix(memory): avoid treating hyphens as FTS5 operators

* feat(memory): make Chinese tokenization optional

* fix(memory): warm every requested retrieval scope

* fix(memory): close retrieval resources on shutdown

* fix(memory): close backend when shutdown flush fails

* fix(memory): recreate corrupt retrieval index

* fix(memory): tolerate partial retrieval rebuilds

* fix(memory): warm retrieval index in background

* fix(memory): preserve shutdown flush budget

* fix(memory): stop retrying partial lazy rebuilds

* fix(memory): close retrieval through storage

* refactor(memory): simplify retrieval scope limit

* docs(memory): clarify retrieval shutdown lifecycle

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-07-27 23:17:18 +08:00
lllyfff
8145d66a33
feat(memory): memory message processing (#4447) 
* feat(memory): signals-based update pipeline + always-on watermark/trivial filter

Refactor the DeerMem memory update pipeline (message_processing -> queue ->
updater) around a signals frozenset seam, replacing the
(filtered, correction_detected, reinforcement_detected) 3-tuple with
(filtered, signals: frozenset[str]) end to end.

message_processing:
- Externalize signal-detection patterns to YAML (message_patterns/*.yaml).
- Extend signals from correction/reinforcement to a 6-class set
  (correction/reinforcement/preference/identity/goal/decision); detect_signals
  returns a frozenset aligned with the fact category enum.
- Pure-acknowledgment turns ("ok"/"好的"/...) are always filtered out before
  enqueue (whole-message fullmatch), saving an extraction LLM call.

queue (core/queue.py):
- In-memory list + debounce timer, with flush_sync (graceful-shutdown drain
  that joins an in-flight worker under a hard timeout) and queue_max_depth
  backpressure (signal-bearing updates always admitted; QueueFull otherwise).
- Same-key updates coalesce with a signal union; per-batch success/fail summary.

updater (core/updater.py):
- head500+tail500 message truncation (replaces the 1000-char head chop).
- Always-on per-thread watermark: feed only messages added since the last
  extraction. The watermark is in-memory and is not advanced on failure, so a
  failed/lost update is re-fed on the next conversation turn.
- [MANUAL] prompt marker for user-authored facts (source.type="manual").
- Post-invoke extraction_callback (host-injected) emitting facts_extracted /
  facts_accepted / rejected_low_confidence; the host default logs metrics and
  flags >60% rejection.

Confidence filtering remains in _apply_updates (the existing
fact_confidence_threshold check); there is no separate write gate.
Consolidation stays opt-in (lossy). The ABC add/add_nowait signature is
unchanged, so the summarization flush hook and host are unaffected.

Tests: add test_message_processing_signals, test_updater_truncation,
test_updater_watermark; update queue/updater/consolidation/staleness/pluggable
tests for the signals seam.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): harden update pipeline per PR review

- Catch QueueFull in DeerMem.add/add_nowait so backpressure degrades to
  'update skipped' instead of propagating into after_agent /
  summarization_hook and breaking the agent run (peer middlewares
  self-guard; MemoryMiddleware was the lone exception). Emergency
  (add_nowait) always admits under backpressure -- its data cannot be
  re-fed next turn.
- Rewrite the watermark from index-based to content/identity-based
  (_message_identity + _feed_after_watermark) so it stays correct when
  summarization removes the conversation front -- an index watermark
  pointed at the wrong message and silently skipped un-extracted tail
  turns. The emergency flush bypasses the watermark (bypass_watermark on
  ConversationContext, threaded through update_memory) and coexists with
  (does not replace) a pending normal update, so a flush cannot drop a
  pending update's un-extracted tail.
- Populate facts_accepted / rejected_low_confidence inside _apply_updates
  at the real confidence-filter site (passed_threshold) instead of
  re-deriving the threshold in _finalize_update -- eliminates metric drift.
- Emit extraction metrics in a finally with an 'attempted' flag so
  exception failures (parse error, apply_changes raise after retry) are
  observable, not only the happy path.
- Re-detect signals on the post-watermark feed for the extraction hint so
  it no longer references turns the LLM cannot see; admission-time signals
  still drive backpressure.
- Move the post-batch reschedule inside the queue lock to close a
  non-atomic self._timer race with a concurrent add.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address follow-up review nits (LRU, metric name, docstring)

- Bound the in-memory watermark cache with a configurable LRU
  (watermark_max_keys, default 4096, 0=unbounded). A dropped key re-extracts
  one batch on that thread's next turn (the documented restart behavior), so
  eviction is safe and preserves the content-identity watermark's
  front-removal guarantee. Adds _watermark_get/_watermark_set helpers and a
  bounded-LRU regression test.
- Rename the extraction metric facts_accepted -> facts_passed_confidence so
  the name matches what the >60% rejection-rate warning assumes (a
  confidence-gate signal, not a persisted-fact count); drop the stale
  "historical semantics" justification. Brand-new callback, one consumer.
- Fix the stale test_message_processing_signals module docstring: the signals
  seam is already swapped to frozenset, and a stale stage-numbering prefix is
  removed.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-26 21:16:36 +08:00
lllyfff
01a89f2379
[feat] memory: pluggable MemoryManager interface for backend onboarding (#4326) 
* refactor(memory): pluggable MemoryManager interface for backend onboarding

Optimize the MemoryManager interface layer so new backends (mem0/openviking)
onboard with less code and the contract stays stable as capabilities are
added. A minimal backend now implements only from_config + add + get_context
(verified by test_memory_manager_interface.py::_MinimalBackend onboarding via
the factory); the factory no longer knows a backend's private hooks.

- MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1
  add/get_context abstract; tier-2 management defaults; tier-3 optional hooks
  warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving
  hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError.
- from_config classmethod: factory thins to resolve + inject storage_path +
  collect host hooks + call from_config; DeerMem-specific hook consumption
  moved from factory to DeerMem.from_config.
- Invariants: @model_validator (mode='tool' requires search via supports_search
  ClassVar); DeerMemConfig storage_path-is-file check moved here from factory.
- Async: aadd/aget_context/asearch default to the sync path (speculative).
- Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call
  subsumes tracing_callback (same signature/timing/mutation); deleted the
  tracing_callback field. DeerMem decoupled from langfuse (portability).
- noop keeps read-op empty overrides (avoids router 500s on the
  disable-memory-via-noop path); only delete/export inherit the base raise.

Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten
(three-tier + from_config + callbacks); samples README updated; removed stale
private doc paths.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): 501 on unsupported read/manage endpoints + accurate warm log

Review follow-up on the three-tier MemoryManager refactor.

- Read/manage endpoints (GET /memory, /memory/export, /memory/status,
  DELETE /memory, POST /memory/import) and the /memory/reload fallback now
  catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The
  hasattr->try/except migration had skipped these: they were @abstractmethod
  before (every backend implemented them, so they never raised), so once they
  became tier-2 default-raise a minimal backend (only add + get_context) hit a
  raw 500 -- there is no global NotImplementedError handler. get_memory is
  shared via _get_memory_or_501 (covers /memory, export, status, reload
  fallback). noop is unchanged: its read-op empty overrides never raise.
- warm() base default returns None (tri-state: True=warmed, False=failed,
  None=nothing to warm) so the Gateway lifespan logs "skipping" for a
  non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed
  successfully" it never earned. DeerMem.warm keeps True/False.
- Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan
  warm-log tests (None->skipping, False->warning); conformance/pluggable
  assert warm() is None.

705 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity

Address review feedback on the three-tier MemoryManager refactor:

- [Medium] supports_search/search drift: the invariant now requires the
  supports_search ClassVar flag to MATCH whether search() is actually
  overridden (type(self).search is not MemoryManager.search), so the flag
  can't drift from the impl. Catches both directions at instantiation: a
  backend that overrides search() but forgets supports_search=True (was a
  misleading tool-mode rejection), and one that sets the flag without
  overriding (was a runtime NotImplementedError on the first memory_search).
  noop sets supports_search=True to match its search() override. Conformance
  adds drift + consistent-backend tests.
- [Low] client.reload_memory fallback: wrap the get_memory fallback so a
  minimal backend (only add + get_context) surfaces a clean NotImplementedError
  ("implements neither reload_memory nor get_memory") instead of an uncaught
  propagation -- mirrors the router's 501. Test added.
- [Low] backend_config purity: DeerMem.from_config restores backend_config to
  the pure data the host passed after model_post_init parses the injected hooks
  into DeerMemConfig (self._config, PrivateAttr); the field stays serializable
  (no callables/LLM) and matches the README ("host hooks NOT in backend_config").
  Test asserts purity + hooks wired.
- [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend
  now fails fast at startup (was silently empty) so operators recognize it on
  upgrade.
- [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is
  local-only, not make test/CI).

709 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(changelog): correct memory tool-mode fail-fast note

The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)"
fails fast at startup, but noop overrides search() (returns []) and sets
supports_search=True (required by the consistency invariant), so noop IS
search-capable and noop+tool does NOT fail fast. The fail-fast only affects a
custom backend that onboards without overriding search(). Reworded to drop the
misleading noop example and state both shipping backends implement search().

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-22 14:40:57 +08:00
RongfuShuiping
4bf028d048
feat(memory): add incremental agent-scoped Markdown fact storage (#4279) 
* feat(memory): add Markdown fact storage repository

* docs(memory): explain storage rewrite for beginners

* docs(memory): fix plan markdown formatting

* refactor(memory): separate global summaries from agent facts

* fix(memory): make Markdown fact updates incremental and safe

* Update STORAGE_REWRITE_CHANGES.md

* Delete docs/plans/STORAGE_REWRITE_PLAN.md

* Delete docs/plans/STORAGE_REWRITE_CHANGES.md

* fix(memory): address Markdown storage review feedback

* fix(memory): complete review follow-ups

* fix(memory): resolve storage review findings

* feat(memory): add proactive Markdown migration CLI

* fix(memory): harden Markdown storage concurrency

* fix(memory): harden markdown storage migration

* fix(memory): close migration review gaps

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: qin-chenghan <qinchenghan@Huawei.com>
 2026-07-22 07:46:03 +08:00
Tu Naichao
959bf13406
fix(memory): flush memory queue on graceful shutdown to prevent loss (#4181) 
* fix(memory): bounded shutdown flush via MemoryManager.shutdown_flush

Re-applies the memory-queue shutdown drain on top of the pluggable
MemoryManager abstraction (#4122): the old top-level MemoryUpdateQueue
singleton is gone, so the drain is now a backend contract instead of
host code reaching into the queue.

- MemoryManager ABC: shutdown_flush(timeout) -> bool. Every backend
  implements a bounded graceful-shutdown drain.
- DeerMem: queue.flush_sync (daemon-thread + Event.wait hard timeout
  for the uninterruptible sync LLM call; joins an in-flight worker
  first so contexts a debounce Timer already pulled out are not lost on
  exit; skips inter-item sleep on the drain path; per-item
  succeeded/failed count), exposed via shutdown_flush.
- noop: shutdown_flush is a clean no-op success.
- Gateway lifespan: call get_memory_manager().shutdown_flush(timeout)
  after channels/scheduler stop, via asyncio.to_thread, try/except
  bounded. No host-level pending/processing guard -- the backend
  short-circuits on an idle buffer, so the host cannot "forget" the
  in-flight case (structurally eliminates the guard race flagged on the
  prior revision).
- shutdown_flush_timeout_seconds added to the shared MemoryConfig
  (host-owned lifecycle budget, default 30, 1-300) + exposed on
  MemoryConfigResponse and the embedded client; config_version 25 -> 26.

Tests: queue flush_sync (7), lifespan drain incl. False-branch caplog
assertion + disabled gate (3), ABC contract noop/deermem (3).

* fix(chart): gateway grace period so memory drain is not SIGKILLed

K8s defaults terminationGracePeriodSeconds to 30s, shorter than the
Gateway's graceful-shutdown work (channel stop ~5s + memory queue drain
default 30s). Without an explicit grace period, K8s SIGKILLs the memory
drain mid-flight and silently re-introduces the loss shutdown_flush is
fixing (flagged on the prior revision).

- gateway pod: terminationGracePeriodSeconds (default 45, configurable).
- gateway container: preStop sleep (default 5, 0 disables) so the
  Service/ingress deregisters the pod before SIGTERM begins the drain.
- values.yaml + README: both configurable; README documents that the
  grace period must track memory.shutdown_flush_timeout_seconds.

* docs(memory): document shutdown_flush_timeout_seconds + lifespan drain

Add the host-shared field to the memory config list and Config Schema
summary in backend/AGENTS.md, noting the lifespan drain and the K8s
grace-period relationship.

* fix(chart): bump embedded config_version to 26

The chart's embedded `config:` block (values.yaml + README example) still
had config_version: 25 after commit f3ca8e9f raised config.example.yaml to
26, failing the validate-chart config_version drift check. Bump both to 26.
 2026-07-15 20:25:41 +08:00
qin-chenghan
ad45f59d66
feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122) 
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>
 2026-07-15 11:21:04 +08:00