* feat(checkpoint): make delta snapshot_frequency configurable
* fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning
Addresses review on #4516: the rename from the flat
database.checkpoint_delta_snapshot_frequency key to nested
database.checkpoint_delta.snapshot_frequency silently dropped the old
value (pydantic extra="ignore"). Add a before-validator that maps the
legacy key onto the nested one with a deprecation warning (nested key
wins when both are set), plus a CHANGELOG breaking-change note covering
the rename and the 1000 -> 10 default change.
* fix(checkpoint): validate frozen snapshot frequency
* refactor(memory): pluggable MemoryManager interface for backend onboarding
Optimize the MemoryManager interface layer so new backends (mem0/openviking)
onboard with less code and the contract stays stable as capabilities are
added. A minimal backend now implements only from_config + add + get_context
(verified by test_memory_manager_interface.py::_MinimalBackend onboarding via
the factory); the factory no longer knows a backend's private hooks.
- MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1
add/get_context abstract; tier-2 management defaults; tier-3 optional hooks
warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving
hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError.
- from_config classmethod: factory thins to resolve + inject storage_path +
collect host hooks + call from_config; DeerMem-specific hook consumption
moved from factory to DeerMem.from_config.
- Invariants: @model_validator (mode='tool' requires search via supports_search
ClassVar); DeerMemConfig storage_path-is-file check moved here from factory.
- Async: aadd/aget_context/asearch default to the sync path (speculative).
- Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call
subsumes tracing_callback (same signature/timing/mutation); deleted the
tracing_callback field. DeerMem decoupled from langfuse (portability).
- noop keeps read-op empty overrides (avoids router 500s on the
disable-memory-via-noop path); only delete/export inherit the base raise.
Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten
(three-tier + from_config + callbacks); samples README updated; removed stale
private doc paths.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): 501 on unsupported read/manage endpoints + accurate warm log
Review follow-up on the three-tier MemoryManager refactor.
- Read/manage endpoints (GET /memory, /memory/export, /memory/status,
DELETE /memory, POST /memory/import) and the /memory/reload fallback now
catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The
hasattr->try/except migration had skipped these: they were @abstractmethod
before (every backend implemented them, so they never raised), so once they
became tier-2 default-raise a minimal backend (only add + get_context) hit a
raw 500 -- there is no global NotImplementedError handler. get_memory is
shared via _get_memory_or_501 (covers /memory, export, status, reload
fallback). noop is unchanged: its read-op empty overrides never raise.
- warm() base default returns None (tri-state: True=warmed, False=failed,
None=nothing to warm) so the Gateway lifespan logs "skipping" for a
non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed
successfully" it never earned. DeerMem.warm keeps True/False.
- Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan
warm-log tests (None->skipping, False->warning); conformance/pluggable
assert warm() is None.
705 passed / 13 skipped; lint clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity
Address review feedback on the three-tier MemoryManager refactor:
- [Medium] supports_search/search drift: the invariant now requires the
supports_search ClassVar flag to MATCH whether search() is actually
overridden (type(self).search is not MemoryManager.search), so the flag
can't drift from the impl. Catches both directions at instantiation: a
backend that overrides search() but forgets supports_search=True (was a
misleading tool-mode rejection), and one that sets the flag without
overriding (was a runtime NotImplementedError on the first memory_search).
noop sets supports_search=True to match its search() override. Conformance
adds drift + consistent-backend tests.
- [Low] client.reload_memory fallback: wrap the get_memory fallback so a
minimal backend (only add + get_context) surfaces a clean NotImplementedError
("implements neither reload_memory nor get_memory") instead of an uncaught
propagation -- mirrors the router's 501. Test added.
- [Low] backend_config purity: DeerMem.from_config restores backend_config to
the pure data the host passed after model_post_init parses the injected hooks
into DeerMemConfig (self._config, PrivateAttr); the field stays serializable
(no callables/LLM) and matches the README ("host hooks NOT in backend_config").
Test asserts purity + hooks wired.
- [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend
now fails fast at startup (was silently empty) so operators recognize it on
upgrade.
- [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is
local-only, not make test/CI).
709 passed / 13 skipped; lint clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(changelog): correct memory tool-mode fail-fast note
The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)"
fails fast at startup, but noop overrides search() (returns []) and sets
supports_search=True (required by the consistency invariant), so noop IS
search-capable and noop+tool does NOT fail fast. The fail-fast only affects a
custom backend that onboards without overriding search(). Reworded to drop the
misleading noop example and state both shipping backends implement search().
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(channels): key inbound dedupe on chat-scoped workspaces
* fix(channels): release the inbound dedupe key on swallowed transient failures
_release_inbound_dedupe_key runs from _handle_message's generic exception
handler, but three sites handle their error in place and never re-raise, so
the key recorded on receipt survives the full dedupe TTL and the provider's
redelivery -- the retry that would have recovered the failure -- is dropped:
- _handle_streaming_chat swallows every stream error into its finally block
- the fire-and-forget runs.create busy branch
- the non-streaming runs.wait busy branch
Keying chat-scoped workspaces gives telegram/feishu/wechat/dingtalk a dedupe
key for the first time, which newly exposes them to this. The mechanism is
pre-existing, not introduced here: WeCom already had a key (streaming plus
aibotid) and shows the same black hole on main.
Also parametrize the dispatch dedupe test over all three chat-scoped
providers so the streaming path is covered end-to-end, and restate the
workspace-fallback comment in terms of what the chain actually guarantees --
both fallbacks are appended last and gated on every earlier source being
absent, so they can only turn "no key" into a key, never change one.
* fix(channels): publish final reply before dedupe release
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): claim the converted markdown filename before writing it
* doc(changelog): record the converted-markdown filename collision fix
Covers both surfaces that report markdown_file: the gateway uploads route
and DeerFlowClient.upload_files.
* fix(uploads): release the claimed markdown name when conversion fails
Claiming the companion .md name before conversion means a conversion that
writes nothing leaves the name reserved for the rest of the request, so a
later same-stem upload is renamed against a name nothing occupies. Uploading
notes.docx (conversion fails) + notes.md stored the user's file as notes_1.md
with no notes.md on disk, where main kept notes.md.
Discard the claim when conversion returns None, at both call sites that
pre-claim it. Covers both victims: a later same-stem .md upload, and the next
convertible's companion.
* fix(helm): default sandbox Services to ClusterIP (#3929)
The K8s sandbox provisioner supports both NodePort and ClusterIP via
SANDBOX_SERVICE_TYPE (added in #4016), but the Helm chart never set it,
so real-cluster installs inherited the NodePort default. That bound the
code-execution sandbox on every node's interfaces - including externally
reachable ones on GKE/EKS/AKS - and pinned every sandbox URL to one node
IP (SPOF on node reboot/drain/ephemeral-IP).
Default the chart to ClusterIP: the provisioner returns a cluster-DNS URL
(http://sandbox-<id>-svc.<ns>.svc.cluster.local:8080) so the gateway->
sandbox hop stays inside the cluster network - no node IP, no 30xxx port,
no external exposure. The chart always runs the gateway in-cluster, so
ClusterIP is always correct there.
NodePort remains an opt-in (provisioner.sandboxServiceType: NodePort +
nodeHost) for the Docker-Compose/hybrid path where the gateway is not in
K8s and cannot resolve .svc.cluster.local; the provisioner code default
stays NodePort for that path.
- values.yaml: add provisioner.sandboxServiceType ("ClusterIP")
- provisioner-deployment.yaml: emit SANDBOX_SERVICE_TYPE; gate the
NODE_HOST block on NodePort mode (default "ClusterIP" for upgrade safety)
- NOTES.txt + README.md: document ClusterIP default + NodePort opt-in
No change to docker/provisioner/app.py (already mode-aware since #4016)
or RBAC (services verbs already cover ClusterIP).
* test(helm): assert sandbox Service-type gating + CHANGELOG the default flip (#3929)
Address review on #4190:
- Add scripts/check_chart_sandbox_service.sh: renders the chart for the
default (ClusterIP, no NODE_HOST), the NodePort opt-in (both emitted),
and NodePort+nodeHost (literal value, not downward API). Locks in the
#3929 gating so a regression (e.g. re-adding an unconditional NODE_HOST,
or dropping the `default "ClusterIP"` upgrade-safety fallback) fails CI.
Wired into .github/workflows/chart.yaml validate-chart job. (#2)
- CHANGELOG [Unreleased] -> Changed: note the NodePort->ClusterIP default
flip on upgrade + the `sandboxServiceType: NodePort` opt-back-in. (#4)
No chart template changes (the gating itself landed in the first commit).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)
Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated
Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema
* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks
Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):
- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
consolidation_min_facts / consolidation_max_groups_per_cycle /
consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
+ CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
_build_consolidation_section module helpers (matching the existing
_select_stale_candidates style); consolidation normalization in
_normalize_memory_update_data; consolidation apply in _apply_updates (after
max_facts trim, with apply-time guardrails mirroring staleness); staleness
KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
incl. the staleness KeyError regression
Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): add noop backend template and backends guide
- backends/noop/: complete drop-in template (config.py with zero deer-flow
imports, noop_manager.py with a 6-step new-backend walkthrough in its
docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
references).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): guard formatTimeAgo against invalid timestamps
Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): wire tool-driven memory mode through the MemoryManager ABC
tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.
Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)
* docs: restore explanatory comments in config.example.yaml memory section
* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py
* fix(memory): address review + port dropped upstream memory fixes
Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
knobs live under backend_config, not top-level - restoring flat would
re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
request-trace ContextVar on the Timer/executor worker via a new
trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
explicit manager_class instead of silently falling back to DeerMem (memory is
persistent state - a wrong store is a silent data-integrity footgun)
Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
(core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
(deer_mem.py DeerMem.search)
host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)
* fix(memory): restore hide_from_ui clarification preservation, expose mode
Two memory-system fixes (F541/E501 lint was already fixed on this branch):
- filter_messages_for_memory: restore default preservation of well-formed
human_input_response clarification answers (v2 regression). The
self-containment refactor made the bare function skip ALL hide_from_ui when
no hook was passed, but upstream preserves well-formed clarification
responses by default (test_hide_from_ui_human_input_response_is_preserved).
Inline a host-agnostic _is_human_clarification_response mirror of
read_human_input_response as the default keep-decision; the host-injected
should_keep_hidden_message hook still overrides (production path unchanged).
Portable package stays zero `from deerflow`.
- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
the config/status endpoints + client.get_memory_config. mode is a host-
shared, behavior-determining field missing from the response projection.
Sync tests (mock .mode; e2e assert mode present).
- Align manager_class field docstring with fail-fast behavior.
Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): resolve ruff format failures in memory module + tests
`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs
Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state):
- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
(storage_path, max_facts, debounce_seconds, model_name, token_counting,
staleness_*, consolidation_*) into backend_config on load + warn, so an
upgrade does NOT silently revert customized settings (was: silent
extra='ignore' drop). model_name -> backend_config.model.model. Unknown
top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
(base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
re-deriving it via content-key matching (coupled the tool to the backend's
content normalization; could misreport a storage cap). create_fact now returns
(memory_data, fact_id); gateway/client/tool updated. Fix terse
{"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
(read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
shape flat->backend_config; custom storage_class path moved + __init__ must
accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
across the abstraction; update create_fact mocks for (memory_data, fact_id).
Tests: 273 passed (memory suite); ruff check + format clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address PR review - storage_path, max_facts, tracing, parsing
Six review findings (willem-bd), each verified against upstream:
- storage_path semantics (file -> root dir): migration drops file-style
(.json) legacy values with a warning; factory raises if storage_path
resolves to an existing file (avoid silent NotADirectoryError write
failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
returns (memory, None) when the cap evicts the new fact; memory_add tool
reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
0) -> TypeError on non-float imported/legacy confidence, swallowed as
silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
broad "if not value" which would skip explicit bool False).
8 new regression tests. make lint + 406 memory tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template
Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):
- create_storage fail-fast: a misspelled/unimportable storage_class now
raises ValueError instead of silently falling back to FileMemoryStorage.
Memory is persistent state, so a wrong store is a data-integrity footgun;
mirrors the existing manager_class resolution policy. (storage.py)
- noop template create_fact signature: the commented template used
keyword-only `content` and returned a bare dict, while DeerMem's actual
create_fact takes positional `content` and returns tuple[dict, str|None]
(the memory_add tool passes content positionally; gateway/client/tools all
tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
Template fixed; delete_fact/update_fact templates left (callers compatible).
(noop_manager.py)
- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
explicit model does not crash app startup -- non-LLM memory ops still work
and an update raises at runtime with the error logged. (llm.py)
- from_backend_config unknown-key warning: log a WARNING for unknown
backend_config keys (mirrors the host layer's load_memory_config_from_dict)
so a typo like `storage_pat` does not silently fall back to the default and
write memory to an unintended location. (config.py)
Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>
* fix(models): scope the OpenAI-compat rules to BaseChatOpenAI, not a class-path allowlist
* address review: drop redundant stream_usage helper, close test matrix
- Remove _enable_stream_usage_by_default and its now-unused _OPENAI_COMPAT_USE_PATHS
tuple. The class-field stream_usage fallback already sets stream_usage=True for
every BaseChatOpenAI subclass (they all declare the field), so the helper's
use-path allowlist gated nothing real — verified a no-op in prod, and the two
stream_usage tests stay green on main with the helper present. Those tests used a
BaseChatModel stub that does not declare the field; point them at a real ChatOpenAI
capturing class so they exercise the fallback they now depend on.
- Give the non-OpenAI normalization-skip test an actual api_base value so it exercises
the skip path (api_base passed through verbatim, never rewritten to base_url).
- Add an Unreleased CHANGELOG entry for the api_base behavior change on the five
affected subclasses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
