* fix(memory): recall CJK sub-phrases when jieba is not installed
Without the memory-zh extra, _tokenize fell back to whitespace splitting, so
the FTS5 index kept a whole CJK run as a single unicode61 token and every CJK
sub-phrase query returned no hits without any warning. The relevance ranker
already emitted CJK character bigrams for the same text, so the two retrieval
strategies disagreed on the same data.
Reuse relevance.cjk_token for CJK runs when jieba is unavailable and route
_preprocess_content through _tokenize, so the index and the query side agree.
Installs with jieba keep the existing jieba path.
Existing index rows are rebuilt on the first search or warm-up after the
upgrade; no schema or version change is involved.
* style(tests): format the CJK fallback regression test
`ruff format --check .` in the backend lint job reformats the new assertion in tests/test_memory_retrieval_adapter.py: the implicitly concatenated message is joined into one line. Apply the formatter output, no behavior change.
* fix(deermem): recall single CJK characters through an FTS5 prefix phrase
The fallback index stores CJK runs as character bigrams, so a one-character
query had no equal token and came back empty; `"重"*` is an FTS5 prefix phrase
that matches the bigram `重试` and still matches a standalone `重` token, while
multi-character tokens keep the plain quoted phrase. Also force the no-jieba
branch with a monkeypatch so the regression test runs on installs that have
the optional `memory-zh` extra, and cover the single-character and absent-
character cases.
* docs(deermem): scope the single-CJK prefix note to leading matches
The prefix phrase only reaches indexed tokens that start with the queried
character, so a character occurring only later inside a bigram (务 in 任务)
is not matched. Say that in the docstring and mark the residual gap as
accepted instead of widening the prefix to the run's trailing character.
* fix(memory): select FTS5 matches by composite score
Score every SQL-filtered match before selecting the final composite top-k.
Keep candidate retention bounded with a streaming heap and preserve the
existing scoring formula and adapter-level filtering behavior.
Add regressions for confidence and time-decay reversals, category filters,
result prefixes, ties, and the public DeerMem storage path.
Fixes#8003
* fix(memory): report matched and retained FTS5 rows
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): read the complete scope in the substring search fallback
_search_substring iterated list_facts without paginating, so the
adapter-less fallback only ever saw the default page of 100 facts.
With 150 matching facts in a scope (max_facts allows up to 500),
memory_search returned 100 results for top_k=150 and silently dropped
every match past the first page; DeerMem.search keeps these results
whenever they are non-empty, so its uncapped fallback never ran. Read
the complete scope instead, as the scoped index rebuild already does.
* test(memory): drop in-page fact-100 from the past-page assertion
fact-100 was the last id inside list_facts' old first page
(``matched[0:100]`` in insertion order), so its presence does not
distinguish the fixed search from the pre-fix page; fact-101 and
fact-150 (plus the ``len(results) == 150`` check) carry the regression
signal. Keep a boundary note so the remaining ids are not misread.
or 0.5 cannot distinguish an explicit 0.0 from an unset field, so a discounted fact got the default confidence weight in both the FTS5 document and the substring fallback, while the same row indexed through FTS5Retrieval.index_fact kept its zero weight.
* feat(memory): integrate FTS5 retrieval adapter
* deps: add jieba as default dependency for Chinese tokenization
Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.
* fix(memory): avoid treating hyphens as FTS5 operators
* feat(memory): make Chinese tokenization optional
* fix(memory): warm every requested retrieval scope
* fix(memory): close retrieval resources on shutdown
* fix(memory): close backend when shutdown flush fails
* fix(memory): recreate corrupt retrieval index
* fix(memory): tolerate partial retrieval rebuilds
* fix(memory): warm retrieval index in background
* fix(memory): preserve shutdown flush budget
* fix(memory): stop retrying partial lazy rebuilds
* fix(memory): close retrieval through storage
* refactor(memory): simplify retrieval scope limit
* docs(memory): clarify retrieval shutdown lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
