shunxinyi 54753636b1
fix(memory): recall CJK sub-phrases when jieba is not installed (#8022)
* fix(memory): recall CJK sub-phrases when jieba is not installed

Without the memory-zh extra, _tokenize fell back to whitespace splitting, so
the FTS5 index kept a whole CJK run as a single unicode61 token and every CJK
sub-phrase query returned no hits without any warning. The relevance ranker
already emitted CJK character bigrams for the same text, so the two retrieval
strategies disagreed on the same data.

Reuse relevance.cjk_token for CJK runs when jieba is unavailable and route
_preprocess_content through _tokenize, so the index and the query side agree.
Installs with jieba keep the existing jieba path.

Existing index rows are rebuilt on the first search or warm-up after the
upgrade; no schema or version change is involved.

* style(tests): format the CJK fallback regression test

`ruff format --check .` in the backend lint job reformats the new assertion in tests/test_memory_retrieval_adapter.py: the implicitly concatenated message is joined into one line. Apply the formatter output, no behavior change.

* fix(deermem): recall single CJK characters through an FTS5 prefix phrase

The fallback index stores CJK runs as character bigrams, so a one-character
query had no equal token and came back empty; `"重"*` is an FTS5 prefix phrase
that matches the bigram `重试` and still matches a standalone `重` token, while
multi-character tokens keep the plain quoted phrase. Also force the no-jieba
branch with a monkeypatch so the regression test runs on installs that have
the optional `memory-zh` extra, and cover the single-character and absent-
character cases.

* docs(deermem): scope the single-CJK prefix note to leading matches

The prefix phrase only reaches indexed tokens that start with the queried
character, so a character occurring only later inside a bigram (务 in 任务)
is not matched. Say that in the docstring and mark the residual gap as
accepted instead of widening the prefix to the run's trailing character.
2026-11-10 09:50:46 +08:00
..