mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-12 22:20:33 +00:00
54753636b1
* fix(memory): recall CJK sub-phrases when jieba is not installed Without the memory-zh extra, _tokenize fell back to whitespace splitting, so the FTS5 index kept a whole CJK run as a single unicode61 token and every CJK sub-phrase query returned no hits without any warning. The relevance ranker already emitted CJK character bigrams for the same text, so the two retrieval strategies disagreed on the same data. Reuse relevance.cjk_token for CJK runs when jieba is unavailable and route _preprocess_content through _tokenize, so the index and the query side agree. Installs with jieba keep the existing jieba path. Existing index rows are rebuilt on the first search or warm-up after the upgrade; no schema or version change is involved. * style(tests): format the CJK fallback regression test `ruff format --check .` in the backend lint job reformats the new assertion in tests/test_memory_retrieval_adapter.py: the implicitly concatenated message is joined into one line. Apply the formatter output, no behavior change. * fix(deermem): recall single CJK characters through an FTS5 prefix phrase The fallback index stores CJK runs as character bigrams, so a one-character query had no equal token and came back empty; `"重"*` is an FTS5 prefix phrase that matches the bigram `重试` and still matches a standalone `重` token, while multi-character tokens keep the plain quoted phrase. Also force the no-jieba branch with a monkeypatch so the regression test runs on installs that have the optional `memory-zh` extra, and cover the single-character and absent- character cases. * docs(deermem): scope the single-CJK prefix note to leading matches The prefix phrase only reaches indexed tokens that start with the queried character, so a character occurring only later inside a bigram (务 in 任务) is not matched. Say that in the docstring and mark the residual gap as accepted instead of widening the prefix to the run's trailing character.