wgwfn3 c54883cce0
fix(memory): drop punctuation tokens from jieba relevance ranking (#6388)
* fix(memory): drop punctuation tokens from jieba relevance ranking

jieba.cut emits punctuation as standalone tokens and tokenize() only filtered whitespace, so commas and full stops counted as query matches, inflated near-duplicate similarity, and consumed the 128-token budget. Keep only tokens with an alphanumeric character, matching the regex fallback and the FTS5 adapter's query filter.

Document the rule in agents/memory/AGENTS.md.

* docs(changelog): add the DeerMem jieba punctuation fix

* docs(changelog): add the PR reference for the jieba punctuation fix (#6388)

* docs(memory): document jieba relevance token filtering

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-07 06:58:59 +08:00
..