142 Commits

Author SHA1 Message Date
yi-coding
94447d12b6
fix(memory): preserve summaries when extraction returns invalid types (#8192)
Co-authored-by: xiaoyi <xyi74976-del@users.noreply.github.com>
2026-11-13 14:13:47 +08:00
Wistom Zhang
2ff88fbb29
feat(memory): opt-in reserved categories for DeerMem hybrid-v1 eviction (#8194)
* feat(memory): support category capacity reservation in DeerMem hybrid-v1 eviction

In long-running personal assistant interactions, core user profile facts
(such as health allergies, coding style, or identity constraints) can suffer
from capacity starvation as high-confidence conversational context accumulates.

This change introduces opt-in category capacity reservation for DeerMem:
- In DeerMemConfig and select_facts_for_capacity, support eviction_reserved_categories
  (defaulting to ['correction'] for historical compatibility) and
  eviction_decay_exempt_categories (defaulting to empty list).
- Wire MemoryUpdater to pass the configured categories to select_facts_for_capacity.
- Add deterministic offline unit tests in test_memory_personal_assistant_protection.py
  verifying root-cause reproduction, bidirectional lifecycle (clean explicit removal
  and contradiction replacement), and strict backward compatibility under defaults.

* fix(memory): preserve correction priority in reserved capacity

Co-authored-by: Wistom Zhang <77424093+Yakyou7@users.noreply.github.com>

---------

Co-authored-by: Totoro <msy626836554@gmail.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
2026-11-13 11:09:41 +08:00
Harvey DU
eb97327ddf
fix(memory): reject duplicate fact ids in imports regardless of entry order (#8153)
* fix(memory): reject duplicate fact ids in imports regardless of entry order

A backup containing two entries for the same fact id was accepted or
rejected depending only on their order. The duplicate check in
_commit_changes_locked consulted the map of prepared writes, but an
entry whose normalized content matches the stored fact is skipped from
that map - so an unchanged entry followed by an edited duplicate passed
the check while the reverse order raised, and the saved snapshot ended
up with whichever entry came last.

Track every incoming id in a seen-set for duplicate detection, before
the unchanged-entry skip. The saved snapshot now survives the rejection
in both orders.

Fixes #8150

* fix(memory): validate duplicate import ids before capacity selection

Copilot review follow-up on #8153: the storage-level duplicate guard runs
after _select_for_capacity, and the capacity policy can drop one of two
same-id entries (e.g. lower-confidence copies trimmed to max_facts) - the
duplicate then never reaches the guard and the import outcome still
depends on entry order.

Validate all normalized incoming ids in import_memory_data before
capacity selection. The storage-level guard stays as defense in depth.
Regression test with max_facts=10 and 9 distinct high-confidence facts
plus two same-id copies.

* fix(memory): surface duplicate fact id errors with their own detail

willem-bd review on #8153, three follow-ups:

1. P2: the bare ValueError fell through to the router's generic catch, so
   POST /api/memory/import answered with the facts-array message even
   when every fact had non-empty content. Add
   MemoryDuplicateFactIdError (same type as #8151) at the manager
   boundary and map it to its own 400 detail that names the duplicated
   id; router test asserts the exact detail.

2. P3: the duplicate check lived inside the apply_changes-override
   branch, so the base MemoryStorage save() path bypassed both guards.
   Move it before the branch split, right after normalize_memory_data,
   matching #8151's validate-before-capacity posture.

3. P3: parametrize the order-dependence regression over the default
   bucket (agent_name=None) and "researcher", and assert the seeded
   snapshot survives the rejection in both orders.

* test(memory): pin the storage duplicate guard and fix the no-op assertion

willem-bd review follow-ups on #8153:

1. The snapshot-survival assertion compared storage.load against
   get_memory_data, which is itself storage.load - both reads happened
   after the rejected import, so a regression that partially applied the
   duplicate would pass. Capture the seeded snapshot before the rejected
   import and compare against it.

2. The updater gate rejects import duplicates before storage is touched,
   so nothing exercised the storage-level guard in _commit_changes_locked
   with an unchanged duplicate first - reverting it would keep the suite
   green. Add a direct apply_changes regression test; verified it fails
   with the guard removed.

* style(memory): apply ruff fixes after #8151 merge resolution

* test(memory): assert untouched store in the storage-guard pin; facade import

willem-bd review follow-ups on #8153:

1. The storage-guard regression test pinned rejection but not the
   untouched-store half: capture the seeded snapshot right after seeding
   and assert storage.load still equals it after the rejection, so a
   regression that partially applies the first entry before raising
   fails.

2. The router regression test imported MemoryDuplicateFactIdError from
   the manager module directly; use the package-facade import consistent
   with the other manager-error usage (the symbol is already re-exported
   from deerflow.agents.memory since #8151).

* test(memory): drop the mock router duplicate-id test

ShenAC-SAC review on #8153: #8151 already drives the real import route
with duplicate IDs and asserts the same 400 response plus an unchanged
snapshot, and the mock test's docstring implied a detail the assertion
did not check. The direct apply_changes() regression is kept.

Also removes the blank line after the docstring that tripped
ruff format --check on the repo-pinned Ruff version.

* test(memory): remove the orphaned mock-router test body

ShenAC-SAC review: my earlier deletion removed the function definition
and docstring but left the test body running inside
test_clear_route_keeps_working_for_windows_reserved_agent_name. Remove
the whole block and the now-unused MemoryDuplicateFactIdError import;
#8151 already covers the duplicate-ID response through the real import
route.
2026-11-12 19:23:44 +08:00
ZephyrWang
b80191fad3
fix(memory): reject duplicate fact IDs during import (#8151)
* fix(memory): reject duplicate fact IDs during import

* fix(memory): explain duplicate IDs in import response

* docs(changelog): link duplicate import fix PR

* fix(memory): keep duplicate validation portable

---------

Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 11:42:55 +08:00
Wenchao An
9699f04cdc
feat(projects): add visible project memory with extraction controls (#8117)
* feat(projects): add visible project memory with extraction controls

* fix(projects): avoid replaying committed memory extractions

* chore(ci): retrigger project memory checks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 09:09:44 +08:00
Xinmin Zeng
2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00
PeaceMaker-best
91f4fb4570
fix(memory): preserve retrieval indexes on operational startup failures (#8084)
Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-11-10 15:14:52 +08:00
shunxinyi
54753636b1
fix(memory): recall CJK sub-phrases when jieba is not installed (#8022)
* fix(memory): recall CJK sub-phrases when jieba is not installed

Without the memory-zh extra, _tokenize fell back to whitespace splitting, so
the FTS5 index kept a whole CJK run as a single unicode61 token and every CJK
sub-phrase query returned no hits without any warning. The relevance ranker
already emitted CJK character bigrams for the same text, so the two retrieval
strategies disagreed on the same data.

Reuse relevance.cjk_token for CJK runs when jieba is unavailable and route
_preprocess_content through _tokenize, so the index and the query side agree.
Installs with jieba keep the existing jieba path.

Existing index rows are rebuilt on the first search or warm-up after the
upgrade; no schema or version change is involved.

* style(tests): format the CJK fallback regression test

`ruff format --check .` in the backend lint job reformats the new assertion in tests/test_memory_retrieval_adapter.py: the implicitly concatenated message is joined into one line. Apply the formatter output, no behavior change.

* fix(deermem): recall single CJK characters through an FTS5 prefix phrase

The fallback index stores CJK runs as character bigrams, so a one-character
query had no equal token and came back empty; `"重"*` is an FTS5 prefix phrase
that matches the bigram `重试` and still matches a standalone `重` token, while
multi-character tokens keep the plain quoted phrase. Also force the no-jieba
branch with a monkeypatch so the regression test runs on installs that have
the optional `memory-zh` extra, and cover the single-character and absent-
character cases.

* docs(deermem): scope the single-CJK prefix note to leading matches

The prefix phrase only reaches indexed tokens that start with the queried
character, so a character occurring only later inside a bigram (务 in 任务)
is not matched. Say that in the docstring and mark the residual gap as
accepted instead of widening the prefix to the run's trailing character.
2026-11-10 09:50:46 +08:00
lilei
31d97fc85b
fix(memory): select FTS5 matches by composite score (#8005)
* fix(memory): select FTS5 matches by composite score

Score every SQL-filtered match before selecting the final composite top-k.
Keep candidate retention bounded with a streaming heap and preserve the
existing scoring formula and adapter-level filtering behavior.

Add regressions for confidence and time-decay reversals, category filters,
result prefixes, ties, and the public DeerMem storage path.

Fixes #8003

* fix(memory): report matched and retained FTS5 rows

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-09 21:16:18 +08:00
Eilen Shin
aaca3eeaeb
refactor(agent): ground runtime decisions in structured evidence (#7986)
* refactor(agent): ground runtime decisions in structured evidence

Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.

Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.

* test(sandbox): expect read_file initialization error callback

* fix(agent): address evidence review follow-ups

* fix(agent): preserve retry budgets for OpenAI transport errors

* fix(agent): expose tool execution status in model requests

Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-08 22:00:23 +08:00
yetuge
5b1bee3d5a
fix(memory): skip text-less turns when formatting the memory prompt (#7985)
* fix(memory): skip text-less turns when formatting the memory prompt

A content-less message rendered as the literal "None", and a block list
without text blocks rendered as its Python repr, including any base64
data URL. Normalize through the same rules as the sibling content-to-text
helpers and skip turns that carry no text.

* test(memory): cover whitespace-only assistant turns

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-08 13:43:06 +08:00
Willem Jiang
a99cd299f9
docs(memory): trim inherited guidance below hard limit (#7976) 2026-11-07 19:08:05 +08:00
liyinlong
8d7e48e5e7
fix(memory): restore facts from older backups (#7921)
* fix(memory): restore facts from older backups

Use the current snapshot revision for existing facts during backup import so saved contents can be restored after later edits. Preserve manifest and fact preconditions, scope isolation, and caller-owned input data.

* refactor(memory): share import revision preconditions

* docs(memory): fit import guidance within chain budget

* fix(memory): validate backup revisions before restoration

---------

Co-authored-by: liyinlong <173714821+lilong555@users.noreply.github.com>
2026-11-06 21:39:18 +08:00
hdu_zhou
9e2cf685b0
fix(memory): treat a content-less deermem message as empty, not None (#7854)
extract_message_text falls back to str(content), so a message whose content is None becomes the truthy literal 'None'. That string then reaches signal detection and the memory-update prompt as if the user had written it. The canonical helper (utils/messages.py:message_content_to_text) and the mem0 copy both return '' for this shape and the canonical one documents exactly this failure mode.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-05 12:25:51 +08:00
Cordis798
6538496dbc
fix(memory): apply Honcho read policies to malformed search metadata (#7890)
Co-authored-by: Cordis798 <284918518+Cordis798@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-05 07:04:15 +08:00
一片一片
38e6be638b
fix(memory): tolerate unbounded stored fact confidence (#7764)
* fix(memory): tolerate unbounded stored fact confidence

A stored memory fact can carry a JSON integer such as 10**400. float() on it
raises OverflowError in _coerce_source_confidence and in the dedup merge
comparison, aborting the whole update cycle even though both sites already
handle every other unusable confidence. Treat the overflow like a missing or
non-finite value: 0.5 for ranking, 0.0 for the merge comparison.

* fix(memory): harden the remaining fact-confidence coercions

The import normalizer, prompt builder and relevance scorer each coerce a
stored fact confidence with float() and caught only TypeError/ValueError, so
an unbounded JSON integer such as 10**400 raised OverflowError: import failed
before the updater fallbacks ran, and prompt/relevance ranking could abort.
Catch OverflowError and take the existing defaults; document the full set of
coercions.
2026-11-01 15:04:16 +08:00
haotianqi666
f7da737118
fix(memory): avoid replaying Honcho checkpoint history (#7687)
* fix(memory): avoid replaying Honcho checkpoint history

* fix(memory): harden Honcho capture lookups

---------

Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
2026-10-30 07:33:41 +08:00
LingYi
c28e64fc8f
fix(memory): enforce the token budget when truncating summaries (#7709)
Signed-off-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
2026-10-30 07:32:51 +08:00
Cordis798
09a98e5f7b
fix(memory): honor Mem0 read policies during context deduplication (#7561)
Co-authored-by: Cordis798 <284918518+Cordis798@users.noreply.github.com>
2026-10-26 19:44:01 +08:00
Hyeonsang Cho
30b3a012cb
fix(memory): flush pre-compaction memory off the event loop (#7530)
* fix(memory): flush pre-compaction memory off the event loop

The async compaction path fired the before-summarization hooks inline.
memory_flush_hook hands the summarized messages to add_nowait, which is
sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking
(its own override), so every automatic compaction with an HTTP memory
backend blocked the Gateway event loop for the whole request.

acompact_state now runs the hooks through asyncio.to_thread, as
MemoryMiddleware.aafter_agent already does for its writes. Ordering,
failure isolation, summary-first timing and context propagation are
unchanged; the sync compact_state path is untouched.

* docs(changelog): reference #7530 for the pre-compaction memory flush fix

* test(memory): anchor the OpenViking pre-compaction flush off the event loop

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-26 15:13:00 +08:00
jianyx1
4b1e06bbf9
feat(runtime): honor the web UI language in replies and DeerMem updates (#7294)
* feat(runtime): honor the web UI language in replies and DeerMem updates

Carry Settings > Appearance > Language as request-only run data: the
Gateway validates context.locale against a finite allowlist, the lead
agent prefers it for replies, and bundled DeerMem writes new facts and
summary prose in it. Explicit user language requests still take
precedence; checkpoint configurable, static prompts, and memory backend
APIs are unchanged.

Refs #7287

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(runtime): guard language context and pin locale maps

* test(runtime): pin UI locales across frontend and Gateway

Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-26 13:24:30 +08:00
PerryLink
338cd26dc2
fix(memory): drop unusable numbers in stale extensions and consolidation confidence (#7442)
* fix(memory): drop unusable numbers in stale extensions and consolidation confidence

* fix(memory): log defaulted consolidation confidence and pin the int passthrough
2026-10-25 16:16:09 +08:00
Lym308758
346ff7b253
fix(memory): rank capacity eviction per record, not per fact_id (#7454)
* fix(memory): rank capacity eviction per record, not per fact_id

select_facts_for_capacity keyed every score by fact_id, so an imported
batch that repeats an id collapsed into a single dict entry: the last
record's score won, and both the ranking pass and the evicted audit
read that one value for every position. A 0.9-confidence record could
therefore be evicted while its own 0.1-confidence duplicate survived.
The import path keeps caller-supplied ids without checking them for
uniqueness, and any over-cap import lands here.

Score by position instead: a parallel list next to the indexed facts,
so ranking and the audit read each record's own score. The public
``scores`` dict stays keyed by fact_id with last-write-wins
semantics, keeping the DeerMem benchmark contract and the callers
that never read it unchanged.

Red on main: test_confidence_policy_scores_duplicate_fact_ids_per_record
fails with the highest-confidence record evicted; green on this branch.

* fix(utils): keep share-less UNC roots out of the extended-length prefix

extended_length_path prefixed every absolute UNC path with
\\?\UNC\, but Win32 only accepts \\?\UNC\server\share\... — a
share-less root (\\server, \\server\, or a bare \) is not a valid
device path, so every Win32 call handed the prefixed spelling fails.
Only prefix when both the server and the share are present;
otherwise return the ordinary spelling unchanged.

Red on main: the \\server, \\server\, //, and //server cases of
test_windows_extended_spelling assert the forged prefix; green on
this branch.
2026-10-25 13:20:15 +08:00
Cordis798
3e9a417ea6
fix(memory): apply Mem0 read policies to record conversion failures (#7307)
* fix(memory): apply Mem0 read policies to record conversion failures

* test(memory): cover Mem0 read policies through routes and tools

* fix(memory): identify malformed Mem0 records in diagnostics

---------

Co-authored-by: Cordis798 <284918518+Cordis798@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-23 17:57:14 +08:00
Fish
7097694abe
fix(memory): honor Mem0 failure policies for malformed JSON responses (#7202)
* fix(memory): honor Mem0 failure policies for malformed JSON responses

* fix(memory): diagnose and normalize deep Mem0 response failures

* fix(memory): include Mem0 page context in response errors

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-23 16:58:19 +08:00
Fish
97b12b1381
fix(memory): exclude hidden framework messages from Honcho writes (#7159)
* fix(memory): exclude hidden framework messages from Honcho writes

* fix(memory): isolate hidden-message hook failures
2026-10-22 16:51:47 +08:00
Nan Gao
dc6f0d7421
fix(memory): persist the DeerMem extraction watermark across restarts and replicas (#7205)
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas

The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.

FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.

* docs(changelog): link the memory watermark entry to #7204 and #7205

* fix(memory): never replace an unreadable watermark sidecar

set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.

* fix(memory): keep the persisted watermark when the last message has no identity

_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
2026-10-21 07:26:44 +08:00
李刚 / Bryan Li
1fd266e0c0
feat(decisions): add an OpenAI Decisions backend for the decision slots (#7127)
* feat(decisions): add an OpenAI Decisions backend for the decision slots

Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.

Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.

* fix(decisions): isolate endpoint identities and correct diagnostics

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-20 11:50:46 +08:00
LuckVd
2174c1d81e
fix(memory): quarantine unreadable memory summaries (#7076)
* fix(memory): quarantine unreadable memory.json in the read-error branch too

MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.

Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.

Fixes #7072

* fix(memory): preserve summaries on transient read failures

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-20 11:25:01 +08:00
LingYi
ee02e51df9
fix(memory): preserve consecutive pre-compaction snapshots (#7152)
* fix(memory): preserve consecutive pre-compaction snapshots

* fix(memory): warn on growing emergency snapshot backlogs

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-20 10:24:16 +08:00
yetuge
18ca09b2da
fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption (#7116)
* fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption

Facts are hand-editable Markdown files, so a Windows editor can save one as
"UTF-8 with BOM" -- the same exposure #7083 fixed for memory.json. The fact
parser checks `text.startswith("---\n")` against a plain-utf-8 read, so the
BOM makes a valid fact read as "missing YAML front matter" and raises
MemoryStorageCorruption, which the module docs describe as taking down the
whole agent.

Read the fact with utf-8-sig, matching the summary loader and the SOUL.md /
SKILL.md convention already recorded in AGENTS.md. `_parse_listed_fact`
delegates here, so the directory-scan path is covered by the same change.

Measured: the new BOM test fails on the base revision with exactly that
"missing YAML front matter" corruption and passes at head; the no-BOM control
passes both. 49 tests in test_memory_storage.py, 106 across the memory
suites, ruff check/format clean. Mutation-checked: reverting to plain utf-8
fails only the BOM case.

* test(memory): cover BOM facts through the public storage API

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-19 23:27:04 +08:00
Jay0130-a
0a1c86a371
fix(memory): parse a UTF-8 BOM in the DeerMem summary instead of raising corruption (#7083) (#7084) 2026-10-19 14:44:15 +08:00
哈基米
816ebd5563
fix(memory): filter honcho search by category instead of stamping it on unfiltered results (#6981)
* fix(memory): filter honcho search by category instead of stamping it on unfiltered results

HonchoMemoryManager.search sent an identical unfiltered query whether or
not a category was requested, then stamped the requested category on
every hit, so memory_search(category=...) returned unfiltered results
attested to be that category (e.g. a work fact labeled "preference")
— the opposite of the contract, where category filters before the
top_k slice and only facts with that exact category are returned.

Send the category as a real server-side metadata filter (Honcho applies
"metadata" filters in its ranked query, before the limit) and label
each result from the record's own metadata, falling back to "memory"
as before. HonchoClient.search gains a keyword-only filters argument
that is only put on the wire when set, so unscoped requests are
byte-identical to before.

* docs(memory): note honcho category filtering requires stored metadata
2026-10-18 21:55:19 +08:00
ordinary being
cd0166ae76
fix(memory): drop out-of-range extraction confidence instead of failing the batch (#6971)
* fix(memory): drop out-of-range extraction confidence instead of failing the batch

A model-emitted confidence outside [0, 1] (a percentage such as 95, or 1.5)
passed extraction normalization, which only rejected non-finite values. The
value then reached storage, whose own validation raises for out-of-range
confidence, so one malformed field failed the whole update: the extraction
watermark never advanced and the same paid call repeated on every turn, while
every valid fact, summary and removal in the batch was lost each cycle.

Treat the range like the other malformed shapes in
_normalize_memory_update_fact: drop only that item and let the rest of the
update apply. Read paths already clamp stored corrupt values
(_coerce_source_confidence), so no read-side change is needed.

* fix(memory): bound the confidence drop and make it observable (review)

Review follow-ups on the out-of-range confidence fix:
- range-check integer confidence before float(), so a model emitting 10**400
  is dropped instead of raising OverflowError and failing the batch;
- log the raw value when a fact is dropped, so a systematically
  percentage-scale model is visible instead of silently losing every fact;
- document that a batch which also proposes factsToRemove still fails closed
  when any new fact is dropped (replacementFactIndex positions shift), and
  pin that boundary with a test. Values are documented as dropped, never
  rescaled.
2026-10-18 21:00:49 +08:00
万家良
4e6c18ad9c
fix(memory): retry the scope lock open to absorb a Windows create race (#6870)
On Windows, two threads/processes racing to create a not-yet-existing
lock file with an append open can transiently fail on open() itself with
PermissionError (EACCES), before the byte-range lock is ever attempted.
The open was outside the retry loop, so the error surfaced from
load()/save() instead of being retried. Move the open inside the loop and
close the handle on any OSError before retrying.
2026-10-14 16:10:56 +08:00
哈基米
f4d10ef42d
fix(memory): read the complete scope in the substring search fallback (#6732)
* fix(memory): read the complete scope in the substring search fallback

_search_substring iterated list_facts without paginating, so the
adapter-less fallback only ever saw the default page of 100 facts.
With 150 matching facts in a scope (max_facts allows up to 500),
memory_search returned 100 results for top_k=150 and silently dropped
every match past the first page; DeerMem.search keeps these results
whenever they are non-empty, so its uncapped fallback never ran. Read
the complete scope instead, as the scoped index rebuild already does.

* test(memory): drop in-page fact-100 from the past-page assertion

fact-100 was the last id inside list_facts' old first page
(``matched[0:100]`` in insertion order), so its presence does not
distinguish the fixed search from the pre-fix page; fact-101 and
fact-150 (plus the ``len(results) == 150`` check) carry the regression
signal. Keep a boundary note so the remaining ids are not misread.
2026-10-12 22:54:02 +08:00
哈基米
1b7ca5ecf1
fix(memory): keep non-core fact categories instead of rewriting to other (#6733)
* fix(memory): keep non-core fact categories instead of rewriting to other

_normalize_category rewrote any category outside CORE_CATEGORIES to
"other" and parked the original label in a write-only
categoryExtension field, so memory_add(category="personal") -- an
example label in the tool's own docstring -- persisted as "other" and
memory_search(category="personal") ("Only facts with this exact category
are returned") never matched it. get_memory documents kept reporting
"other" as well.

Keep the caller's label as the stored category so it round-trips
through search, documents, and the retrieval index.

* fix(memory): finish migrations interrupted across the category change

A pre-native-category Gateway committed a per-agent legacy JSON migration
as category="other" + categoryExtension=<label> and could stop before
deleting the legacy JSON (crash / failed unlink). The retained JSON holds
the label natively, so _migrate_locked compares "personal" against its
own committed "other"/"personal" alias, raises MemoryStorageCorruption,
and fails every subsequent read, reload and migration retry for that
agent (HTTP 500 at the Gateway) -- a recoverable state the previous
build recovered from.

Fold the legacy alias into _migration_equivalent's category comparison
("other"/<label> equals <label>/no-extension) so retries recognize and
complete recovery, while a genuinely different label still stops the
migration with the legacy file preserved.

Also drop a stale categoryExtension when a fact is normalized away from
"other": _normalize_fact deep-copies unknown fields through, so a legacy
fact updated to its native category kept surfacing the leftover
extension, contradicting the Gateway contract that the field is only
set "when category is 'other'".

Both regressions are pinned by replaying the historical writer in tests.

---------

Co-authored-by: BlueX888 <BlueX888@users.noreply.github.com>
2026-10-12 22:45:08 +08:00
Jie Ji
bdc2d12367
fix(memory): prevent cross-conversation debounce starvation (#6672)
* fix(memory): prevent cross-conversation debounce starvation

* fix(config): bump schema version for memory max wait
2026-10-11 17:16:25 +08:00
Nan Gao
5ae953fc2f
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers

DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.

- New backend-private key memory.backend_config.retrieval_index_path: empty
  keeps {storage_path}/.retrieval, a relative path resolves against
  storage_path, an absolute path is used as-is. The full rebuild and the
  corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
  (mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
  scope whose live signature differs because another process wrote the user's
  memory. An own commit advances the recorded signatures of that user's scopes
  only when they were in sync at the previous revision, so own writes never
  rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
  declared multi-instance deployment keeps the DeerMem index inside
  storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
  the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
  it, drops the legacy memory.storage_path: memory.json line, and documents the
  memory layout next to the other multi-replica prerequisites.

* docs(changelog): link the memory retrieval index entry to #6494

* fix(memory): never advance the index signature on a no-op commit and resync whole scopes

Two review findings on the cross-replica retrieval re-sync:

- save() and apply_changes() inferred the replaced manifest revision as
  "returned revision - 1". The commit helper returns the existing manifest
  unchanged for a no-op (unchanged-value or empty change set), so after a peer
  bumped the manifest an own no-op write advanced this process's stale index
  signature to the peer's revision without carrying the peer's delta: a fact the
  peer deleted stayed searchable until the manifest changed again. Both paths
  now read the real pre-commit revision under the user lock and pass
  committed=(pre_revision, signature) only when the commit produced a new
  revision; a no-op passes None so the next search still rebuilds the scope.

- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
  listed facts through list_facts() with its default limit of 100 and then
  replaced every row of the scope, so a scope with more than 100 facts lost its
  tail on the first re-sync. It now reads the scope's complete fact list.

* fix(memory): fence index signature promotion against a concurrent scope rebuild

Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.

Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.

Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).

* fix(memory): publish a rebuild's signature under the lock that replaced its rows

Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.

Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).

Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.

Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.

* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error

Two suggestion-level review items on the cross-replica retrieval re-sync.

- The full rebuild cached manifest signatures keyed by (manifest path, agent
  name), but _scope_signature reads only the manifest file -- every agent
  bucket of a user shares one memory.json -- so the startup scan stat+read+
  parsed the same file once per bucket that has facts. The cache is now keyed
  by the manifest path alone while synced signatures are still recorded per
  (user, agent) scope: users manifest reads instead of users x agents.

- The per-search freshness compare put a shared-volume manifest read on every
  search_facts call, and _load_memory_file raises MemoryStorageCorruption on
  OSError and malformed JSON, so a transient network-filesystem read error or a
  corrupt memory.json failed a search the Pod-local index could still answer.
  The compare now catches OSError, ValueError and MemoryStorageCorruption: it
  logs a warning naming the scope (traceback at debug), serves the current
  local index for that scope and leaves the recorded synced signature
  untouched so the next search compares again. It deliberately does not mark
  the scope dirty: a scoped rebuild would hit the same failing read, forget the
  scope's signature and fail the search path too.

* docs(changelog): resolve merge markers left in the Unreleased section

* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible

Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.

Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.

Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.

Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 21:20:45 +08:00
Hyeonsang Cho
c15ecb688d
fix(migration): read thread owners from the configured database (#6450)
* fix(migration): read thread owners from the configured database

migrate_user_isolation.py looked for thread owners in
{base_dir}/deer-flow.db, a file DeerFlow never creates. The database is
{sqlite_dir}/deerflow.db or PostgreSQL, so the owner map was always empty
and every legacy thread was moved to users/default/.

Read threads_meta through the configured database's sync SQLAlchemy URL
(SQLite or PostgreSQL, honoring postgres_schema). When legacy threads
exist but owners cannot be read (no database file, a memory backend, a
missing driver, or a failed query), stop before migrating anything;
--allow-missing-thread-owners keeps the old assign-to-default behavior
for installs that never recorded owners.

* docs(changelog): reference #6450

* fix(migration): document recovery and report unusable database URLs

Installs that already ran the old script have every legacy thread under
users/default/threads/, and re-running finds no threads/ to migrate. The
provisioner README and AUTH_DESIGN now give the recovery: move the owned
threads back to threads/ and re-run; an existing owner directory is never
overwritten and the old copy goes to migration-conflicts/. A test pins
that re-run behavior.

create_engine errors other than ImportError (e.g. ArgumentError from an
empty postgres_url) now stop with the same "Nothing was migrated." message
instead of a traceback.

The memory AGENTS.md note said threads go to --user-id; they go to their
threads_meta owner.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 10:27:10 +08:00
ZJPex
220d62d79a
feat(memory): add scoped fact lookup by ID (#6443)
* feat(memory): add scoped fact lookup by ID

* docs(memory): synchronize tool-mode lookup guidance

* docs(memory): clarify named-agent read capability requirement
2026-10-08 09:25:12 +08:00
Guo-Yixin
789bbaaa0d
fix(memory): redact invalid and legacy tool-call arguments (#6387) 2026-10-07 07:23:56 +08:00
wgwfn3
c54883cce0
fix(memory): drop punctuation tokens from jieba relevance ranking (#6388)
* fix(memory): drop punctuation tokens from jieba relevance ranking

jieba.cut emits punctuation as standalone tokens and tokenize() only filtered whitespace, so commas and full stops counted as query matches, inflated near-duplicate similarity, and consumed the 128-token budget. Keep only tokens with an alphanumeric character, matching the regex fallback and the FTS5 adapter's query filter.

Document the rule in agents/memory/AGENTS.md.

* docs(changelog): add the DeerMem jieba punctuation fix

* docs(changelog): add the PR reference for the jieba punctuation fix (#6388)

* docs(memory): document jieba relevance token filtering

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-07 06:58:59 +08:00
Guo-Yixin
1523a94551
fix(memory): exclude invalid and raw assistant tool calls (#6379) 2026-10-06 20:43:27 +08:00
Hyeonsang Cho
1f7600e355
fix(memory): treat facts deleted during unlocked DeerMem scans as removed (#6255)
* fix(memory): treat facts deleted during unlocked DeerMem scans as removed

load(), reload(), and the full rebuild_index() scan list agent fact
files without the storage locks. A delete committed between the listing
and opening a file raised MemoryStorageCorruption for intact data, which
surfaced as HTTP 500 from the memory API and an empty memory block in
the prompt. A listed fact that no longer exists is now skipped; an entry
that is still present but unreadable is still corruption.

* docs(changelog): reference #6255 for the DeerMem vanished-fact fix

* docs(memory): note lock coverage and heuristic limits of the vanished-fact skip
2026-10-04 12:37:33 +08:00
Hyeonsang Cho
e3064f541b
fix(memory): sign DeerMem reload before reading the document (#6238)
* fix(memory): sign DeerMem reload before reading the document

reload() read the document before computing its cache signature, so a
write committed in between cached the old document under the new
signature and later load() calls served stale memory until the next
write. Compute the signature first, matching load().

* test(memory): cover markdown storage and agent scope in reload race test

* docs(changelog): reference #6238 for the DeerMem reload fix
2026-10-04 07:29:56 +08:00
Grapette.L
f61372764e
fix(agents): anchor agent-name validation against a trailing newline (#5960)
* fix(agents): anchor agent-name validation against a trailing newline

Two copies of the shared ^[A-Za-z0-9-]+$ grammar still used re.match,
whose $ also matches before a final newline, while every other copy uses
fullmatch. POST /api/agents {"name": "reviewer\n"} therefore passed the
router guard and died inside the strict file store as an HTTP 500, and
DeerMem accepted the same name as an agents/{name}/facts directory.

* test(agents): state which trailing-newline params the regression actually covered

.match only accepted "reviewer\n" on main; "reviewer\n\n" and "reviewer \n"
were already rejected, so the docstrings should not read as if every param in
these parametrizations was broken before the fix.
2026-09-30 19:38:09 +08:00
PeaceMaker-best
b73a4e79bf
feat(memory): scope management API by agent (#5565)
* feat(memory): scope management API by agent

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(memory): preserve scoped management isolation

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-09-30 16:50:07 +08:00
Zeren Wang
d700ad9edf
feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906)
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client

- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
  configuration: one connection-precedence rule, credential fingerprint, wire-size
  counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
  DeerMem through one injected judge hook (both off by default, `enforce` gated by
  the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
  `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
  the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
  per-stratum gates, censoring and the no-network baseline.

Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
  upload-event scrub that precedes persistence, only removals/reinforcements of ids
  that existed, and persisted near-duplicate merges (matched by id, or by identity
  for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
  `typesafe` block changes (including a credential rotation); publication is
  serialized and revalidates the config generation, so a build started before an
  `enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
  the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
  and correction.

* fix(typesafe): resolve one credential per layer and refuse unusable connection values

Review follow-ups on the shared TypeSafe client (#5906):

- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
  shared `typesafe:` block: the first layer that sets either credential setting
  decides it, and inside one layer the literal key wins (the standalone
  semantics). A consumer whose endpoint does not accept the block's key no longer
  sends it on every call, and the missing-credential message keeps naming the
  variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
  fragment, embedded credentials, whitespace or control characters, and a
  malformed port fails at construction instead of on every request (httpx raises
  `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
  swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
  a secret mounted from a file, a non-printable character — is refused when the
  connection resolves. It used to construct fine and then put the whole
  `Bearer <key>` value into h11's `LocalProtocolError` message, which the
  guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
  so that message can no longer carry the credential through the cause chain; the
  exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
  path: `max_attempts: true` was silently read as `1`. An integer stays a usable
  float, so `timeout: 5` keeps working.

Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.

* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence

Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.

Typesafe client (the reply is treated as hostile input too):

- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
  `Content-Encoding`, and read through a bounded `iter_bytes` that stops past
  `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
  decoded in full and, on the gate path, allowed the tool call unexamined; it is
  now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
  escaped the module's taxonomy, so the memory consumers lost the round's record
  instead of recording a fallback.

Memory judge:

- A request-level failure is recorded as `request_failed`, never `no_verdict`
  (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
  endpoint was indistinguishable from a provider with no usable answer). The
  adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
  and log; `no_verdict` stays for a question-level failure, and an ineligible side
  keeps its own reason even when the round's request failed.

Shadow evaluation (the enablement gate reads the evidence, not one number):

- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
  meet the 1% target; a sample that only meets the point estimate is
  `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
  the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
  treated as absent rather than scored: `-3` read as "nothing lost", adding a
  phantom saving to the numerator and the denominator.

Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.

Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.

* fix(memory-eval): pair the baseline comparison on the same records

Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).

Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).

- `Evidence` carries the paired numbers (`baseline_paired_skips`,
  `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
  heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
  when the annotation covers only a subset of the scored skips (the message carries
  the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
  paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
  `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.

Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.

* fix(typesafe): bound the sync body read by the deadline

`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.

`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.

* fix(memory): read the YAML off spelling, and veto a skip over the whole batch

Two review findings on the memory judging path.

`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.

The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.

Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
2026-09-27 20:28:05 +08:00
Shxiao
230d8f03d7
fix(memory): validate Honcho's allow_insecure_http as a boolean (#5887)
* fix(memory): validate Honcho's allow_insecure_http as a boolean

* docs(memory): list the full Pydantic boolean vocabulary for allow_insecure_http
2026-09-27 11:18:03 +08:00