d700ad9edf
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client - refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its configuration: one connection-precedence rule, credential fingerprint, wire-size counting and error taxonomy, shared by the guardrail gate and the memory hooks. - feat(memory): wire `memory.prescreen` and `memory.signal_classification` into DeerMem through one injected judge hook (both off by default, `enforce` gated by the shadow evaluation). - config: add the top-level `typesafe:` block and the two judging slots to `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync the Helm README example with it. - test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with per-stratum gates, censoring and the no-network baseline. Review follow-ups folded in: - `mutations_accepted` counts only facts that survive capacity enforcement and the upload-event scrub that precedes persistence, only removals/reinforcements of ids that existed, and persisted near-duplicate merges (matched by id, or by identity for legacy id-less facts); it is published only after persistence succeeds. - The cached manager hot-reloads its judge when a memory slot or the shared `typesafe` block changes (including a credential rotation); publication is serialized and revalidates the config generation, so a build started before an `enforce` -> `off` rollback cannot land after it. - Cache provenance is per answer: the `cached` flag and the reported model follow the answers a side actually consumed, across partial merges and answerless retries. - An oversized noul integer stays a per-question error instead of raising. - Incremental call savings use the shared scored denominator. - The sensitive-stratum gate requires evidence for each of identity, preference and correction. * fix(typesafe): resolve one credential per layer and refuse unusable connection values Review follow-ups on the shared TypeSafe client (#5906): - A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the shared `typesafe:` block: the first layer that sets either credential setting decides it, and inside one layer the literal key wins (the standalone semantics). A consumer whose endpoint does not accept the block's key no longer sends it on every call, and the missing-credential message keeps naming the variable that layer configured. - `base_url` is validated where it is configured: `http(s)` only, with no query, fragment, embedded credentials, whitespace or control characters, and a malformed port fails at construction instead of on every request (httpx raises `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been swallowed into the appended endpoint path). - A credential that cannot travel as a header value — surrounding whitespace from a secret mounted from a file, a non-printable character — is refused when the connection resolves. It used to construct fine and then put the whole `Bearer <key>` value into h11's `LocalProtocolError` message, which the guardrail middleware's `logger.exception` prints. - The client raises transport failures without chaining the original exception, so that message can no longer carry the credential through the cause chain; the exception type stays in the message for diagnosis. - `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config` path: `max_attempts: true` was silently read as `1`. An integer stays a usable float, so `timeout: 5` keeps working. Tests: 20 cases across `tests/test_typesafe_config.py` and `tests/test_typesafe_client.py`, each red on the previous revision and green now. 367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean. Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`, `config.example.yaml`. * fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence Second review round on #5906: the remaining items from the review thread and from the follow-up pass. Typesafe client (the reply is treated as hostile input too): - Send `Accept-Encoding: identity`, refuse a body that carries a non-identity `Content-Encoding`, and read through a bounded `iter_bytes` that stops past `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be decoded in full and, on the gate path, allowed the tool call unexamined; it is now `invalid_response` (a denial with `fail_closed: true`). - Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting escaped the module's taxonomy, so the memory consumers lost the round's record instead of recording a fallback. Memory judge: - A request-level failure is recorded as `request_failed`, never `no_verdict` (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable endpoint was indistinguishable from a provider with no usable answer). The adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record and log; `no_verdict` stays for a question-level failure, and an ineligible side keeps its own reason even when the round's request failed. Shadow evaluation (the enablement gate reads the evidence, not one number): - The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to meet the 1% target; a sample that only meets the point estimate is `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound). - New `baseline_incremental_savings` gate: a baseline that saves as many calls as the model — the shipped all-trivial sample — fails instead of enabling. - Negative counts (`mutations_accepted`, message / batch / token counts) are treated as absent rather than scored: `-3` read as "nothing lost", adding a phantom saving to the numerator and the denominator. Docs: the design documents cited but absent from the repository are replaced with in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`, `GUARDRAILS.md`), and the gate and limit wording now matches what the code enforces. Tests: 31 new or updated cases, each red on the previous revision (compressed body refused even when it decodes to a valid envelope, body over the cap refused, body at the cap answered, deep nesting mapped, `request_failed` on each request-level failure, the bound and baseline gates, the negative counter). 454 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` and `ruff format --check` are clean. * fix(memory-eval): pair the baseline comparison on the same records Review follow-up on the incremental-savings gate added in ed930478 (P2): both sides of the comparison are now the records the no-network baseline can score (`trivial_only` plus a readable signals list). Subtracting that subset's heuristic savings from model savings over *all* scorable skips credited the model with batches the baseline never saw: 300 reviewed skips with a single annotated record computed 300-1=299 "incremental", passed the gate and set `enable_recommended`, while the paired comparison was model=1 against heuristic=1 (incremental 0). - `Evidence` carries the paired numbers (`baseline_paired_skips`, `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one heuristic count, filled by `build_report` from the baseline summary. - The gate reports `INSUFFICIENT` when no scored skip carries the annotation and when the annotation covers only a subset of the scored skips (the message carries the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive paired gain, and `PASS` only on a positive gain over the full paired set. - Docs state the pairing/coverage requirement in `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring. Tests: the reviewer's counterexample as a report-level regression (it failed with `assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true), a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields. 344 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` / `ruff format --check` clean. * fix(typesafe): bound the sync body read by the deadline `deadline_seconds` only covered the sync path's headers, the completed body and the parse, so an endpoint that drips bytes just under the read timeout held the caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget. The read loop now checks the budget before every transport read, which bounds the overshoot to one read. The pieces are the transport's own reads, not re-chunked output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk, so a check at that granularity would never see a drip. `tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24 pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now (~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track the new granularity. * fix(memory): read the YAML off spelling, and veto a skip over the whole batch Two review findings on the memory judging path. `memory.prescreen.mode` / `memory.signal_classification.mode` rejected the `mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1 parses the unquoted value as a boolean, so both fields saw `False`. The failed reload left the previous judge running, so both fields now normalize `False` to `off` before the `Literal` runs. `True` stays a validation error (no mode it could mean). The examples quote the value, and the docs record why. The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch it answers for is the whole post-watermark feed, so an explicit correction in an earlier turn of an eight-message batch did not stop the skip that then advanced the watermark past it. `detect_signals` gains a `window` parameter (default unchanged) and the updater scans the full feed for the veto; the extraction hint and the reinforcement gate keep the window. Both regressions are red on the previous head: the config test reproduced the `input_value=False` rejection through `AppConfig.from_file`, and the batch test reproduced the prescreen judging (and skipping) a batch whose first turn carries the correction.
6.8 KiB
Memory System Improvements
This document tracks memory injection behavior and roadmap status.
Status (As Of 2026-03-10)
Implemented in main:
- Accurate token counting via
tiktokeninformat_memory_for_injection. - Facts are injected into prompt memory context.
- Facts are ranked by confidence (descending).
- Injection respects
max_injection_tokensbudget.
Planned / not yet merged:
- TF-IDF similarity-based fact retrieval.
current_contextinput for context-aware scoring.- Configurable similarity/confidence weights (
similarity_weight,confidence_weight). - Middleware/runtime wiring for context-aware retrieval before each model call.
Current Behavior
Function today:
def format_memory_for_injection(memory_data: dict[str, Any], max_tokens: int = 2000) -> str:
Current injection format:
User Contextsection fromuser.*.summaryHistorysection fromhistory.*.summaryFactssection fromfacts[], sorted by confidence, appended until token budget is reached
Token counting:
- Uses
tiktoken(cl100k_base) when available - Falls back to a network-free CJK-aware character estimate if tokenizer import or encoding load fails (CJK characters count as ~2 chars/token, other characters as ~4 chars/token)
Extraction cost gate (opt-in, off by default)
memory.prescreen decides whether a batch of new conversation is worth paying
for the extraction LLM call; memory.signal_classification adds model hint
labels (and, in one narrow case, vetoes a skip). Both are off by default, and
off resolves no class path, constructs no provider and validates no credential.
Modes:
| Setting | Modes | Effect |
|---|---|---|
memory.prescreen.mode |
off / shadow / enforce |
shadow records the verdict and extracts as usual; enforce lets a low-probability batch skip the call and advance the watermark (that batch is then consumed as "nothing durable here") |
memory.signal_classification.mode |
off / shadow / hints |
hints merges reinforcement / correction labels into the extraction hint text and, only while the pre-screen is enforce, vetoes a skip |
memory.signal_classification.combine |
auto / always / never |
Whether the two sides share one request. auto shares only when every effective client setting matches (model, base URL, credential fingerprint, timeouts, retries, max_state_chars, cache settings); always fails at construction on a mismatch; never always splits |
Both mode values are written quoted in config.example.yaml (mode: "off").
Unquoted off is still accepted — YAML parses it as a boolean and both fields
normalize that back to off — but a quoted value says what an operator means.
Failure direction is always extraction: an error, timeout, unusable response, missing verdict, provider exception, or an over-limit batch extracts as usual. Nothing is truncated to fit a limit. A skip changes no post-extraction gate (confidence, scope, durability, authority, dedup, capacity eviction) and is not a safety boundary.
Batches that are never judged: no judge configured; the emergency
(pre-summarization) flush; the shutdown drain (flush_sync, whose budget belongs
to persistence); batches with a deterministic signal anywhere in the batch (a
positive signal outranks a model negative — the veto scans the whole batch, since
a skip consumes all of it, while the extraction hint keeps detect_signals' own
6-message window); and any batch while
staleness_review_enabled (default true) or consolidation_enabled is on,
because a skip would also skip that batch's maintenance review. With the shipped
defaults the pre-screen therefore sees no batches until staleness_review_enabled
is turned off — do that deliberately, with the evaluation below.
Egress. The formatted conversation text (user plus final assistant turns, which
may contain file paths or credentials) is sent to the TypeSafe API: a second
destination beyond the deployment's own extraction model. Existing memory, fact
ids and tool arguments are never sent. pii_redaction_middleware does not cover
this path. mode: "off" is the rollback — no request, no behavior change, no data
migration; the unquoted mode: off a hand-edit may type is accepted too.
Before enforce. Run the shadow evaluation and publish its gates:
miss_rate (a skip on a batch whose extraction would have been accepted) at or
below 1% and its one-sided 95% upper bound at or below that target — a sample
that only meets the point estimate is reported INSUFFICIENT, so the answer is
more skips rather than a smaller number to quote — zero misses on the identity /
preference / correction strata, at least 200 reviewed skips whose manual review
confirms none was worth remembering, and the recorded savings evidence — calls and
tokens saved, their p50/p95 verdict latency, and the no-network heuristic baseline
on the same data, with a positive incremental saving over that baseline measured on
the same records — the comparison must cover the scored skips, so a run that
annotates only a subset is reported INSUFFICIENT rather than crediting the model
for batches the heuristic never saw. If the heuristic already captures the savings
(it saves as many calls as the model), the third-party call is not justified and the
gate fails. scripts/eval_memory_prescreen.py reads
shadow records and reports those populations and gates, and reports a gate that
lacks evidence as INSUFFICIENT rather than passing it. skip_threshold (0.2)
and hint_threshold (0.5) are provisional and unmeasured.
Online hints is a different gate. The signal-classification design §6
requires its own independently human-reviewed dataset, a pre-registered maximum
allowed increase δ with a confidence upper bound and per-stratum sample sizes,
and a veto_recovered_facts benefit reading built on human-confirmed recovered
facts. The pre-screen evaluation above measures none of that, so its gates
approve enforce only; hints stays off/shadow until that evidence exists.
Known Gap
Previous versions of this document described TF-IDF/context-aware retrieval as if it were already shipped.
That was not accurate for main and caused confusion.
Issue reference: #1059
Roadmap (Planned)
Planned scoring strategy:
final_score = (similarity * 0.6) + (confidence * 0.4)
Planned integration shape:
- Extract recent conversational context from filtered user/final-assistant turns.
- Compute TF-IDF cosine similarity between each fact and current context.
- Rank by weighted score and inject under token budget.
- Fall back to confidence-only ranking if context is unavailable.
Validation
Current regression coverage includes:
- facts inclusion in memory injection output
- confidence ordering
- token-budget-limited fact inclusion
Tests:
backend/tests/test_memory_prompt_injection.py