github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-09 13:39:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

eval(memory): add a reproducible hybrid eviction evaluation (#4810)

Browse Source 
* eval(memory): scaffold reproducible eviction evaluation

* refactor(eval): align with benchmark layout

* eval(memory): add deterministic QA grading

Implement the disclosed deterministic-overlap-v1 grader as a pure offline
module. Grading is blind by construction: grade_answer() accepts only the
prediction and reference strings, never a policy identity.

The undisclosed stopword list is committed as a fixed part of this grader
version; yes/no/not are deliberately excluded because negation can be the
entire answer. Before freezing, the grader locally reproduced all 90
historical (prediction, grade) pairs disclosed in #4789 with zero
mismatches and no post-hoc tuning.

validate-contracts now rejects a config whose qa.grader_version does not
match the committed grader.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): add environment-configured QA runner

Add the exact answer-prompt renderer (retained facts sorted by ID, CURRENT
DATE line omitted when absent), an OpenAI-compatible provider adapter
configured only through the environment variable names pinned in the
config, and a resumable run-qa command that calls both policies with
identical versioned settings.

Each row persists as its own file on success, so a partial paid run
resumes without repeating completed calls; qa_run.json binds an output
directory to one config identity. Row files and errors carry predictions
and non-secret metadata only -- never questions, references, memory text,
credentials, or response headers. All tests are offline via mocked
transports; run-qa fails fast before touching the dataset when the
provider environment is missing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): add blind QA grading report and paired statistics

Add grade-qa: it recomputes the deterministic selector output, rejects any
answer row whose kept facts, capacity, or policy disagree with it, grades
every prediction through the policy-blind grade_answer(prediction,
reference) call, and only then joins grades back through stable row IDs.

Published artifacts are qa.rows.jsonl (graded rows with non-secret
metadata), qa.summary.json (accuracy by source/scenario/policy; official
and synthetic suites never folded together), and qa.stats.json (exact
paired McNemar and seeded paired bootstrap difference for the official,
synthetic, and overall suites using the pinned statistics parameters).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): pin the official DeepSeek model ID

The historical protocol recorded the answer model with an aggregator-style
namespace (deepseek/deepseek-v4-flash). The live run calls the same
underlying model (DeepSeek-V4-Flash-0731, released before the historical
run) directly through DeepSeek's official OpenAI-compatible API, whose
canonical ID is deepseek-v4-flash. The served model is recorded from the
provider response in every answer row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): publish paired eviction QA results

Publish the equal-budget live QA artifacts for pr4789-reproduction-v1:
provenance, 90 graded rows, per-scenario summary, and paired statistics.
At capacity 7 with identical settings, confidence answers 24/45 and
hybrid-v1 40/45 (official 23/40 vs 35/40, exact McNemar p=0.0042; overall
p=0.0004). The noisy-signal control is the one scenario where hybrid-v1
scored below the baseline (10/10 vs 8/10) and is reported separately.

The offline suite now verifies the published statistics are recomputable
from the published rows and that the artifacts carry no dataset text or
credentials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(eval): decompose the noisy-signal QA cell

Both policies retained the support fact in all ten noisy-signal cases, so
the two rows hybrid-v1 lost are grader phrasing boundaries (verbose
numeric answers rejected by the numeric-conflict rule), not eviction
failures. Documented from the published rows; the grader stays frozen.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): address review hardening findings

- ignore the responses/ directory the runner actually writes instead of
  the stale provider-responses/ entry
- cover the official selection-rule recomputation with direct synthetic
  tests: matching manifests pass, rule-breaking IDs and missing eligible
  rows fail, and every published exclusion is load-bearing
- align the report docstring and README with the statistics contract:
  the summary never folds sources; the explicitly labeled overall suite
  is reported alongside the separate official and synthetic suites
- recompute the published bootstrap intervals (not only McNemar) in the
  published-results test
- wire required_policy_version to the production
  EVICTION_POLICY_HYBRID_V1 constant so validate-contracts rejects
  policy drift

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): restore historical evidence rendering and harden resume identity

Address both blocking findings from the #4789 artifact cross-check.

The evidence renderer now emits the historical SESSION {id} AT {date}
line instead of the divergent bracket format. The byte representation is
protocol-critical: the witness record 35a27287 renders at 697 characters
again, stays inside the 700-character distractor-bank bound, and 60d45044
leaves the bank, restoring row-level pool reproduction. Deterministic
capacity-7 retention is unchanged at 27/45 vs 45/45.

qa_run.json now binds a run directory to the SHA-256 of all five protocol
inputs (config, both manifests, answer prompt, dataset) and names the
changed artifact when it refuses to resume. Stored rows are reused only
when row identity, policy, capacity, kept facts, and the request
fingerprint recomputed from the current task all match; the disclosed
probe (changed message under the same config) is now a regression test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): republish QA results under the historical protocol

Replace the published artifacts with the fresh equal-budget run executed
at 497ff3d0 under the restored historical evidence rendering; the earlier
run under the divergent rendering is discarded entirely. At capacity 7
with identical settings, confidence answers 24/45 and hybrid-v1 38/45
(official 23/40 vs 33/40, exact McNemar p=0.0129; overall p=0.0013).
The confidence control is the one scenario below baseline for hybrid-v1
(8/10 vs 6/10); both lost rows retained the support fact and are grader
phrasing/abstention boundaries, documented from the published rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): adopt the historical fact IDs and prompt serialization

Pool facts now carry the historical protocol IDs (gold_{case} for the
support fact, d_{case}_{index}_{source} for distractors in bank-draw
order), and the rendered STORED MEMORY joins fact blocks with a blank
line. Sorting by these IDs reproduces the historical selection tie-break:
witness case 41698283 at capacity 7 again keeps the 58bf7951 distractor
and evicts 001be529 under both policies. Deterministic capacity-7
retention is unchanged at 27/45 vs 45/45.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): republish QA results under the historical serialization

Replace the published artifacts with the fresh equal-budget run executed
at 01f99d61 under the historical fact IDs and prompt serialization;
earlier runs under divergent serializations are discarded entirely. At
capacity 7 with identical settings, confidence answers 24/45 and
hybrid-v1 40/45 (official 23/40 vs 35/40, exact McNemar p=0.0018;
overall p=0.0001). The single row below baseline (1cea1afa,
confidence-control) retained its support fact; the model abstained.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(memory): self-certify the publish path and pin offset coverage

grade-qa now verifies (read-only) that the run marker's five protocol
artifact hashes match the current inputs, rebuilds every answer task, and
rejects any stored row whose request fingerprint does not match the task
recomputed from the current protocol — the staleness class that
previously required an out-of-band cross-check to detect. Verified
end-to-end against the published run: all 90 rows pass and regrade to
byte-identical artifacts, while a tampered fingerprint is refused by row
ID.

The distractor offset derivation and wraparound selection are now pinned
by unit tests with hardcoded indices, including a wrapping offset, so a
digest-slice or modulus regression can no longer stay green offline.

Closes both non-blocking suggestions from the re-review.

* fix(bench): bind persisted answer rows to their expected case identity

Grading derived the reference case from the stored row's embedded case_id,
so reassigning a valid row to another valid case passed every integrity
check while silently changing the published grade. The resume path had the
same gap: _row_matches_task() never compared case_id, source, or scenario.

The recomputed task is now authoritative in both paths: grade_answer_rows()
resolves the reference case from the expected PolicyResult and rejects any
mismatch in the persisted row_id/case_id/source/scenario, and
_row_matches_task() checks the same identity fields so a reassigned row is
re-run instead of reused. Regressions tamper each field individually and
exercise both paths.

* docs(bench): document where to download the pinned LongMemEval file

The README named the dataset but never said it lives on Hugging Face or how
to fetch the pinned revision, so a reviewer could not run the offline
commands. Add the direct download URL, the expected SHA-256, and the mirror
and huggingface-cli alternatives; the CLI still never downloads anything.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
qingbo1011 2026-08-29 14:36:38 +08:00 committed by GitHub
parent 73e3699347
commit 2eba65449f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
37 changed files with 3631 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
backend/AGENTS.md
View File

@ -68,6 +68,7 @@ deer-flow/
│ │ │ ├── app.py # FastAPI application
│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
│ │ └── channels/ # IM platform integrations
│ ├── scripts/benchmark/ # Standalone reproducible backend benchmarks
│ ├── tests/ # Test suite
│ └── docs/ # Documentation
├── frontend/ # Next.js frontend application
@ -87,6 +88,47 @@ When making code changes, you MUST update the relevant documentation:
- Keep documentation synchronized with the codebase at all times
- Ensure accuracy and timeliness of all documentation
### Backend Benchmarks
`scripts/benchmark/` contains standalone, reproducible measurements and
evaluations of production backend behavior. A benchmark may import the
production function it measures, but it must not duplicate or introduce an
alternative runtime implementation.
- Pin every external dataset by immutable revision and SHA-256. Callers provide
the local dataset path; evaluation commands must not silently download data.
- Never commit upstream dataset text, credentials, complete provider requests,
or response headers. Committed manifests may contain stable IDs and source
locators. Synthetic cases must identify themselves as synthetic.
- Read provider credentials and endpoints from named environment variables.
Version model IDs, inference parameters, prompts, retry rules, clocks, and
random seeds in the evaluation config.
- Public raw results may contain case IDs, policy decisions, model hypotheses,
grades, and non-secret response metadata. Keep dataset questions, reference
answers, memory content, and full provider payloads in ignored local run
directories.
- Use fixed clocks and deterministic ordering for offline selection. Results
must record the config, manifest, prompt, dataset, and git revisions used.
`scripts/benchmark/deermem_eviction/` evaluates the production
`select_facts_for_capacity()` implementation used by DeerMem. It compares only
the historical `confidence` policy and PR #4789's opt-in `hybrid-v1`; do not add
another eviction strategy to this evaluation. Run its offline checks from
`backend/`:
```bash
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate-contracts
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate --dataset "$LONGMEMEVAL_ORACLE_PATH"
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction run-policy \
--dataset "$LONGMEMEVAL_ORACLE_PATH" \
--output-dir /tmp/deermem-eviction-policy-run
PYTHONPATH=. uv run pytest tests/test_bench_deermem_eviction_*.py -q
```
The offline test suite must not require network access, provider credentials,
or the LongMemEval dataset. Small LongMemEval-shaped fixtures must be synthetic
and generated by tests.
## Commands
**Root directory** (for full application):

4
backend/scripts/benchmark/deermem_eviction/.gitignore vendored Normal file
View File

@ -0,0 +1,4 @@
runs/
.cache/
prepared/
responses/

190
backend/scripts/benchmark/deermem_eviction/README.md Normal file
View File

@ -0,0 +1,190 @@
# DeerMem Capacity-Eviction Evaluation
This directory makes the controlled comparison behind DeerMem's opt-in `hybrid-v1` capacity policy reproducible. It depends on [deer-flow#4789](https://github.com/bytedance/deer-flow/pull/4789), which implements the remediation proposed after the confidence-only eviction flaw reported in [deer-flow#4641](https://github.com/bytedance/deer-flow/issues/4641).
The evaluation calls the production `select_facts_for_capacity()` function. It does not copy the scoring implementation and does not introduce another eviction strategy.
## Current scope
The first stage is entirely offline:
- pins the cleaned LongMemEval oracle file by repository revision and SHA-256;
- commits only the 40 official question IDs, not the upstream questions, answers, or histories;
- commits the five independently authored synthetic correction guards disclosed in #4789;
- reconstructs each 10-fact pool deterministically;
- compares `confidence` and the production `hybrid-v1` policy at capacities 5, 7, and 9;
- writes metadata-only row results that are safe to publish.
The deterministic grader (`grading.py`), the resumable live QA runner (`qa.py`, `provider.py`, `runner.py`), and the blind grading/statistics report (`report.py`, `stats.py`) are implemented; all are documented below. Both policies receive fresh calls with the same `max_tokens=2048`; the historical optimization that reused a 1024-token confidence baseline is not reproduced.
## Pinned inputs
| Input | Value |
| --- | --- |
| Dataset | `xiaowu0162/longmemeval-cleaned` |
| Revision | `98d7416c24c778c2fee6e6f3006e7a073259d48f` |
| File | `longmemeval_oracle.json` |
| SHA-256 | `821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c` |
| Official cases | 40 fixed IDs: 20 `knowledge-update`, 20 `temporal-reasoning` |
| Synthetic cases | 5 correction guards |
| Pool | 1 support fact + 9 deterministic distractors |
| Capacities | 5, 7, 9; QA capacity 7 |
| Evaluation clock | `2026-08-13T00:00:00Z` |
The CLI never downloads LongMemEval. The pinned file is `longmemeval_oracle.json` (about 15 MB) in the public, ungated Hugging Face dataset repository [`xiaowu0162/longmemeval-cleaned`](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned); no account or token is needed. Download it once at the pinned revision and expose its path locally:
```bash
export LONGMEMEVAL_ORACLE_PATH=/absolute/path/to/longmemeval_oracle.json
curl -L -o "$LONGMEMEVAL_ORACLE_PATH" \
"https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/98d7416c24c778c2fee6e6f3006e7a073259d48f/longmemeval_oracle.json"
shasum -a 256 "$LONGMEMEVAL_ORACLE_PATH"
# expected: 821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c
```
If `huggingface.co` is not reachable from your network, the same `/datasets/.../resolve/<revision>/...` path works through a Hugging Face mirror (for example, replace the host with `hf-mirror.com`), and `huggingface-cli download xiaowu0162/longmemeval-cleaned longmemeval_oracle.json --repo-type dataset --revision 98d7416c24c778c2fee6e6f3006e7a073259d48f` is equivalent. Whatever the source, every command rejects a file whose hash differs from the pinned value, so a wrong or modified download cannot pass silently. The dataset itself and prepared text-bearing pools belong outside the repository or under ignored local directories.
## Commands
Run all commands from `backend/`.
Validate the committed config, manifests, and prompt without an upstream dataset:
```bash
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate-contracts
```
Validate the dataset hash, recompute the declared sample-selection rule, build the distractor bank, and prepare all 45 cases:
```bash
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction validate \
--dataset "$LONGMEMEVAL_ORACLE_PATH"
```
Run deterministic capacity selection without any provider calls:
```bash
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction run-policy \
--dataset "$LONGMEMEVAL_ORACLE_PATH" \
--output-dir /tmp/deermem-eviction-policy-run
```
The command refuses to overwrite an existing run. Use a new output directory for every run.
Call the configured answer provider for both policies at the QA capacity (45 cases x 2 policies = 90 calls on a fresh run):
```bash
export DEERMEM_EVAL_ANSWER_API_KEY=... # never committed or logged
export DEERMEM_EVAL_ANSWER_BASE_URL=... # OpenAI-compatible endpoint
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction run-qa \
--dataset "$LONGMEMEVAL_ORACLE_PATH" \
--output-dir /tmp/deermem-eviction-qa-run
```
The runner resolves credentials only from the two environment variables named in the config and fails before touching the dataset when either is missing. Model, temperature, `max_tokens`, stream, timeout, retry attempts, and worker count all come from the versioned config; both policies use identical settings. Each row is written to `responses/<case>__<policy>.json` as soon as its call succeeds, so rerunning the same command resumes a partial run without repeating completed calls. `qa_run.json` binds the output directory to the full protocol identity — the SHA-256 of the config, both manifests, the answer prompt, and the dataset — and rejects resumption when any of them changed. A stored row is reused only when its row identity, kept facts, and `request_fingerprint` all match the task recomputed from the current protocol; a row whose fingerprint no longer matches is re-called rather than silently reused. Row files contain the prediction and non-secret metadata only — never questions, reference answers, memory content, credentials, or response headers.
Grade a completed answer run and write the public QA results (no provider calls):
```bash
PYTHONPATH=. uv run python -m scripts.benchmark.deermem_eviction grade-qa \
--dataset "$LONGMEMEVAL_ORACLE_PATH" \
--output-dir /tmp/deermem-eviction-qa-run
```
Grading happens through `grade_answer(prediction, reference)` — two strings, no policy identity — and is joined back to policies only afterwards through stable row IDs. Before grading, the command verifies (read-only) that the run marker's five protocol artifact hashes match the current config, manifests, prompt, and dataset, recomputes the deterministic selector output and every task's request fingerprint, and rejects any answer row whose kept facts, capacity, policy, or stored fingerprint disagree with the recomputed protocol; it refuses to proceed while any of the 90 rows is missing. A published `qa.rows.jsonl` therefore certifies that its predictions were produced under the exact protocol being graded, not under an earlier serialization.
## Deterministic reconstruction
Official samples are independently recomputed from the pinned dataset rather than merely checked for existence. For each of the two eligible question types, the loader applies the published exclusions, sorts by `question_id`, and selects the first 20. Consecutive groups of five are assigned according to the manifest's explicit `scenario_order` field.
Evidence extraction iterates `haystack_sessions`. Within each session it selects turns marked `has_answer`; when a session contains no marked turn, it falls back to user turns. Each rendered session is prefixed with the historical `SESSION {id} AT {date}` line — the exact byte representation matters, because evidence-length filters apply to this final rendered value and the 700-character distractor-bank bound decides bank membership (the cross-check in #4810 caught a divergent prefix format precisely this way).
The distractor bank contains the first 40 eligible `single-session-user` and `single-session-preference` records sorted by question ID. A case offset is derived from the first four bytes of:
```text
sha256("deermem-medium-v1:{case_id}")
```
Nine consecutive records are selected with wraparound. Pool facts carry the historical protocol fact IDs: the support fact is `gold_{case}` and distractor `i` (0-based, in bank-draw order) is `d_{case}_{i}_{source}`. Facts are sorted by these IDs before they enter the production selector and before prompt rendering, so the selector's stable input-order tie break reproduces the historical tie-break exactly (distractors in draw order first, the support fact last), and the rendered `STORED MEMORY` joins fact blocks with a blank line. Access metadata uses the fixed evaluation clock, so no wall-clock decay can change a rerun.
At capacity 7, the offline result reproduces the disclosed support-retention totals:
| Suite | `confidence` | `hybrid-v1` |
| --- | ---: | ---: |
| 40 official + 5 synthetic | 27/45 | 45/45 |
This is a deterministic selector result, not evidence that production reinforcement detection or query access heat is unbiased. The eventual QA report must keep official, synthetic-correction, and noisy-signal results separate.
## Deterministic grading
`grading.py` implements the disclosed grader as a pure, offline module versioned as `deterministic-overlap-v1`; the config pins that identity via `qa.grader_version`, and `validate-contracts` rejects a mismatch. The grader is blind by construction: `grade_answer(prediction, reference)` accepts only the two answer strings and never a policy identity.
Normalization lowercases, replaces every non-alphanumeric character with a space, and maps the English number words one through ten and fifteen to digits. Rules apply in order:
1. reject an empty prediction or the exact `INSUFFICIENT` sentinel;
2. accept exact normalized-token equality;
3. accept containment of one token sequence in the other as a contiguous subsequence (token-level, so `5` never matches inside `25`);
4. accept a prediction whose integer tokens all fall inside an explicit `ranging from X ... to Y` reference range;
5. reject conflicting integer tokens when both sides contain integers;
6. otherwise require at least 60% unique non-stopword token overlap in both directions.
The disclosure in #4789 did not publish an exact stopword list, so the list committed in `grading.py` is a fixed part of this grader version: common English function words, with `yes`, `no`, and `not` deliberately excluded because negation can be the entire answer. Changing the list or any rule requires a new `grader_version`.
Before freezing, the grader was cross-checked locally against all 90 historical `(prediction, reference)` pairs disclosed in #4789 — the saved QA grades for both policies across 45 cases — and reproduced every historical grade exactly, with no per-result tuning afterward.
## Output contract
`run-policy` creates three files:
- `run.json` records the git state, immutable dataset identity, evaluation clock, capacities, and SHA-256 values for config, manifests, and prompt.
- `policy.raw.jsonl` contains one row per case, capacity, and policy. Rows include fact IDs, kept/evicted IDs, score components, support retention, and correction reservation. They never include fact content, questions, or reference answers.
- `summary.json` aggregates support retention by source, scenario, capacity, and policy. Synthetic corrections are not folded into an official-only metric.
`grade-qa` adds three publishable files to a completed answer run and refuses to overwrite them:
- `qa.rows.jsonl` contains one graded row per case and policy: IDs, scenario/source, kept-fact metadata, the model's prediction, the grade with its deciding rule, the grader version, and non-secret response metadata.
- `qa.summary.json` reports accuracy by source, scenario, and policy; the summary never folds official scenarios and synthetic corrections into one figure. The only combined figure is the explicitly labeled `overall` suite in `qa.stats.json`, reported alongside — never instead of — the separate official and synthetic suites.
- `qa.stats.json` reports the exact paired McNemar test and the seeded paired bootstrap difference (`hybrid-v1` minus `confidence`) for the official, synthetic, and overall suites, using the statistics parameters pinned in the config.
Full provider requests, dataset text, and prepared pools must remain in ignored local directories. Provider response headers must never be persisted because they can contain sensitive or account-specific data.
## Model addressing
The historical protocol disclosed in #4789 recorded the answer model as `deepseek/deepseek-v4-flash`, an aggregator-style namespace. This evaluation calls the same underlying model (DeepSeek-V4-Flash-0731, released before the historical run) directly through DeepSeek's official OpenAI-compatible API, whose canonical ID is `deepseek-v4-flash`; the config pins that ID. The model actually serving each call is recorded from the provider response in every answer row as `response_model`.
## Published live QA results
`results/pr4789-reproduction-v1/` contains the published artifacts of the equal-budget live run executed at repository revision `01f99d61` (2026-08-18, DeepSeek official API, `deepseek-v4-flash`), after the evidence rendering, the historical fact-ID scheme, and the prompt serialization identified in the artifact cross-check were all adopted: `qa_run.json` (provenance), `qa.rows.jsonl` (90 graded rows), `qa.summary.json`, and `qa.stats.json`. Earlier live runs executed under divergent serializations were discarded entirely rather than partially reused. The offline suite verifies that the published statistics are recomputable from the published rows.
QA accuracy at capacity 7 with identical settings for both policies:
| Suite | `confidence` | `hybrid-v1` | Exact McNemar p | Accuracy difference (95% CI) |
| --- | ---: | ---: | ---: | --- |
| 40 official | 23/40 | 35/40 | 0.0018 | +0.300 [+0.150, +0.450] |
| 5 synthetic corrections | 1/5 | 5/5 | 0.1250 | +0.800 [+0.400, +1.000] |
| 45 overall | 24/45 | 40/45 | 0.0001 | +0.356 [+0.200, +0.511] |
Scenario breakdown: `confirmation_help` 3/10 vs 10/10, `access_help` 3/10 vs 9/10, `confidence_control` 8/10 vs 7/10, `noisy_signal_control` 9/10 vs 9/10, synthetic corrections 1/5 vs 5/5. The confidence control is the one scenario where `hybrid-v1` scored below the confidence baseline in this run; it is reported separately and not folded into any other metric.
Decomposing the discordant cells from the published rows: `hybrid-v1` lost exactly one case against the baseline (`1cea1afa`, confidence-control), and its support fact was retained, so it is not an eviction failure — the model abstained with `INSUFFICIENT` despite the retained support. The policies retain different distractor sets, which nudges answer phrasing and abstention; the grader remains frozen and the cell is reported as-is.
Both totals sit well above the historical `14/45` vs `23/45`, primarily because the historical confidence baseline was limited to 1024 output tokens while this run gives both policies the same 2048-token budget. The run consumed 86,342 input and 12,340 output tokens across the 90 calls.
## Historical-result caveats
The row-level artifacts disclosed in #4789 corrected the PR text's noisy-signal QA result from `5/10 vs 5/10` to `5/10 vs 6/10`. They also showed that the historical confidence rows used a 1024-token baseline, while hybrid rows used 2048 tokens and new calls. The follow-up live run will therefore:
1. rerun both policies rather than reuse the historical baseline;
2. use the same model, prompt, 2048-token budget, retry policy, and concurrency;
3. blind the grader to policy identity;
4. save public row-level outputs without upstream dataset text;
5. report official and synthetic statistics separately.
## Tests
The default tests are offline and use only synthetic LongMemEval-shaped rows:
```bash
PYTHONPATH=. uv run pytest tests/test_bench_deermem_eviction_*.py -q
```
They cover config and manifest contracts, prompt hashing, dataset-integrity rejection, evidence extraction, distractor filtering, deterministic pool construction, production selector behavior, correction reservation, public-result redaction, overwrite protection, and every grading rule with its edge cases.

5
backend/scripts/benchmark/deermem_eviction/__init__.py Normal file
View File

@ -0,0 +1,5 @@
"""Reproducible capacity-eviction evaluation for DeerMem."""
from .config import EvaluationConfig, load_evaluation_config
__all__ = ["EvaluationConfig", "load_evaluation_config"]

4
backend/scripts/benchmark/deermem_eviction/__main__.py Normal file
View File

@ -0,0 +1,4 @@
from .cli import main
if __name__ == "__main__":
raise SystemExit(main())

161
backend/scripts/benchmark/deermem_eviction/cli.py Normal file
View File

@ -0,0 +1,161 @@
from __future__ import annotations
import argparse
from collections.abc import Sequence
from pathlib import Path
from .config import EvaluationConfig, load_evaluation_config
from .dataset import LongMemEvalDataset, load_longmemeval
from .grading import GRADER_VERSION
from .io import sha256_file
from .manifest import OfficialManifest, SyntheticManifest, load_official_manifest, load_synthetic_manifest
from .policy import evaluate_case, require_production_policy
from .protocol import build_protocol_cases, validate_official_selection
from .provider import build_client, request_fingerprint, resolve_provider_settings
from .qa import build_answer_task
from .report import collect_answer_rows, compute_qa_statistics, grade_answer_rows, summarize_qa_rows, write_qa_report
from .results import write_policy_run
from .runner import ensure_run_config_identity, run_answer_calls, verify_run_identity
EVAL_ROOT = Path(__file__).resolve().parent
BACKEND_ROOT = EVAL_ROOT.parents[2]
DEFAULT_CONFIG = EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml"
DEFAULT_OFFICIAL_MANIFEST = EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json"
DEFAULT_SYNTHETIC_MANIFEST = EVAL_ROOT / "manifests" / "synthetic-corrections-pr4789-v1.json"
def _load_contracts(args: argparse.Namespace) -> tuple[EvaluationConfig, OfficialManifest, SyntheticManifest, Path]:
config = load_evaluation_config(args.config)
official = load_official_manifest(args.official_manifest)
synthetic = load_synthetic_manifest(args.synthetic_manifest)
if {config.protocol_id, official.protocol_id, synthetic.protocol_id} != {config.protocol_id}:
raise ValueError("config and manifests use different protocol IDs")
if config.qa.grader_version != GRADER_VERSION:
raise ValueError(f"config pins grader {config.qa.grader_version!r} but the committed grader is {GRADER_VERSION!r}")
require_production_policy(config.required_policy_version)
prompt_path = EVAL_ROOT / config.qa.answer_prompt.path
actual_prompt_sha = sha256_file(prompt_path)
if actual_prompt_sha != config.qa.answer_prompt.sha256:
raise ValueError(f"answer prompt SHA-256 mismatch: expected {config.qa.answer_prompt.sha256}, got {actual_prompt_sha}")
return config, official, synthetic, prompt_path
def _load_validated_dataset(args: argparse.Namespace, config: EvaluationConfig, official: OfficialManifest) -> LongMemEvalDataset:
dataset = load_longmemeval(args.dataset, expected_sha256=config.dataset.sha256)
validate_official_selection(dataset, official)
return dataset
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Reproduce DeerMem confidence vs hybrid-v1 capacity evaluation")
parser.set_defaults(config=DEFAULT_CONFIG, official_manifest=DEFAULT_OFFICIAL_MANIFEST, synthetic_manifest=DEFAULT_SYNTHETIC_MANIFEST)
subparsers = parser.add_subparsers(dest="command", required=True)
contracts = subparsers.add_parser("validate-contracts", help="Validate committed config, manifests, and prompt without the dataset")
contracts.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
contracts.add_argument("--official-manifest", type=Path, default=DEFAULT_OFFICIAL_MANIFEST)
contracts.add_argument("--synthetic-manifest", type=Path, default=DEFAULT_SYNTHETIC_MANIFEST)
validate = subparsers.add_parser("validate", help="Validate contracts and the caller-supplied LongMemEval file")
validate.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
validate.add_argument("--official-manifest", type=Path, default=DEFAULT_OFFICIAL_MANIFEST)
validate.add_argument("--synthetic-manifest", type=Path, default=DEFAULT_SYNTHETIC_MANIFEST)
validate.add_argument("--dataset", type=Path, required=True)
run_policy = subparsers.add_parser("run-policy", help="Run deterministic retention evaluation with no provider calls")
run_policy.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
run_policy.add_argument("--official-manifest", type=Path, default=DEFAULT_OFFICIAL_MANIFEST)
run_policy.add_argument("--synthetic-manifest", type=Path, default=DEFAULT_SYNTHETIC_MANIFEST)
run_policy.add_argument("--dataset", type=Path, required=True)
run_policy.add_argument("--output-dir", type=Path, required=True)
run_qa = subparsers.add_parser("run-qa", help="Call the configured answer provider for both policies at the QA capacity (resumable; requires provider environment variables)")
run_qa.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
run_qa.add_argument("--official-manifest", type=Path, default=DEFAULT_OFFICIAL_MANIFEST)
run_qa.add_argument("--synthetic-manifest", type=Path, default=DEFAULT_SYNTHETIC_MANIFEST)
run_qa.add_argument("--dataset", type=Path, required=True)
run_qa.add_argument("--output-dir", type=Path, required=True)
grade_qa = subparsers.add_parser("grade-qa", help="Grade completed answer rows blindly and write the public QA rows, summary, and paired statistics")
grade_qa.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
grade_qa.add_argument("--official-manifest", type=Path, default=DEFAULT_OFFICIAL_MANIFEST)
grade_qa.add_argument("--synthetic-manifest", type=Path, default=DEFAULT_SYNTHETIC_MANIFEST)
grade_qa.add_argument("--dataset", type=Path, required=True)
grade_qa.add_argument("--output-dir", type=Path, required=True)
return parser
def main(argv: Sequence[str] | None = None) -> int:
args = build_parser().parse_args(argv)
config, official, synthetic, prompt_path = _load_contracts(args)
if args.command == "validate-contracts":
official_count = sum(len(question_ids) for question_ids in official.scenarios.values())
print(f"validated {official_count} official and {len(synthetic.cases)} synthetic cases")
return 0
provider_settings = resolve_provider_settings(config.qa) if args.command == "run-qa" else None
dataset = _load_validated_dataset(args, config, official)
cases = build_protocol_cases(dataset, config, official, synthetic)
if args.command == "validate":
print(f"validated dataset {dataset.sha256} and prepared {len(cases)} cases")
return 0
if args.command == "run-policy":
results = [evaluate_case(case, policy_name=policy_name, capacity=capacity, hybrid_config=config.policies.hybrid_v1) for case in cases for capacity in config.pool.capacities for policy_name in ("confidence", "hybrid-v1")]
write_policy_run(
args.output_dir,
results=results,
config=config,
config_path=args.config,
official_manifest_path=args.official_manifest,
synthetic_manifest_path=args.synthetic_manifest,
prompt_path=prompt_path,
dataset_path=args.dataset,
backend_root=BACKEND_ROOT,
)
print(f"wrote {len(results)} policy rows for {len(cases)} cases to {args.output_dir}")
return 0
if args.command == "run-qa":
assert provider_settings is not None
template = prompt_path.read_text(encoding="utf-8")
tasks = [build_answer_task(case, evaluate_case(case, policy_name=policy_name, capacity=config.pool.qa_capacity, hybrid_config=config.policies.hybrid_v1), template) for case in cases for policy_name in ("confidence", "hybrid-v1")]
ensure_run_config_identity(
args.output_dir,
config=config,
config_path=args.config,
official_manifest_path=args.official_manifest,
synthetic_manifest_path=args.synthetic_manifest,
prompt_path=prompt_path,
dataset_path=args.dataset,
backend_root=BACKEND_ROOT,
)
with build_client(provider_settings, config.qa) as client:
report = run_answer_calls(tasks, config=config, client=client, output_dir=args.output_dir)
print(f"answer rows: {report.reused} reused, {report.called} called, {len(report.failed)} failed")
for failure in report.failed:
print(f" failed {failure}")
return 1 if report.failed else 0
if args.command == "grade-qa":
template = prompt_path.read_text(encoding="utf-8")
results_by_row = {}
expected_fingerprints = {}
for case in cases:
for policy_name in ("confidence", "hybrid-v1"):
result = evaluate_case(case, policy_name=policy_name, capacity=config.pool.qa_capacity, hybrid_config=config.policies.hybrid_v1)
task = build_answer_task(case, result, template)
results_by_row[task.row_id] = result
expected_fingerprints[task.row_id] = request_fingerprint(config.qa, task.messages)
verify_run_identity(args.output_dir, config_path=args.config, official_manifest_path=args.official_manifest, synthetic_manifest_path=args.synthetic_manifest, prompt_path=prompt_path, dataset_path=args.dataset)
rows = collect_answer_rows(args.output_dir, cases)
graded = grade_answer_rows(cases, results_by_row, rows, expected_fingerprints=expected_fingerprints)
summary = summarize_qa_rows(graded)
statistics = compute_qa_statistics(graded, config)
write_qa_report(args.output_dir, graded=graded, summary=summary, statistics=statistics, config=config)
for suite, values in statistics["suites"].items():
mcnemar = values["mcnemar"]
correct_first = mcnemar["both_correct"] + mcnemar["only_first_correct"]
correct_second = mcnemar["both_correct"] + mcnemar["only_second_correct"]
print(f"{suite}: confidence {correct_first}/{values['cases']}, hybrid-v1 {correct_second}/{values['cases']}, exact McNemar p={mcnemar['p_value']:.6f}")
print(f"graded {len(graded)} rows to {args.output_dir}")
return 0
raise AssertionError(f"unhandled command: {args.command}")

130
backend/scripts/benchmark/deermem_eviction/config.py Normal file
View File

@ -0,0 +1,130 @@
from __future__ import annotations
import math
from datetime import UTC, datetime
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from .io import load_yaml
class StrictModel(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
class DatasetConfig(StrictModel):
repository: str
revision: str = Field(pattern=r"^[0-9a-f]{40}$")
filename: str
sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
class PoolConfig(StrictModel):
size: int = Field(ge=2)
capacities: list[int]
qa_capacity: int = Field(ge=1)
distractors: int = Field(ge=1)
distractor_bank_size: int = Field(ge=1)
distractor_types: list[str]
distractor_min_evidence_chars: int = Field(ge=0)
distractor_max_evidence_chars: int = Field(ge=1)
offset_namespace: str
fact_order: Literal["fact-id"]
@model_validator(mode="after")
def validate_shape(self) -> PoolConfig:
if self.distractors != self.size - 1:
raise ValueError("pool.distractors must equal pool.size - 1")
if self.distractor_bank_size < self.distractors:
raise ValueError("pool.distractor_bank_size must cover all distractors")
if not self.distractor_types or len(self.distractor_types) != len(set(self.distractor_types)):
raise ValueError("pool.distractor_types must be non-empty and unique")
if not self.capacities or len(self.capacities) != len(set(self.capacities)):
raise ValueError("pool.capacities must be non-empty and unique")
if any(capacity < 1 or capacity >= self.size for capacity in self.capacities):
raise ValueError("pool capacities must be between 1 and pool.size - 1")
if self.qa_capacity not in self.capacities:
raise ValueError("pool.qa_capacity must appear in pool.capacities")
if self.distractor_min_evidence_chars > self.distractor_max_evidence_chars:
raise ValueError("distractor evidence bounds are reversed")
return self
class ConfidencePolicyConfig(StrictModel):
policy: Literal["confidence"] = "confidence"
class HybridPolicyConfig(StrictModel):
policy: Literal["hybrid-v1"] = "hybrid-v1"
weights: dict[Literal["confidence", "confirmation", "access"], float]
confirmation_half_life_days: int = Field(ge=1)
access_half_life_days: int = Field(ge=1)
correction_reserved_fraction: float = Field(ge=0.0, le=1.0)
correction_reserved_max: int = Field(ge=0)
@field_validator("weights")
@classmethod
def validate_weights(cls, value: dict[str, float]) -> dict[str, float]:
if set(value) != {"confidence", "confirmation", "access"}:
raise ValueError("hybrid-v1 weights must define confidence, confirmation, and access")
if any(weight < 0.0 or weight > 1.0 for weight in value.values()):
raise ValueError("hybrid-v1 weights must be bounded between 0 and 1")
if not math.isclose(sum(value.values()), 1.0, rel_tol=0.0, abs_tol=1e-9):
raise ValueError("hybrid-v1 weights must sum to 1.0")
return value
class PoliciesConfig(StrictModel):
confidence: ConfidencePolicyConfig
hybrid_v1: HybridPolicyConfig
class PromptFileConfig(StrictModel):
path: str
sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
class QAConfig(StrictModel):
answer_prompt: PromptFileConfig
grader_version: str
provider: Literal["openai-compatible"]
api_key_env: str
base_url_env: str
model: str
temperature: float
max_tokens: int = Field(ge=1)
stream: bool
timeout_seconds: int = Field(ge=1)
max_attempts: int = Field(ge=1)
workers: int = Field(ge=1)
class StatisticsConfig(StrictModel):
bootstrap_seed: int
bootstrap_iterations: int = Field(ge=1)
alpha: float = Field(gt=0.0, lt=1.0)
class EvaluationConfig(StrictModel):
schema_version: Literal[1]
protocol_id: str
required_policy_version: Literal["hybrid-v1"]
evaluation_time: datetime
dataset: DatasetConfig
pool: PoolConfig
policies: PoliciesConfig
qa: QAConfig
statistics: StatisticsConfig
@field_validator("evaluation_time")
@classmethod
def require_aware_utc(cls, value: datetime) -> datetime:
if value.tzinfo is None:
raise ValueError("evaluation_time must include a timezone")
return value.astimezone(UTC)
def load_evaluation_config(path: Path) -> EvaluationConfig:
return EvaluationConfig.model_validate(load_yaml(path))

57
backend/scripts/benchmark/deermem_eviction/configs/pr4789-reproduction-v1.yaml Normal file
View File

@ -0,0 +1,57 @@
schema_version: 1
protocol_id: deermem-hybrid-v1-pr4789-reproduction-v1
required_policy_version: hybrid-v1
evaluation_time: "2026-08-13T00:00:00Z"
dataset:
repository: xiaowu0162/longmemeval-cleaned
revision: 98d7416c24c778c2fee6e6f3006e7a073259d48f
filename: longmemeval_oracle.json
sha256: 821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c
pool:
size: 10
capacities: [5, 7, 9]
qa_capacity: 7
distractors: 9
distractor_bank_size: 40
distractor_types: [single-session-user, single-session-preference]
distractor_min_evidence_chars: 40
distractor_max_evidence_chars: 700
offset_namespace: deermem-medium-v1
fact_order: fact-id
policies:
confidence:
policy: confidence
hybrid_v1:
policy: hybrid-v1
weights:
confidence: 0.65
confirmation: 0.25
access: 0.10
confirmation_half_life_days: 90
access_half_life_days: 30
correction_reserved_fraction: 0.10
correction_reserved_max: 10
qa:
answer_prompt:
path: prompts/answer-v1.txt
sha256: 2bd4f166219946ccff7ac86f52a053c6dbf8be6d27e178c398d6bf70cf8a7423
grader_version: deterministic-overlap-v1
provider: openai-compatible
api_key_env: DEERMEM_EVAL_ANSWER_API_KEY
base_url_env: DEERMEM_EVAL_ANSWER_BASE_URL
model: deepseek-v4-flash
temperature: 0
max_tokens: 2048
stream: false
timeout_seconds: 120
max_attempts: 3
workers: 3
statistics:
bootstrap_seed: 4789
bootstrap_iterations: 10000
alpha: 0.05

126
backend/scripts/benchmark/deermem_eviction/dataset.py Normal file
View File

@ -0,0 +1,126 @@
from __future__ import annotations
import json
from collections.abc import Iterable
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from .io import load_json, sha256_file
class DatasetIntegrityError(ValueError):
"""Raised when an upstream dataset cannot satisfy the pinned protocol."""
@dataclass(frozen=True)
class EvidenceRecord:
question_id: str
question_type: str
question: str
answer: str
question_date: str | None
content: str
@dataclass(frozen=True)
class LongMemEvalDataset:
path: Path
sha256: str
rows: tuple[dict[str, Any], ...]
rows_by_id: dict[str, dict[str, Any]]
def _stringify_answer(value: Any) -> str:
if isinstance(value, str):
return value
return json.dumps(value, ensure_ascii=False, sort_keys=True)
def _selected_evidence_sessions(row: dict[str, Any]) -> list[tuple[Any, Any, list[dict[str, Any]]]]:
session_ids = row.get("haystack_session_ids")
dates = row.get("haystack_dates")
sessions = row.get("haystack_sessions")
if not isinstance(session_ids, list) or not isinstance(dates, list) or not isinstance(sessions, list):
raise DatasetIntegrityError("LongMemEval row has invalid haystack arrays")
if not (len(session_ids) == len(dates) == len(sessions)):
raise DatasetIntegrityError("LongMemEval haystack arrays have different lengths")
selected_sessions: list[tuple[Any, Any, list[dict[str, Any]]]] = []
for session_id, date, turns in zip(session_ids, dates, sessions, strict=True):
if not isinstance(turns, list):
raise DatasetIntegrityError("LongMemEval session is not a turn list")
valid_turns = [turn for turn in turns if isinstance(turn, dict) and isinstance(turn.get("content"), str)]
selected = [turn for turn in valid_turns if turn.get("has_answer") is True]
if not selected:
selected = [turn for turn in valid_turns if str(turn.get("role") or "").lower() == "user"]
if not selected:
continue
selected_sessions.append((session_id, date, selected))
return selected_sessions
def extract_evidence(row: dict[str, Any]) -> str:
rendered_sessions: list[str] = []
for session_id, date, selected in _selected_evidence_sessions(row):
lines = [f"SESSION {session_id} AT {date}"]
lines.extend(f"{str(turn.get('role') or 'unknown').upper()}: {turn['content']}" for turn in selected)
rendered_sessions.append("\n".join(lines))
return "\n\n".join(rendered_sessions)
def evidence_record(row: dict[str, Any]) -> EvidenceRecord:
question_id = row.get("question_id")
question_type = row.get("question_type")
question = row.get("question")
if not all(isinstance(value, str) and value for value in (question_id, question_type, question)):
raise DatasetIntegrityError("LongMemEval row is missing question identity fields")
question_date = row.get("question_date")
if question_date is not None and not isinstance(question_date, str):
raise DatasetIntegrityError(f"LongMemEval question_date is invalid for {question_id}")
return EvidenceRecord(
question_id=question_id,
question_type=question_type,
question=question,
answer=_stringify_answer(row.get("answer")),
question_date=question_date,
content=extract_evidence(row),
)
def load_longmemeval(path: Path, *, expected_sha256: str) -> LongMemEvalDataset:
actual_sha256 = sha256_file(path)
if actual_sha256 != expected_sha256:
raise DatasetIntegrityError(f"LongMemEval SHA-256 mismatch: expected {expected_sha256}, got {actual_sha256}")
raw = load_json(path)
if not isinstance(raw, list) or not all(isinstance(row, dict) for row in raw):
raise DatasetIntegrityError("LongMemEval root must be a list of objects")
rows = tuple(raw)
rows_by_id: dict[str, dict[str, Any]] = {}
for row in rows:
question_id = row.get("question_id")
if not isinstance(question_id, str) or not question_id:
raise DatasetIntegrityError("LongMemEval row has an invalid question_id")
if question_id in rows_by_id:
raise DatasetIntegrityError(f"Duplicate LongMemEval question_id: {question_id}")
rows_by_id[question_id] = row
return LongMemEvalDataset(path=path, sha256=actual_sha256, rows=rows, rows_by_id=rows_by_id)
def build_distractor_bank(
rows: Iterable[dict[str, Any]],
*,
allowed_types: set[str],
min_evidence_chars: int,
max_evidence_chars: int,
limit: int,
) -> list[EvidenceRecord]:
candidates: list[EvidenceRecord] = []
for row in rows:
if row.get("question_type") not in allowed_types:
continue
record = evidence_record(row)
if min_evidence_chars <= len(record.content) <= max_evidence_chars:
candidates.append(record)
candidates.sort(key=lambda item: item.question_id)
return candidates[:limit]

117
backend/scripts/benchmark/deermem_eviction/grading.py Normal file
View File

@ -0,0 +1,117 @@
"""Deterministic, offline QA grader for the pr4789 reproduction protocol.
The grader is blind by construction: it receives only a prediction string and a
reference string, never a policy identity. Its behavior is versioned as
``GRADER_VERSION`` and pinned by ``qa.grader_version`` in the evaluation config;
any behavioral change requires a new version string.
Rule order, following the protocol disclosed in pr4789:
1. reject an empty prediction or the literal ``INSUFFICIENT`` sentinel;
2. accept exact equality of the normalized token sequences;
3. accept containment of one normalized token sequence in the other as a
contiguous subsequence (token-level, so ``5`` never matches inside ``25``);
4. accept a numeric prediction whose integer tokens all fall inside an explicit
``ranging from X ... to Y`` reference range;
5. reject conflicting integer tokens when both sides contain integers;
6. otherwise require at least ``OVERLAP_THRESHOLD`` unique non-stopword token
overlap in both directions.
Normalization lowercases, replaces every non-alphanumeric character with a
space, and maps the English number words one through ten and fifteen to digits.
The stopword list is a fixed part of this grader version; ``yes``, ``no``, and
``not`` are deliberately excluded from it because negation can be the entire
answer.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
GRADER_VERSION = "deterministic-overlap-v1"
OVERLAP_THRESHOLD = 0.6
_INSUFFICIENT_SENTINEL = "insufficient"
_NON_ALPHANUMERIC = re.compile(r"[^0-9a-z]+")
_REFERENCE_RANGE = re.compile(r"\branging from (\d+)(?:\s\S+)*?\sto (\d+)\b")
_NUMBER_WORDS = {
"one": "1",
"two": "2",
"three": "3",
"four": "4",
"five": "5",
"six": "6",
"seven": "7",
"eight": "8",
"nine": "9",
"ten": "10",
"fifteen": "15",
}
_STOPWORDS = frozenset(
(
"a an the this that these those all any both each few more most other some such same own "
"i you he she it we they me him her us them my your his its our their mine yours hers ours theirs whose "
"am is are was were be been being do does did doing have has had having will would shall should can could may might must "
"and or but nor so yet if then than because while until once although though whether "
"of in on at by for with about against between into through during before after above below to from up down out off over under again further "
"what which who whom when where why how there here only too very just also"
).split()
)
@dataclass(frozen=True)
class GradeResult:
correct: bool
rule: str
def normalize_answer(text: str) -> list[str]:
tokens = _NON_ALPHANUMERIC.split(text.lower())
return [_NUMBER_WORDS.get(token, token) for token in tokens if token]
def _is_contiguous_subsequence(needle: list[str], haystack: list[str]) -> bool:
if not needle or len(needle) > len(haystack):
return False
return any(haystack[start : start + len(needle)] == needle for start in range(len(haystack) - len(needle) + 1))
def _integer_tokens(tokens: list[str]) -> set[int]:
return {int(token) for token in tokens if token.isdigit()}
def grade_answer(prediction: str, reference: str) -> GradeResult:
prediction_tokens = normalize_answer(prediction)
reference_tokens = normalize_answer(reference)
if not reference_tokens:
raise ValueError("The grading reference must contain at least one token")
if not prediction_tokens:
return GradeResult(correct=False, rule="empty-prediction")
if prediction_tokens == [_INSUFFICIENT_SENTINEL]:
return GradeResult(correct=False, rule="insufficient")
if prediction_tokens == reference_tokens:
return GradeResult(correct=True, rule="exact")
if _is_contiguous_subsequence(reference_tokens, prediction_tokens) or _is_contiguous_subsequence(prediction_tokens, reference_tokens):
return GradeResult(correct=True, rule="substring")
prediction_integers = _integer_tokens(prediction_tokens)
reference_integers = _integer_tokens(reference_tokens)
range_match = _REFERENCE_RANGE.search(" ".join(reference_tokens))
if range_match and prediction_integers:
low, high = sorted((int(range_match.group(1)), int(range_match.group(2))))
if all(low <= value <= high for value in prediction_integers):
return GradeResult(correct=True, rule="numeric-range")
if prediction_integers and reference_integers and prediction_integers != reference_integers:
return GradeResult(correct=False, rule="numeric-conflict")
prediction_content = {token for token in prediction_tokens if token not in _STOPWORDS}
reference_content = {token for token in reference_tokens if token not in _STOPWORDS}
if not prediction_content or not reference_content:
return GradeResult(correct=False, rule="overlap-reject")
shared = prediction_content & reference_content
if len(shared) / len(prediction_content) >= OVERLAP_THRESHOLD and len(shared) / len(reference_content) >= OVERLAP_THRESHOLD:
return GradeResult(correct=True, rule="overlap-accept")
return GradeResult(correct=False, rule="overlap-reject")

26
backend/scripts/benchmark/deermem_eviction/io.py Normal file
View File

@ -0,0 +1,26 @@
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any
import yaml
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_json(path: Path) -> Any:
with path.open(encoding="utf-8") as handle:
return json.load(handle)
def load_yaml(path: Path) -> Any:
with path.open(encoding="utf-8") as handle:
return yaml.safe_load(handle)

93
backend/scripts/benchmark/deermem_eviction/manifest.py Normal file
View File

@ -0,0 +1,93 @@
from __future__ import annotations
from pathlib import Path
from typing import Literal
from pydantic import Field, field_validator, model_validator
from .config import StrictModel
from .io import load_json
OFFICIAL_SCENARIOS = {
"confirmation_help",
"access_help",
"confidence_control",
"noisy_signal_control",
}
class SelectionConfig(StrictModel):
eligible_question_types: list[Literal["knowledge-update", "temporal-reasoning"]]
excluded_pilot_ids: list[str]
exclude_abstention_suffix: str
answer_min_chars: int = Field(ge=0)
answer_max_chars: int = Field(ge=1)
answer_excluded_substrings: list[str]
evidence_min_chars: int = Field(ge=0)
evidence_max_chars: int = Field(ge=1)
take_per_question_type: int = Field(ge=1)
cases_per_type_per_scenario: int = Field(ge=1)
class OfficialManifest(StrictModel):
schema_version: Literal[1]
protocol_id: str
selection: SelectionConfig
scenario_order: list[str]
loss_ranks: list[int]
scenarios: dict[str, list[str]]
@field_validator("loss_ranks")
@classmethod
def validate_loss_ranks(cls, value: list[int]) -> list[int]:
if len(value) != 10 or any(rank < 1 or rank > 10 for rank in value):
raise ValueError("official loss_ranks must contain ten values between 1 and 10")
return value
@model_validator(mode="after")
def validate_scenarios(self) -> OfficialManifest:
if set(self.scenarios) != OFFICIAL_SCENARIOS:
raise ValueError("official manifest must define the four registered scenarios")
if len(self.scenario_order) != len(set(self.scenario_order)) or set(self.scenario_order) != OFFICIAL_SCENARIOS:
raise ValueError("official scenario_order must list each registered scenario exactly once")
all_ids: list[str] = []
for scenario, question_ids in self.scenarios.items():
if len(question_ids) != 10:
raise ValueError(f"scenario {scenario!r} must contain ten question IDs")
all_ids.extend(question_ids)
if len(all_ids) != len(set(all_ids)):
raise ValueError("official question IDs must be unique")
return self
class SyntheticCase(StrictModel):
case_id: str
support_fact: str
question: str
answer: str
loss_rank: int = Field(ge=1, le=10)
class SyntheticManifest(StrictModel):
schema_version: Literal[1]
protocol_id: str
scenario: Literal["correction_reserve"]
cases: list[SyntheticCase]
@field_validator("cases")
@classmethod
def validate_cases(cls, value: list[SyntheticCase]) -> list[SyntheticCase]:
if len(value) != 5:
raise ValueError("synthetic correction manifest must contain five cases")
case_ids = [case.case_id for case in value]
if len(case_ids) != len(set(case_ids)):
raise ValueError("synthetic case IDs must be unique")
return value
def load_official_manifest(path: Path) -> OfficialManifest:
return OfficialManifest.model_validate(load_json(path))
def load_synthetic_manifest(path: Path) -> SyntheticManifest:
return SyntheticManifest.model_validate(load_json(path))

88
backend/scripts/benchmark/deermem_eviction/manifests/longmemeval-pr4789-v1.json Normal file
View File

@ -0,0 +1,88 @@
{
"schema_version": 1,
"protocol_id": "deermem-hybrid-v1-pr4789-reproduction-v1",
"selection": {
"eligible_question_types": [
"knowledge-update",
"temporal-reasoning"
],
"excluded_pilot_ids": [
"0f05491a",
"dfde3500",
"4b24c848",
"7e974930",
"8077ef71",
"bcbe585f",
"gpt4_85da3956",
"gpt4_af6db32f"
],
"exclude_abstention_suffix": "_abs",
"answer_min_chars": 1,
"answer_max_chars": 100,
"answer_excluded_substrings": [
"not enough",
"only mentioned"
],
"evidence_min_chars": 1,
"evidence_max_chars": 2000,
"take_per_question_type": 20,
"cases_per_type_per_scenario": 5
},
"scenario_order": [
"confirmation_help",
"access_help",
"confidence_control",
"noisy_signal_control"
],
"loss_ranks": [6, 6, 6, 8, 8, 8, 10, 10, 10, 10],
"scenarios": {
"confirmation_help": [
"01493427",
"06db6396",
"07741c44",
"07741c45",
"08e075c7",
"08f4fc43",
"0bb5a684",
"0bc8ad92",
"0db4c65d",
"2a1811e2"
],
"access_help": [
"0977f2af",
"0ddfec37",
"0e4e4c46",
"10e09553",
"184da446",
"2c63a862",
"2ebe6c90",
"2ebe6c92",
"370a8ff4",
"4dfccbf7"
],
"confidence_control": [
"18bc8abd",
"1cea1afa",
"2133c1b5",
"22d2cb42",
"2698e78f",
"5e1b23de",
"6e984301",
"71017276",
"71017277",
"8c18457d"
],
"noisy_signal_control": [
"26bdc477",
"3ba21379",
"41698283",
"42ec0761",
"45dc21b6",
"982b5123",
"993da5e2",
"9a707b81",
"a3045048",
"af082822"
]
}
}

42
backend/scripts/benchmark/deermem_eviction/manifests/synthetic-corrections-pr4789-v1.json Normal file
View File

@ -0,0 +1,42 @@
{
"schema_version": 1,
"protocol_id": "deermem-hybrid-v1-pr4789-reproduction-v1",
"scenario": "correction_reserve",
"cases": [
{
"case_id": "correction_peanut_allergy",
"support_fact": "USER CORRECTION: I am allergic to peanuts. The earlier note saying I enjoy peanut snacks was wrong.",
"question": "Should you recommend peanut snacks to me? Answer YES or NO.",
"answer": "NO",
"loss_rank": 6
},
{
"case_id": "correction_language",
"support_fact": "USER CORRECTION: My preferred response language is Chinese, not English.",
"question": "Should responses default to English? Answer YES or NO.",
"answer": "NO",
"loss_rank": 8
},
{
"case_id": "correction_timezone",
"support_fact": "USER CORRECTION: My current timezone is Asia/Shanghai, not UTC.",
"question": "Should reminders use UTC as my timezone? Answer YES or NO.",
"answer": "NO",
"loss_rank": 8
},
{
"case_id": "correction_review_resolution",
"support_fact": "USER CORRECTION: Never resolve a review thread merely because a reply was posted; wait for explicit authorization.",
"question": "May a review thread be resolved automatically after posting a reply? Answer YES or NO.",
"answer": "NO",
"loss_rank": 10
},
{
"case_id": "correction_shipping_address",
"support_fact": "USER CORRECTION: My current shipping address is in Shanghai, not Beijing.",
"question": "Should a package be shipped to Beijing? Answer YES or NO.",
"answer": "NO",
"loss_rank": 10
}
]
}

107
backend/scripts/benchmark/deermem_eviction/policy.py Normal file
View File

@ -0,0 +1,107 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Literal
from deerflow.agents.memory.backends.deermem.deermem.core.eviction import EVICTION_POLICY_HYBRID_V1, select_facts_for_capacity
from .config import HybridPolicyConfig
from .pool import PreparedCase
PolicyName = Literal["confidence", "hybrid-v1"]
def require_production_policy(version: str) -> None:
"""Reject a config whose required policy version has drifted from the production implementation."""
if version != EVICTION_POLICY_HYBRID_V1:
raise ValueError(f"config requires eviction policy {version!r} but production implements {EVICTION_POLICY_HYBRID_V1!r}")
@dataclass(frozen=True)
class PolicyResult:
case_id: str
source: str
scenario: str
question_type: str
policy: PolicyName
capacity: int
support_fact_ids: tuple[str, ...]
kept_fact_ids: tuple[str, ...]
evicted: tuple[dict[str, Any], ...]
scores: dict[str, dict[str, Any]]
support_all_retained: bool
support_recall: float
reserved_correction_slots: int
def to_public_dict(self) -> dict[str, Any]:
return {
"schema_version": 1,
"case_id": self.case_id,
"source": self.source,
"scenario": self.scenario,
"question_type": self.question_type,
"policy": self.policy,
"capacity": self.capacity,
"support_fact_ids": list(self.support_fact_ids),
"kept_fact_ids": list(self.kept_fact_ids),
"evicted": list(self.evicted),
"scores": self.scores,
"support_all_retained": self.support_all_retained,
"support_recall": self.support_recall,
"reserved_correction_slots": self.reserved_correction_slots,
}
def evaluate_case(
case: PreparedCase,
*,
policy_name: PolicyName,
capacity: int,
hybrid_config: HybridPolicyConfig | None = None,
) -> PolicyResult:
kwargs: dict[str, Any] = {}
if hybrid_config is not None:
kwargs = {
"confidence_weight": hybrid_config.weights["confidence"],
"confirmation_weight": hybrid_config.weights["confirmation"],
"access_weight": hybrid_config.weights["access"],
"confirmation_half_life_days": hybrid_config.confirmation_half_life_days,
"access_half_life_days": hybrid_config.access_half_life_days,
"correction_reserved_fraction": hybrid_config.correction_reserved_fraction,
"correction_reserved_max": hybrid_config.correction_reserved_max,
}
decision = select_facts_for_capacity(
case.facts,
max_facts=capacity,
policy=policy_name,
usage=case.usage,
now=case.evaluation_time,
**kwargs,
)
kept_fact_ids = tuple(str(fact["id"]) for fact in decision.kept)
retained_support = set(case.support_fact_ids) & set(kept_fact_ids)
evicted = tuple(
{
"fact_id": item.fact_id,
"category": item.category,
"score": item.score,
"components": dict(item.components),
}
for item in decision.evicted
)
scores = {fact_id: {"value": score.value, "components": dict(score.components)} for fact_id, score in sorted(decision.scores.items())}
return PolicyResult(
case_id=case.case_id,
source=case.source,
scenario=case.scenario,
question_type=case.question_type,
policy=policy_name,
capacity=capacity,
support_fact_ids=case.support_fact_ids,
kept_fact_ids=kept_fact_ids,
evicted=evicted,
scores=scores,
support_all_retained=len(retained_support) == len(case.support_fact_ids),
support_recall=len(retained_support) / len(case.support_fact_ids),
reserved_correction_slots=decision.reserved_correction_slots,
)

155
backend/scripts/benchmark/deermem_eviction/pool.py Normal file
View File

@ -0,0 +1,155 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any, Literal
from .dataset import EvidenceRecord
Scenario = Literal["confirmation_help", "access_help", "confidence_control", "noisy_signal_control", "correction_reserve"]
@dataclass(frozen=True)
class PreparedCase:
case_id: str
source: Literal["longmemeval", "synthetic"]
scenario: Scenario
question_type: str
question: str
answer: str
question_date: str | None
evaluation_time: datetime
facts: list[dict[str, Any]]
usage: dict[str, dict[str, Any]]
support_fact_ids: tuple[str, ...]
@dataclass(frozen=True)
class _FactMetadata:
confidence: float
category: str
created_days_ago: int
last_confirmed_days_ago: int | None = None
access_heat: float = 0.0
def _timestamp_days_ago(now: datetime, days: int) -> str:
value = now.astimezone(UTC) - timedelta(days=days)
return value.isoformat().removesuffix("+00:00") + "Z"
def _support_metadata(scenario: Scenario) -> _FactMetadata:
if scenario == "confirmation_help":
return _FactMetadata(confidence=0.70, category="context", created_days_ago=180, last_confirmed_days_ago=7)
if scenario == "access_help":
return _FactMetadata(confidence=0.70, category="context", created_days_ago=180, access_heat=8)
if scenario == "confidence_control":
return _FactMetadata(confidence=0.95, category="context", created_days_ago=30)
if scenario == "noisy_signal_control":
return _FactMetadata(confidence=0.90, category="context", created_days_ago=180)
if scenario == "correction_reserve":
return _FactMetadata(confidence=0.65, category="correction", created_days_ago=180)
raise ValueError(f"Unknown evaluation scenario: {scenario}")
def _distractor_metadata(scenario: Scenario, *, index: int, outranking_count: int, question_type: str) -> _FactMetadata:
category = "preference" if question_type == "single-session-preference" else "context"
if scenario in {"confirmation_help", "access_help"}:
confidence = 0.94 - 0.02 * index if index < outranking_count else 0.68 - 0.02 * (index - outranking_count)
return _FactMetadata(confidence=confidence, category=category, created_days_ago=180)
if scenario == "confidence_control":
if index < 5:
return _FactMetadata(confidence=0.94 - 0.02 * index, category=category, created_days_ago=180)
return _FactMetadata(
confidence=0.70,
category=category,
created_days_ago=180,
last_confirmed_days_ago=7 if index % 2 == 1 else None,
access_heat=8 if index % 2 == 0 else 0,
)
if scenario == "noisy_signal_control":
has_noise = index < outranking_count
return _FactMetadata(
confidence=0.70,
category=category,
created_days_ago=180,
last_confirmed_days_ago=7 if has_noise and index % 2 == 1 else None,
access_heat=8 if has_noise and index % 2 == 0 else 0,
)
if scenario == "correction_reserve":
confidence = 0.90 - 0.03 * index if index < outranking_count else 0.60 - 0.02 * (index - outranking_count)
return _FactMetadata(confidence=confidence, category=category, created_days_ago=180)
raise ValueError(f"Unknown evaluation scenario: {scenario}")
def _fact(record: EvidenceRecord, metadata: _FactMetadata, *, fact_id: str, evaluation_time: datetime) -> tuple[dict[str, Any], dict[str, Any] | None]:
fact: dict[str, Any] = {
"id": fact_id,
"content": record.content,
"category": metadata.category,
"confidence": metadata.confidence,
"createdAt": _timestamp_days_ago(evaluation_time, metadata.created_days_ago),
"source": f"deermem-eviction-eval:{record.question_id}",
}
if metadata.last_confirmed_days_ago is not None:
fact["lastConfirmedAt"] = _timestamp_days_ago(evaluation_time, metadata.last_confirmed_days_ago)
fact["confirmationCount"] = 1
usage = None
if metadata.access_heat > 0:
usage = {
"accessHeat": metadata.access_heat,
"accessCount": int(metadata.access_heat),
"lastAccessedAt": evaluation_time.astimezone(UTC).isoformat().removesuffix("+00:00") + "Z",
}
return fact, usage
def build_case(
*,
support: EvidenceRecord,
distractors: list[EvidenceRecord],
scenario: Scenario,
loss_rank: int,
evaluation_time: datetime,
source: Literal["longmemeval", "synthetic"] | None = None,
) -> PreparedCase:
if len(distractors) != 9:
raise ValueError("The pr4789 reproduction protocol requires exactly nine distractors")
if not 1 <= loss_rank <= 10:
raise ValueError("loss_rank must be between 1 and 10")
source_ids = [support.question_id, *(item.question_id for item in distractors)]
if len(source_ids) != len(set(source_ids)):
raise ValueError("A prepared pool cannot contain duplicate source records")
facts: list[dict[str, Any]] = []
usage: dict[str, dict[str, Any]] = {}
support_fact_id = f"gold_{support.question_id}"
support_fact, support_usage = _fact(support, _support_metadata(scenario), fact_id=support_fact_id, evaluation_time=evaluation_time)
facts.append(support_fact)
if support_usage is not None:
usage[support_fact_id] = support_usage
outranking_count = loss_rank - 1
for index, record in enumerate(distractors):
metadata = _distractor_metadata(scenario, index=index, outranking_count=outranking_count, question_type=record.question_type)
fact_id = f"d_{support.question_id}_{index}_{record.question_id}"
fact, fact_usage = _fact(record, metadata, fact_id=fact_id, evaluation_time=evaluation_time)
facts.append(fact)
if fact_usage is not None:
usage[fact_id] = fact_usage
facts.sort(key=lambda fact: str(fact["id"]))
resolved_source = source or ("synthetic" if scenario == "correction_reserve" else "longmemeval")
return PreparedCase(
case_id=support.question_id,
source=resolved_source,
scenario=scenario,
question_type=support.question_type,
question=support.question,
answer=support.answer,
question_date=support.question_date,
evaluation_time=evaluation_time.astimezone(UTC),
facts=facts,
usage=usage,
support_fact_ids=(support_fact_id,),
)

8
backend/scripts/benchmark/deermem_eviction/prompts/answer-v1.txt Normal file
View File

@ -0,0 +1,8 @@
SYSTEM:
Answer using only the stored memory below. If the answer is not supported, output exactly INSUFFICIENT. For a YES/NO question, output only YES or NO. Otherwise give only the shortest direct answer.
USER:
{{CURRENT_DATE_SECTION}}STORED MEMORY:
{{STORED_MEMORY}}
QUESTION: {{QUESTION}}

120
backend/scripts/benchmark/deermem_eviction/protocol.py Normal file
View File

@ -0,0 +1,120 @@
from __future__ import annotations
import hashlib
from typing import cast
from .config import EvaluationConfig
from .dataset import DatasetIntegrityError, EvidenceRecord, LongMemEvalDataset, build_distractor_bank, evidence_record, extract_evidence
from .manifest import OfficialManifest, SyntheticManifest
from .pool import PreparedCase, Scenario, build_case
def _eligible_official_ids(dataset: LongMemEvalDataset, manifest: OfficialManifest) -> dict[str, list[str]]:
selection = manifest.selection
by_type: dict[str, list[str]] = {question_type: [] for question_type in selection.eligible_question_types}
excluded_pilot_ids = set(selection.excluded_pilot_ids)
excluded_answer_fragments = [fragment.lower() for fragment in selection.answer_excluded_substrings]
for row in dataset.rows:
question_type = row.get("question_type")
question_id = row.get("question_id")
if question_type not in by_type or not isinstance(question_id, str):
continue
if question_id in excluded_pilot_ids or question_id.endswith(selection.exclude_abstention_suffix):
continue
answer = evidence_record(row).answer.strip()
if not selection.answer_min_chars <= len(answer) <= selection.answer_max_chars:
continue
if any(fragment in answer.lower() for fragment in excluded_answer_fragments):
continue
evidence = extract_evidence(row)
if not selection.evidence_min_chars <= len(evidence) <= selection.evidence_max_chars:
continue
by_type[question_type].append(question_id)
return {question_type: sorted(question_ids)[: selection.take_per_question_type] for question_type, question_ids in by_type.items()}
def validate_official_selection(dataset: LongMemEvalDataset, manifest: OfficialManifest) -> None:
selected_by_type = _eligible_official_ids(dataset, manifest)
scenario_names = manifest.scenario_order
group_size = manifest.selection.cases_per_type_per_scenario
expected_take = len(scenario_names) * group_size
if manifest.selection.take_per_question_type != expected_take:
raise DatasetIntegrityError("official selection count does not match scenario grouping")
for scenario_index, scenario in enumerate(scenario_names):
expected: list[str] = []
start = scenario_index * group_size
end = start + group_size
for question_type in manifest.selection.eligible_question_types:
candidates = selected_by_type[question_type]
if len(candidates) != manifest.selection.take_per_question_type:
raise DatasetIntegrityError(f"not enough eligible {question_type!r} rows for the pinned selection")
expected.extend(candidates[start:end])
if manifest.scenarios[scenario] != expected:
raise DatasetIntegrityError(f"pinned IDs for {scenario!r} do not match the declared selection rule: expected {expected}, got {manifest.scenarios[scenario]}")
def _distractors(case_id: str, bank: list[EvidenceRecord], *, count: int, namespace: str) -> list[EvidenceRecord]:
if len(bank) < count:
raise DatasetIntegrityError(f"distractor bank has {len(bank)} rows but {count} are required")
digest = hashlib.sha256(f"{namespace}:{case_id}".encode()).digest()
offset = int.from_bytes(digest[:4], "big") % len(bank)
return [bank[(offset + index) % len(bank)] for index in range(count)]
def build_protocol_cases(
dataset: LongMemEvalDataset,
config: EvaluationConfig,
official_manifest: OfficialManifest,
synthetic_manifest: SyntheticManifest,
) -> list[PreparedCase]:
if len(dataset.rows_by_id) != len(dataset.rows):
raise DatasetIntegrityError("LongMemEval question IDs are not unique")
validate_official_selection(dataset, official_manifest)
bank = build_distractor_bank(
dataset.rows,
allowed_types=set(config.pool.distractor_types),
min_evidence_chars=config.pool.distractor_min_evidence_chars,
max_evidence_chars=config.pool.distractor_max_evidence_chars,
limit=config.pool.distractor_bank_size,
)
if len(bank) != config.pool.distractor_bank_size:
raise DatasetIntegrityError(f"distractor bank has {len(bank)} rows; expected {config.pool.distractor_bank_size}")
cases: list[PreparedCase] = []
for scenario_name in official_manifest.scenario_order:
question_ids = official_manifest.scenarios[scenario_name]
scenario = cast(Scenario, scenario_name)
sorted_ids = sorted(question_ids, key=lambda question_id: (str(dataset.rows_by_id[question_id].get("question_type")), question_id))
for question_id, loss_rank in zip(sorted_ids, official_manifest.loss_ranks, strict=True):
support = evidence_record(dataset.rows_by_id[question_id])
cases.append(
build_case(
support=support,
distractors=_distractors(question_id, bank, count=config.pool.distractors, namespace=config.pool.offset_namespace),
scenario=scenario,
loss_rank=loss_rank,
evaluation_time=config.evaluation_time,
source="longmemeval",
)
)
for case in synthetic_manifest.cases:
support = EvidenceRecord(
question_id=case.case_id,
question_type="synthetic-correction",
question=case.question,
answer=case.answer,
question_date=None,
content=case.support_fact,
)
cases.append(
build_case(
support=support,
distractors=_distractors(case.case_id, bank, count=config.pool.distractors, namespace=config.pool.offset_namespace),
scenario="correction_reserve",
loss_rank=case.loss_rank,
evaluation_time=config.evaluation_time,
source="synthetic",
)
)
return cases

102
backend/scripts/benchmark/deermem_eviction/provider.py Normal file
View File

@ -0,0 +1,102 @@
"""OpenAI-compatible answer provider configured only through named environment variables.
Credentials and the endpoint are resolved from the environment variable names
recorded in the versioned config; they are never read from files, logged, or
persisted. Error messages name the missing variables, never their values.
Responses are reduced to the prediction text and non-secret metadata — response
headers and complete payloads are never returned to callers.
"""
from __future__ import annotations
import hashlib
import json
import os
import time
from dataclasses import dataclass
import httpx
from .config import QAConfig
RETRY_BACKOFF_SECONDS = 2.0
class ProviderConfigurationError(RuntimeError):
pass
class ProviderCallError(RuntimeError):
pass
@dataclass(frozen=True)
class ProviderSettings:
base_url: str
api_key: str
@dataclass(frozen=True)
class AnswerCall:
prediction: str
attempts: int
request_fingerprint: str
response_model: str | None
usage: dict[str, int]
def resolve_provider_settings(qa: QAConfig) -> ProviderSettings:
base_url = os.environ.get(qa.base_url_env, "").strip()
api_key = os.environ.get(qa.api_key_env, "").strip()
missing = [name for name, value in ((qa.base_url_env, base_url), (qa.api_key_env, api_key)) if not value]
if missing:
raise ProviderConfigurationError(f"missing provider environment variables: {', '.join(missing)}")
return ProviderSettings(base_url=base_url, api_key=api_key)
def build_client(settings: ProviderSettings, qa: QAConfig) -> httpx.Client:
return httpx.Client(base_url=settings.base_url, headers={"Authorization": f"Bearer {settings.api_key}"}, timeout=qa.timeout_seconds)
def _is_retryable_status(status_code: int) -> bool:
return status_code == 429 or status_code >= 500
def request_payload(qa: QAConfig, messages: tuple[dict[str, str], ...]) -> dict[str, object]:
return {"model": qa.model, "temperature": qa.temperature, "max_tokens": qa.max_tokens, "stream": qa.stream, "messages": list(messages)}
def request_fingerprint(qa: QAConfig, messages: tuple[dict[str, str], ...]) -> str:
return hashlib.sha256(json.dumps(request_payload(qa, messages), ensure_ascii=False, sort_keys=True).encode()).hexdigest()
def request_answer(client: httpx.Client, qa: QAConfig, messages: tuple[dict[str, str], ...], *, backoff_seconds: float = RETRY_BACKOFF_SECONDS) -> AnswerCall:
payload = request_payload(qa, messages)
fingerprint = request_fingerprint(qa, messages)
last_error = "no attempt was made"
for attempt in range(1, qa.max_attempts + 1):
if attempt > 1 and backoff_seconds > 0:
time.sleep(backoff_seconds)
try:
response = client.post("/chat/completions", json=payload)
except httpx.HTTPError as error:
last_error = f"network error: {type(error).__name__}"
continue
if _is_retryable_status(response.status_code):
last_error = f"retryable status {response.status_code}"
continue
if response.status_code != 200:
raise ProviderCallError(f"provider returned non-retryable status {response.status_code}")
try:
body = response.json()
message = body["choices"][0]["message"]
prediction = message["content"]
except (ValueError, KeyError, IndexError, TypeError) as error:
raise ProviderCallError("provider response is missing choices[0].message.content") from error
if not isinstance(prediction, str):
raise ProviderCallError("provider prediction is not a string")
raw_usage = body.get("usage")
usage = {key: value for key, value in raw_usage.items() if isinstance(value, int)} if isinstance(raw_usage, dict) else {}
response_model = body.get("model") if isinstance(body.get("model"), str) else None
return AnswerCall(prediction=prediction, attempts=attempt, request_fingerprint=fingerprint, response_model=response_model, usage=usage)
raise ProviderCallError(f"provider call failed after {qa.max_attempts} attempts: {last_error}")

77
backend/scripts/benchmark/deermem_eviction/qa.py Normal file
View File

@ -0,0 +1,77 @@
"""Answer-prompt rendering for the live QA comparison.
Rendering is pinned by the committed template (hash-checked by the config) and
by this module: retained facts are sorted by their historical protocol fact IDs
(``gold_{case}`` / ``d_{case}_{index}_{source}``), each fact renders as
``[{fact_id}]`` followed by its content on the next line, fact blocks are
joined with a blank line, and the ``CURRENT DATE`` line is omitted when the
case has no question date.
"""
from __future__ import annotations
from dataclasses import dataclass
from .policy import PolicyResult
from .pool import PreparedCase
_SYSTEM_HEADER = "SYSTEM:\n"
_USER_SEPARATOR = "\n\nUSER:\n"
_CURRENT_DATE_PLACEHOLDER = "{{CURRENT_DATE_SECTION}}"
_STORED_MEMORY_PLACEHOLDER = "{{STORED_MEMORY}}"
_QUESTION_PLACEHOLDER = "{{QUESTION}}"
@dataclass(frozen=True)
class AnswerTask:
row_id: str
case_id: str
source: str
scenario: str
policy: str
capacity: int
kept_fact_ids: tuple[str, ...]
messages: tuple[dict[str, str], ...]
def split_prompt_template(template: str) -> tuple[str, str]:
if not template.startswith(_SYSTEM_HEADER) or _USER_SEPARATOR not in template:
raise ValueError("The answer prompt template must contain SYSTEM: and USER: sections")
system_part, user_part = template.removeprefix(_SYSTEM_HEADER).split(_USER_SEPARATOR, 1)
for placeholder in (_CURRENT_DATE_PLACEHOLDER, _STORED_MEMORY_PLACEHOLDER, _QUESTION_PLACEHOLDER):
if placeholder not in user_part:
raise ValueError(f"The answer prompt template is missing {placeholder}")
return system_part.strip("\n"), user_part.strip("\n")
def render_answer_messages(template: str, *, question: str, question_date: str | None, retained_facts: list[tuple[str, str]]) -> tuple[dict[str, str], ...]:
if not retained_facts:
raise ValueError("Rendering requires at least one retained fact")
system_part, user_part = split_prompt_template(template)
ordered = sorted(retained_facts, key=lambda fact: fact[0])
if len({fact_id for fact_id, _ in ordered}) != len(ordered):
raise ValueError("Retained facts must have unique IDs")
current_date_section = f"CURRENT DATE: {question_date}\n" if question_date else ""
stored_memory = "\n\n".join(f"[{fact_id}]\n{content}" for fact_id, content in ordered)
user = user_part.replace(_CURRENT_DATE_PLACEHOLDER, current_date_section).replace(_STORED_MEMORY_PLACEHOLDER, stored_memory).replace(_QUESTION_PLACEHOLDER, question)
return ({"role": "system", "content": system_part}, {"role": "user", "content": user})
def build_answer_task(case: PreparedCase, result: PolicyResult, template: str) -> AnswerTask:
if result.case_id != case.case_id:
raise ValueError(f"policy result {result.case_id!r} does not belong to case {case.case_id!r}")
kept = set(result.kept_fact_ids)
retained_facts = [(str(fact["id"]), str(fact["content"])) for fact in case.facts if str(fact["id"]) in kept]
if len(retained_facts) != len(kept):
raise ValueError(f"case {case.case_id!r} is missing content for kept facts")
messages = render_answer_messages(template, question=case.question, question_date=case.question_date, retained_facts=retained_facts)
return AnswerTask(
row_id=f"{case.case_id}__{result.policy}",
case_id=case.case_id,
source=case.source,
scenario=case.scenario,
policy=result.policy,
capacity=result.capacity,
kept_fact_ids=result.kept_fact_ids,
messages=messages,
)

145
backend/scripts/benchmark/deermem_eviction/report.py Normal file
View File

@ -0,0 +1,145 @@
"""Blind grading of completed answer rows and public QA reporting.
Grading is join-free by construction: every prediction is graded through
``grade_answer(prediction, reference)`` — two strings, no policy identity —
and only afterwards joined back to its policy through the stable row ID.
Published rows carry predictions, grades, and non-secret metadata; they never
contain questions, reference answers, or memory content. The per-scenario
summary keeps the official and synthetic sources separate; the statistics
report the ``official`` and ``synthetic`` suites separately and additionally
an explicitly labeled combined ``overall`` suite.
"""
from __future__ import annotations
import json
from collections import defaultdict
from dataclasses import asdict
from pathlib import Path
from typing import Any
from .config import EvaluationConfig
from .grading import GRADER_VERSION, grade_answer
from .policy import PolicyResult
from .pool import PreparedCase
from .results import _atomic_write_text
from .runner import load_completed_row, response_path
from .stats import exact_mcnemar, paired_bootstrap_difference
POLICY_ORDER = ("confidence", "hybrid-v1")
class AnswerRowIntegrityError(RuntimeError):
pass
def collect_answer_rows(output_dir: Path, cases: list[PreparedCase]) -> dict[str, dict[str, Any]]:
rows: dict[str, dict[str, Any]] = {}
missing: list[str] = []
for case in cases:
for policy in POLICY_ORDER:
row_id = f"{case.case_id}__{policy}"
row = load_completed_row(response_path(output_dir, row_id))
if row is None:
missing.append(row_id)
else:
rows[row_id] = row
if missing:
raise AnswerRowIntegrityError(f"missing or invalid answer rows: {', '.join(sorted(missing))}")
return rows
def grade_answer_rows(cases: list[PreparedCase], results_by_row: dict[str, PolicyResult], rows: dict[str, dict[str, Any]], *, expected_fingerprints: dict[str, str]) -> list[dict[str, Any]]:
cases_by_id = {case.case_id: case for case in cases}
graded: list[dict[str, Any]] = []
for row_id in sorted(rows):
row = rows[row_id]
result = results_by_row[row_id]
# The recomputed task is authoritative: the reference case is derived from it,
# never from the stored row, and every persisted identity field must match it.
case = cases_by_id[result.case_id]
if row.get("row_id") != row_id or row.get("case_id") != result.case_id or row.get("source") != result.source or row.get("scenario") != result.scenario:
raise AnswerRowIntegrityError(f"row {row_id} case identity does not match the expected task")
if tuple(row.get("kept_fact_ids", ())) != result.kept_fact_ids:
raise AnswerRowIntegrityError(f"row {row_id} kept facts do not match the deterministic selector output")
if row.get("capacity") != result.capacity or row.get("policy") != result.policy:
raise AnswerRowIntegrityError(f"row {row_id} capacity/policy does not match the protocol")
if row.get("request_fingerprint") != expected_fingerprints.get(row_id):
raise AnswerRowIntegrityError(f"row {row_id} request fingerprint does not match the task recomputed from the current protocol")
prediction = str(row["prediction"])
grade = grade_answer(prediction, case.answer)
graded.append(
{
"schema_version": 1,
"row_id": row_id,
"case_id": case.case_id,
"source": case.source,
"scenario": case.scenario,
"question_type": case.question_type,
"policy": result.policy,
"capacity": result.capacity,
"kept_fact_ids": list(result.kept_fact_ids),
"support_all_retained": result.support_all_retained,
"support_recall": result.support_recall,
"prediction": prediction,
"grade_correct": grade.correct,
"grade_rule": grade.rule,
"grader_version": GRADER_VERSION,
"attempts": row.get("attempts"),
"request_fingerprint": row.get("request_fingerprint"),
"response_model": row.get("response_model"),
"usage": row.get("usage"),
}
)
return graded
def summarize_qa_rows(graded: list[dict[str, Any]]) -> list[dict[str, Any]]:
groups: dict[tuple[str, str, str], list[dict[str, Any]]] = defaultdict(list)
for row in graded:
groups[(row["source"], row["scenario"], row["policy"])].append(row)
summary: list[dict[str, Any]] = []
for (source, scenario, policy), rows in sorted(groups.items()):
correct = sum(1 for row in rows if row["grade_correct"])
summary.append({"source": source, "scenario": scenario, "policy": policy, "cases": len(rows), "correct": correct, "accuracy": correct / len(rows)})
return summary
def _paired_grades(graded: list[dict[str, Any]]) -> dict[str, tuple[bool, bool]]:
by_case: dict[str, dict[str, bool]] = defaultdict(dict)
for row in graded:
by_case[row["case_id"]][row["policy"]] = bool(row["grade_correct"])
pairs: dict[str, tuple[bool, bool]] = {}
for case_id, grades in by_case.items():
if set(grades) != set(POLICY_ORDER):
raise AnswerRowIntegrityError(f"case {case_id} is missing one policy row")
pairs[case_id] = (grades[POLICY_ORDER[0]], grades[POLICY_ORDER[1]])
return pairs
def compute_qa_statistics(graded: list[dict[str, Any]], config: EvaluationConfig) -> dict[str, Any]:
pairs_by_case = _paired_grades(graded)
sources_by_case = {row["case_id"]: row["source"] for row in graded}
suites = {
"official": [pairs_by_case[case_id] for case_id in sorted(pairs_by_case) if sources_by_case[case_id] == "longmemeval"],
"synthetic": [pairs_by_case[case_id] for case_id in sorted(pairs_by_case) if sources_by_case[case_id] == "synthetic"],
"overall": [pairs_by_case[case_id] for case_id in sorted(pairs_by_case)],
}
statistics: dict[str, Any] = {"schema_version": 1, "protocol_id": config.protocol_id, "grader_version": GRADER_VERSION, "policies": list(POLICY_ORDER), "suites": {}}
for suite, pairs in suites.items():
mcnemar = exact_mcnemar(pairs)
bootstrap = paired_bootstrap_difference(pairs, seed=config.statistics.bootstrap_seed, iterations=config.statistics.bootstrap_iterations, alpha=config.statistics.alpha)
statistics["suites"][suite] = {"cases": len(pairs), "mcnemar": asdict(mcnemar), "bootstrap": asdict(bootstrap)}
return statistics
def write_qa_report(output_dir: Path, *, graded: list[dict[str, Any]], summary: list[dict[str, Any]], statistics: dict[str, Any], config: EvaluationConfig) -> None:
targets = [output_dir / "qa.rows.jsonl", output_dir / "qa.summary.json", output_dir / "qa.stats.json"]
existing = [path for path in targets if path.exists()]
if existing:
raise FileExistsError(f"refusing to overwrite existing result files: {', '.join(str(path) for path in existing)}")
rows_lines = "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in graded)
summary_document = {"schema_version": 1, "protocol_id": config.protocol_id, "grader_version": GRADER_VERSION, "qa_capacity": config.pool.qa_capacity, "groups": summary}
_atomic_write_text(output_dir / "qa.rows.jsonl", rows_lines)
_atomic_write_text(output_dir / "qa.summary.json", json.dumps(summary_document, ensure_ascii=False, indent=2, sort_keys=True) + "\n")
_atomic_write_text(output_dir / "qa.stats.json", json.dumps(statistics, ensure_ascii=False, indent=2, sort_keys=True) + "\n")

100
backend/scripts/benchmark/deermem_eviction/results.py Normal file
View File

@ -0,0 +1,100 @@
from __future__ import annotations
import json
import os
import subprocess
import tempfile
from collections import defaultdict
from collections.abc import Iterable
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from .config import EvaluationConfig
from .io import sha256_file
from .policy import PolicyResult
def _atomic_write_text(path: Path, content: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=path.parent, delete=False) as handle:
handle.write(content)
temporary_path = Path(handle.name)
os.replace(temporary_path, path)
def _git_metadata(backend_root: Path) -> dict[str, Any]:
def run(*args: str) -> str:
completed = subprocess.run(["git", *args], cwd=backend_root, check=True, capture_output=True, text=True)
return completed.stdout.strip()
return {
"head": run("rev-parse", "HEAD"),
"dirty": bool(run("status", "--porcelain")),
}
def summarize_policy_results(results: Iterable[PolicyResult]) -> list[dict[str, Any]]:
groups: dict[tuple[str, str, int, str], list[PolicyResult]] = defaultdict(list)
for result in results:
groups[(result.source, result.scenario, result.capacity, result.policy)].append(result)
summary: list[dict[str, Any]] = []
for (source, scenario, capacity, policy), rows in sorted(groups.items()):
retained = sum(row.support_all_retained for row in rows)
summary.append(
{
"source": source,
"scenario": scenario,
"capacity": capacity,
"policy": policy,
"cases": len(rows),
"support_all_retained": retained,
"support_all_retained_rate": retained / len(rows),
"mean_support_recall": sum(row.support_recall for row in rows) / len(rows),
}
)
return summary
def write_policy_run(
output_dir: Path,
*,
results: list[PolicyResult],
config: EvaluationConfig,
config_path: Path,
official_manifest_path: Path,
synthetic_manifest_path: Path,
prompt_path: Path,
dataset_path: Path,
backend_root: Path,
) -> None:
targets = [output_dir / "run.json", output_dir / "policy.raw.jsonl", output_dir / "summary.json"]
existing = [path for path in targets if path.exists()]
if existing:
raise FileExistsError(f"refusing to overwrite existing result files: {', '.join(str(path) for path in existing)}")
raw_lines = "".join(json.dumps(result.to_public_dict(), ensure_ascii=False, sort_keys=True) + "\n" for result in results)
summary = {"schema_version": 1, "protocol_id": config.protocol_id, "groups": summarize_policy_results(results)}
run = {
"schema_version": 1,
"protocol_id": config.protocol_id,
"created_at": datetime.now(UTC).isoformat().removesuffix("+00:00") + "Z",
"git": _git_metadata(backend_root),
"dataset": {
"repository": config.dataset.repository,
"revision": config.dataset.revision,
"filename": config.dataset.filename,
"sha256": sha256_file(dataset_path),
},
"artifacts": {
"config_sha256": sha256_file(config_path),
"official_manifest_sha256": sha256_file(official_manifest_path),
"synthetic_manifest_sha256": sha256_file(synthetic_manifest_path),
"answer_prompt_sha256": sha256_file(prompt_path),
},
"evaluation_time": config.evaluation_time.isoformat(),
"policies": ["confidence", "hybrid-v1"],
"capacities": config.pool.capacities,
}
_atomic_write_text(output_dir / "policy.raw.jsonl", raw_lines)
_atomic_write_text(output_dir / "summary.json", json.dumps(summary, ensure_ascii=False, indent=2, sort_keys=True) + "\n")
_atomic_write_text(output_dir / "run.json", json.dumps(run, ensure_ascii=False, indent=2, sort_keys=True) + "\n")

90
backend/scripts/benchmark/deermem_eviction/results/pr4789-reproduction-v1/qa.rows.jsonl Normal file
View File

@ -0,0 +1,90 @@
{"attempts": 1, "capacity": 7, "case_id": "01493427", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_01493427_0_09d032c9", "d_01493427_1_0a34ad58", "d_01493427_2_0edc2aef", "d_01493427_3_118b2229", "d_01493427_4_15745da0", "gold_01493427", "d_01493427_5_195a1a1b"], "policy": "confidence", "prediction": "25", "question_type": "knowledge-update", "request_fingerprint": "422fa7d0efd98d48b9f3c85c1647cd31766b4fb111ea061d9e0ef8c7fa45292b", "response_model": "deepseek-v4-flash", "row_id": "01493427__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 173, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 881, "prompt_tokens": 881, "total_tokens": 1054}}
{"attempts": 1, "capacity": 7, "case_id": "01493427", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_01493427", "d_01493427_0_09d032c9", "d_01493427_1_0a34ad58", "d_01493427_2_0edc2aef", "d_01493427_3_118b2229", "d_01493427_4_15745da0", "d_01493427_5_195a1a1b"], "policy": "hybrid-v1", "prediction": "25", "question_type": "knowledge-update", "request_fingerprint": "422fa7d0efd98d48b9f3c85c1647cd31766b4fb111ea061d9e0ef8c7fa45292b", "response_model": "deepseek-v4-flash", "row_id": "01493427__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 242, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 881, "prompt_tokens": 881, "total_tokens": 1123}}
{"attempts": 1, "capacity": 7, "case_id": "06db6396", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_06db6396_0_07b6f563", "d_06db6396_1_0862e8bf", "d_06db6396_2_09d032c9", "d_06db6396_3_0a34ad58", "d_06db6396_4_0edc2aef", "gold_06db6396", "d_06db6396_5_118b2229"], "policy": "confidence", "prediction": "5", "question_type": "knowledge-update", "request_fingerprint": "4afe5d528764bed25c00c0b8f2d46e19d2bdaee8454327760f9353479bb19173", "response_model": "deepseek-v4-flash", "row_id": "06db6396__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 133, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 891, "prompt_tokens": 891, "total_tokens": 1024}}
{"attempts": 1, "capacity": 7, "case_id": "06db6396", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_06db6396", "d_06db6396_0_07b6f563", "d_06db6396_1_0862e8bf", "d_06db6396_2_09d032c9", "d_06db6396_3_0a34ad58", "d_06db6396_4_0edc2aef", "d_06db6396_5_118b2229"], "policy": "hybrid-v1", "prediction": "5", "question_type": "knowledge-update", "request_fingerprint": "4afe5d528764bed25c00c0b8f2d46e19d2bdaee8454327760f9353479bb19173", "response_model": "deepseek-v4-flash", "row_id": "06db6396__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 95, "prompt_cache_hit_tokens": 768, "prompt_cache_miss_tokens": 123, "prompt_tokens": 891, "total_tokens": 986}}
{"attempts": 1, "capacity": 7, "case_id": "07741c44", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_07741c44_0_1d4e3b97", "d_07741c44_1_1da05512", "d_07741c44_2_1e043500", "d_07741c44_3_1faac195", "d_07741c44_4_21436231", "gold_07741c44", "d_07741c44_5_25e5aa4f"], "policy": "confidence", "prediction": "Under my bed.", "question_type": "knowledge-update", "request_fingerprint": "7bf41f6cf580fd508d3bf1dff0851175395f8ce6a1b55bac743fd5aed4d4ce98", "response_model": "deepseek-v4-flash", "row_id": "07741c44__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 43, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1022, "prompt_tokens": 1022, "total_tokens": 1065}}
{"attempts": 1, "capacity": 7, "case_id": "07741c44", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_07741c44", "d_07741c44_0_1d4e3b97", "d_07741c44_1_1da05512", "d_07741c44_2_1e043500", "d_07741c44_3_1faac195", "d_07741c44_4_21436231", "d_07741c44_5_25e5aa4f"], "policy": "hybrid-v1", "prediction": "Under my bed.", "question_type": "knowledge-update", "request_fingerprint": "7bf41f6cf580fd508d3bf1dff0851175395f8ce6a1b55bac743fd5aed4d4ce98", "response_model": "deepseek-v4-flash", "row_id": "07741c44__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 63, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1022, "prompt_tokens": 1022, "total_tokens": 1085}}
{"attempts": 1, "capacity": 7, "case_id": "07741c45", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_07741c45_0_37d43f65", "d_07741c45_1_38146c39", "d_07741c45_2_3b6f954b", "d_07741c45_3_3d86fd0a", "d_07741c45_4_3f1e9474", "d_07741c45_5_4100d0a0", "d_07741c45_6_4fd1909e"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "b6db488e05c5052d4e7663a21bd662b5fa28ff09bf17507600b6e9e6ac73e134", "response_model": "deepseek-v4-flash", "row_id": "07741c45__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 97, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 950, "prompt_tokens": 950, "total_tokens": 1047}}
{"attempts": 1, "capacity": 7, "case_id": "07741c45", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_07741c45", "d_07741c45_0_37d43f65", "d_07741c45_1_38146c39", "d_07741c45_2_3b6f954b", "d_07741c45_3_3d86fd0a", "d_07741c45_4_3f1e9474", "d_07741c45_5_4100d0a0"], "policy": "hybrid-v1", "prediction": "In a shoe rack in my closet.", "question_type": "knowledge-update", "request_fingerprint": "93e0ff82c897e2ffc34bb2020093ec723e973369a76bf8b63268d2fc6bcae475", "response_model": "deepseek-v4-flash", "row_id": "07741c45__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 343, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1009, "prompt_tokens": 1009, "total_tokens": 1352}}
{"attempts": 1, "capacity": 7, "case_id": "08e075c7", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_08e075c7_0_3b6f954b", "d_08e075c7_1_3d86fd0a", "d_08e075c7_2_3f1e9474", "d_08e075c7_3_4100d0a0", "d_08e075c7_4_4fd1909e", "d_08e075c7_5_505af2f5", "d_08e075c7_6_51a45a95"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "3eb621436d6fc658d9c9dac4cdd580ce492a5ad6a0f3b1886c99fce95d92fa36", "response_model": "deepseek-v4-flash", "row_id": "08e075c7__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 51, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 917, "prompt_tokens": 917, "total_tokens": 968}}
{"attempts": 1, "capacity": 7, "case_id": "08e075c7", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_08e075c7", "d_08e075c7_0_3b6f954b", "d_08e075c7_1_3d86fd0a", "d_08e075c7_2_3f1e9474", "d_08e075c7_3_4100d0a0", "d_08e075c7_4_4fd1909e", "d_08e075c7_5_505af2f5"], "policy": "hybrid-v1", "prediction": "9 months.", "question_type": "knowledge-update", "request_fingerprint": "866e60ded84a7fbf4b7b219edfee207285570e987c7bee20720e76865b1a1614", "response_model": "deepseek-v4-flash", "row_id": "08e075c7__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 92, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1068, "prompt_tokens": 1068, "total_tokens": 1160}}
{"attempts": 1, "capacity": 7, "case_id": "08f4fc43", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_08f4fc43_0_1faac195", "d_08f4fc43_1_21436231", "d_08f4fc43_2_25e5aa4f", "d_08f4fc43_3_29f2956b", "d_08f4fc43_4_311778f1", "d_08f4fc43_5_32260d93", "d_08f4fc43_6_35a27287"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "5ff0364c74643789dcc2ca16d7b34df759b8fdd3b55731f626067321ff78e6f7", "response_model": "deepseek-v4-flash", "row_id": "08f4fc43__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 24, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 923, "prompt_tokens": 923, "total_tokens": 947}}
{"attempts": 1, "capacity": 7, "case_id": "08f4fc43", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_08f4fc43", "d_08f4fc43_0_1faac195", "d_08f4fc43_1_21436231", "d_08f4fc43_2_25e5aa4f", "d_08f4fc43_3_29f2956b", "d_08f4fc43_4_311778f1", "d_08f4fc43_5_32260d93"], "policy": "hybrid-v1", "prediction": "30", "question_type": "temporal-reasoning", "request_fingerprint": "58ddb053f789f0c3027bccd0981e6b6618c8afad35403dbd147275ed0f649154", "response_model": "deepseek-v4-flash", "row_id": "08f4fc43__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 534, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 963, "prompt_tokens": 963, "total_tokens": 1497}}
{"attempts": 1, "capacity": 7, "case_id": "0977f2af", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0977f2af_0_3f1e9474", "d_0977f2af_1_4100d0a0", "d_0977f2af_2_4fd1909e", "d_0977f2af_3_505af2f5", "d_0977f2af_4_51a45a95", "gold_0977f2af", "d_0977f2af_5_54026fce"], "policy": "confidence", "prediction": "Instant Pot", "question_type": "knowledge-update", "request_fingerprint": "b37e9fc92293fe27da16c9b3535ac0fa6ea309b87ef6999c6a95f9b1110a2753", "response_model": "deepseek-v4-flash", "row_id": "0977f2af__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 85, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 935, "prompt_tokens": 935, "total_tokens": 1020}}
{"attempts": 1, "capacity": 7, "case_id": "0977f2af", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0977f2af_0_3f1e9474", "d_0977f2af_1_4100d0a0", "d_0977f2af_2_4fd1909e", "d_0977f2af_3_505af2f5", "d_0977f2af_4_51a45a95", "gold_0977f2af", "d_0977f2af_5_54026fce"], "policy": "hybrid-v1", "prediction": "Instant Pot", "question_type": "knowledge-update", "request_fingerprint": "b37e9fc92293fe27da16c9b3535ac0fa6ea309b87ef6999c6a95f9b1110a2753", "response_model": "deepseek-v4-flash", "row_id": "0977f2af__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 99, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 935, "prompt_tokens": 935, "total_tokens": 1034}}
{"attempts": 1, "capacity": 7, "case_id": "0bb5a684", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0bb5a684_0_3b6f954b", "d_0bb5a684_1_3d86fd0a", "d_0bb5a684_2_3f1e9474", "d_0bb5a684_3_4100d0a0", "d_0bb5a684_4_4fd1909e", "d_0bb5a684_5_505af2f5", "d_0bb5a684_6_51a45a95"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "1ca5a64a3ee01c64e6f3bae704bc6a8bbdba2ade5207942f053b5e40ecb79bab", "response_model": "deepseek-v4-flash", "row_id": "0bb5a684__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 27, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 929, "prompt_tokens": 929, "total_tokens": 956}}
{"attempts": 1, "capacity": 7, "case_id": "0bb5a684", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_0bb5a684", "d_0bb5a684_0_3b6f954b", "d_0bb5a684_1_3d86fd0a", "d_0bb5a684_2_3f1e9474", "d_0bb5a684_3_4100d0a0", "d_0bb5a684_4_4fd1909e", "d_0bb5a684_5_505af2f5"], "policy": "hybrid-v1", "prediction": "7", "question_type": "temporal-reasoning", "request_fingerprint": "0582940f1499f562fd7541094b48ea129b35c45fe9c68a600f4e8911fdcc80ad", "response_model": "deepseek-v4-flash", "row_id": "0bb5a684__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 109, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1057, "prompt_tokens": 1057, "total_tokens": 1166}}
{"attempts": 1, "capacity": 7, "case_id": "0bc8ad92", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0bc8ad92_0_57f827a0", "d_0bc8ad92_1_58bf7951", "d_0bc8ad92_2_58ef2f1c", "d_0bc8ad92_3_5d3d2817", "d_0bc8ad92_4_001be529", "d_0bc8ad92_5_06f04340", "d_0bc8ad92_6_07b6f563"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "b3579d8971360bcfb8130572bf844ef6a1b3a1de443ab22696361f7f7328c8f6", "response_model": "deepseek-v4-flash", "row_id": "0bc8ad92__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 57, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 852, "prompt_tokens": 852, "total_tokens": 909}}
{"attempts": 1, "capacity": 7, "case_id": "0bc8ad92", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_0bc8ad92", "d_0bc8ad92_0_57f827a0", "d_0bc8ad92_1_58bf7951", "d_0bc8ad92_2_58ef2f1c", "d_0bc8ad92_3_5d3d2817", "d_0bc8ad92_4_001be529", "d_0bc8ad92_5_06f04340"], "policy": "hybrid-v1", "prediction": "5", "question_type": "temporal-reasoning", "request_fingerprint": "3e8ff1d7f87625306abab36605f40033bd9764155d3e75b15594b6be09d20dc5", "response_model": "deepseek-v4-flash", "row_id": "0bc8ad92__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 181, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1071, "prompt_tokens": 1071, "total_tokens": 1252}}
{"attempts": 1, "capacity": 7, "case_id": "0db4c65d", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0db4c65d_0_311778f1", "d_0db4c65d_1_32260d93", "d_0db4c65d_2_35a27287", "d_0db4c65d_3_36580ce8", "d_0db4c65d_4_37d43f65", "d_0db4c65d_5_38146c39", "d_0db4c65d_6_3b6f954b"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "a028a9ba342a8a151f187912d507ab0f97f4129c0728731d06b634a897e542a9", "response_model": "deepseek-v4-flash", "row_id": "0db4c65d__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 33, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 980, "prompt_tokens": 980, "total_tokens": 1013}}
{"attempts": 1, "capacity": 7, "case_id": "0db4c65d", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_0db4c65d", "d_0db4c65d_0_311778f1", "d_0db4c65d_1_32260d93", "d_0db4c65d_2_35a27287", "d_0db4c65d_3_36580ce8", "d_0db4c65d_4_37d43f65", "d_0db4c65d_5_38146c39"], "policy": "hybrid-v1", "prediction": "18", "question_type": "temporal-reasoning", "request_fingerprint": "401ead2f3298686e62e5f5dcf763c2d5e960a0dd26b12dbf65f7f30442a20bd2", "response_model": "deepseek-v4-flash", "row_id": "0db4c65d__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 255, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1027, "prompt_tokens": 1027, "total_tokens": 1282}}
{"attempts": 1, "capacity": 7, "case_id": "0ddfec37", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0ddfec37_0_29f2956b", "d_0ddfec37_1_311778f1", "d_0ddfec37_2_32260d93", "d_0ddfec37_3_35a27287", "d_0ddfec37_4_36580ce8", "gold_0ddfec37", "d_0ddfec37_5_37d43f65"], "policy": "confidence", "prediction": "15", "question_type": "knowledge-update", "request_fingerprint": "e98be5ef2cfa6574f83cdde648aac879d8c9f0ba498c16cf41560087cce79587", "response_model": "deepseek-v4-flash", "row_id": "0ddfec37__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 97, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1043, "prompt_tokens": 1043, "total_tokens": 1140}}
{"attempts": 1, "capacity": 7, "case_id": "0ddfec37", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0ddfec37_0_29f2956b", "d_0ddfec37_1_311778f1", "d_0ddfec37_2_32260d93", "d_0ddfec37_3_35a27287", "d_0ddfec37_4_36580ce8", "gold_0ddfec37", "d_0ddfec37_5_37d43f65"], "policy": "hybrid-v1", "prediction": "15", "question_type": "knowledge-update", "request_fingerprint": "e98be5ef2cfa6574f83cdde648aac879d8c9f0ba498c16cf41560087cce79587", "response_model": "deepseek-v4-flash", "row_id": "0ddfec37__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 70, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1043, "prompt_tokens": 1043, "total_tokens": 1113}}
{"attempts": 1, "capacity": 7, "case_id": "0e4e4c46", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0e4e4c46_0_51a45a95", "d_0e4e4c46_1_54026fce", "d_0e4e4c46_2_545bd2b5", "d_0e4e4c46_3_577d4d32", "d_0e4e4c46_4_57f827a0", "gold_0e4e4c46", "d_0e4e4c46_5_58bf7951"], "policy": "confidence", "prediction": "132 points", "question_type": "knowledge-update", "request_fingerprint": "a1a7a6f145325a33fd529d09e232fe50368fea2355449bb5e7cd284f8f318e90", "response_model": "deepseek-v4-flash", "row_id": "0e4e4c46__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 120, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 933, "prompt_tokens": 933, "total_tokens": 1053}}
{"attempts": 1, "capacity": 7, "case_id": "0e4e4c46", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_0e4e4c46_0_51a45a95", "d_0e4e4c46_1_54026fce", "d_0e4e4c46_2_545bd2b5", "d_0e4e4c46_3_577d4d32", "d_0e4e4c46_4_57f827a0", "gold_0e4e4c46", "d_0e4e4c46_5_58bf7951"], "policy": "hybrid-v1", "prediction": "132", "question_type": "knowledge-update", "request_fingerprint": "a1a7a6f145325a33fd529d09e232fe50368fea2355449bb5e7cd284f8f318e90", "response_model": "deepseek-v4-flash", "row_id": "0e4e4c46__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 70, "prompt_cache_hit_tokens": 896, "prompt_cache_miss_tokens": 37, "prompt_tokens": 933, "total_tokens": 1003}}
{"attempts": 1, "capacity": 7, "case_id": "10e09553", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_10e09553_0_1a1907b4", "d_10e09553_1_1c0ddc50", "d_10e09553_2_1d4e3b97", "d_10e09553_3_1da05512", "d_10e09553_4_1e043500", "d_10e09553_5_1faac195", "d_10e09553_6_21436231"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "e0c81f9d410dce7d05eec2f14a9ba734139654341582110345b00e3cab562747", "response_model": "deepseek-v4-flash", "row_id": "10e09553__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 78, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 944, "prompt_tokens": 944, "total_tokens": 1022}}
{"attempts": 1, "capacity": 7, "case_id": "10e09553", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_10e09553_0_1a1907b4", "d_10e09553_1_1c0ddc50", "d_10e09553_2_1d4e3b97", "d_10e09553_3_1da05512", "d_10e09553_4_1e043500", "gold_10e09553", "d_10e09553_5_1faac195"], "policy": "hybrid-v1", "prediction": "7", "question_type": "knowledge-update", "request_fingerprint": "fd7132a93d06eb1d0a29157090b74f2b01479cdf65c4276f4c73aeb2f9580d46", "response_model": "deepseek-v4-flash", "row_id": "10e09553__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 99, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1031, "prompt_tokens": 1031, "total_tokens": 1130}}
{"attempts": 1, "capacity": 7, "case_id": "184da446", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_184da446_0_505af2f5", "d_184da446_1_51a45a95", "d_184da446_2_54026fce", "d_184da446_3_545bd2b5", "d_184da446_4_577d4d32", "d_184da446_5_57f827a0", "d_184da446_6_58bf7951"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "4232829986d192d2aff07835b459a430156b85f1d0e6e07756af184bd247f256", "response_model": "deepseek-v4-flash", "row_id": "184da446__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 22, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 790, "prompt_tokens": 790, "total_tokens": 812}}
{"attempts": 1, "capacity": 7, "case_id": "184da446", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_184da446_0_505af2f5", "d_184da446_1_51a45a95", "d_184da446_2_54026fce", "d_184da446_3_545bd2b5", "d_184da446_4_577d4d32", "gold_184da446", "d_184da446_5_57f827a0"], "policy": "hybrid-v1", "prediction": "220", "question_type": "knowledge-update", "request_fingerprint": "5e1e074cf7a13fd69f489e523cfd410baebf05f61c42b833a300d572a8f744e8", "response_model": "deepseek-v4-flash", "row_id": "184da446__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 149, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 904, "prompt_tokens": 904, "total_tokens": 1053}}
{"attempts": 1, "capacity": 7, "case_id": "18bc8abd", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_18bc8abd", "d_18bc8abd_0_36580ce8", "d_18bc8abd_1_37d43f65", "d_18bc8abd_2_38146c39", "d_18bc8abd_3_3b6f954b", "d_18bc8abd_4_3d86fd0a", "d_18bc8abd_5_3f1e9474"], "policy": "confidence", "prediction": "Kansas City Masterpiece", "question_type": "knowledge-update", "request_fingerprint": "9d3a11e10c1518d11db8c4620b634516c4d530f6b23e5746f050ac1a0f819712", "response_model": "deepseek-v4-flash", "row_id": "18bc8abd__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 54, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 996, "prompt_tokens": 996, "total_tokens": 1050}}
{"attempts": 1, "capacity": 7, "case_id": "18bc8abd", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_18bc8abd", "d_18bc8abd_5_3f1e9474", "d_18bc8abd_7_4fd1909e", "d_18bc8abd_0_36580ce8", "d_18bc8abd_1_37d43f65", "d_18bc8abd_2_38146c39", "d_18bc8abd_3_3b6f954b"], "policy": "hybrid-v1", "prediction": "Kansas City Masterpiece", "question_type": "knowledge-update", "request_fingerprint": "c14f2cd4e91d3cb91166661bf259fda66279b2f7695b3c88f04f382f1a5b0df5", "response_model": "deepseek-v4-flash", "row_id": "18bc8abd__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 88, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1055, "prompt_tokens": 1055, "total_tokens": 1143}}
{"attempts": 1, "capacity": 7, "case_id": "1cea1afa", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_1cea1afa", "d_1cea1afa_0_32260d93", "d_1cea1afa_1_35a27287", "d_1cea1afa_2_36580ce8", "d_1cea1afa_3_37d43f65", "d_1cea1afa_4_38146c39", "d_1cea1afa_5_3b6f954b"], "policy": "confidence", "prediction": "600", "question_type": "knowledge-update", "request_fingerprint": "cc9a3413cc717ea79cf6897b6237c7bea2378cef50eb08159260efc3c70d7e55", "response_model": "deepseek-v4-flash", "row_id": "1cea1afa__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 129, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1007, "prompt_tokens": 1007, "total_tokens": 1136}}
{"attempts": 1, "capacity": 7, "case_id": "1cea1afa", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_1cea1afa", "d_1cea1afa_5_3b6f954b", "d_1cea1afa_7_3f1e9474", "d_1cea1afa_0_32260d93", "d_1cea1afa_1_35a27287", "d_1cea1afa_2_36580ce8", "d_1cea1afa_3_37d43f65"], "policy": "hybrid-v1", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "e1ed17bcf7021f9c45754b43a2ca959382525d9b7b5a318ed79072fb404e8774", "response_model": "deepseek-v4-flash", "row_id": "1cea1afa__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 59, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1061, "prompt_tokens": 1061, "total_tokens": 1120}}
{"attempts": 1, "capacity": 7, "case_id": "2133c1b5", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_2133c1b5", "d_2133c1b5_0_3f1e9474", "d_2133c1b5_1_4100d0a0", "d_2133c1b5_2_4fd1909e", "d_2133c1b5_3_505af2f5", "d_2133c1b5_4_51a45a95", "d_2133c1b5_5_54026fce"], "policy": "confidence", "prediction": "3 months", "question_type": "knowledge-update", "request_fingerprint": "64519995b69d3233d6cfb3452b68516b40dbaaafaa969a406c987e99445b3a75", "response_model": "deepseek-v4-flash", "row_id": "2133c1b5__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 121, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 987, "prompt_tokens": 987, "total_tokens": 1108}}
{"attempts": 1, "capacity": 7, "case_id": "2133c1b5", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_2133c1b5", "d_2133c1b5_5_54026fce", "d_2133c1b5_7_577d4d32", "d_2133c1b5_0_3f1e9474", "d_2133c1b5_1_4100d0a0", "d_2133c1b5_2_4fd1909e", "d_2133c1b5_3_505af2f5"], "policy": "hybrid-v1", "prediction": "3 months.", "question_type": "knowledge-update", "request_fingerprint": "482526464cceca7c8d67be09cc2ef33cc11cccc9c536561845b30e949c5e1fe8", "response_model": "deepseek-v4-flash", "row_id": "2133c1b5__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 251, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1048, "prompt_tokens": 1048, "total_tokens": 1299}}
{"attempts": 1, "capacity": 7, "case_id": "22d2cb42", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_22d2cb42", "d_22d2cb42_0_3f1e9474", "d_22d2cb42_1_4100d0a0", "d_22d2cb42_2_4fd1909e", "d_22d2cb42_3_505af2f5", "d_22d2cb42_4_51a45a95", "d_22d2cb42_5_54026fce"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "2b2ec02904c4f3e332371f1f7027031e580db153daafcd39ae6196b2973fd0b0", "response_model": "deepseek-v4-flash", "row_id": "22d2cb42__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 100, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1231, "prompt_tokens": 1231, "total_tokens": 1331}}
{"attempts": 1, "capacity": 7, "case_id": "22d2cb42", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_22d2cb42", "d_22d2cb42_5_54026fce", "d_22d2cb42_7_577d4d32", "d_22d2cb42_0_3f1e9474", "d_22d2cb42_1_4100d0a0", "d_22d2cb42_2_4fd1909e", "d_22d2cb42_3_505af2f5"], "policy": "hybrid-v1", "prediction": "INSUFFICIENT", "question_type": "knowledge-update", "request_fingerprint": "5c0b31ea2ea883f59c4027883c6cd90107561caf02e89b7117b29e5b0ee9164d", "response_model": "deepseek-v4-flash", "row_id": "22d2cb42__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 142, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1292, "prompt_tokens": 1292, "total_tokens": 1434}}
{"attempts": 1, "capacity": 7, "case_id": "2698e78f", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_2698e78f", "d_2698e78f_0_5d3d2817", "d_2698e78f_1_001be529", "d_2698e78f_2_06f04340", "d_2698e78f_3_07b6f563", "d_2698e78f_4_0862e8bf", "d_2698e78f_5_09d032c9"], "policy": "confidence", "prediction": "Every week.", "question_type": "knowledge-update", "request_fingerprint": "708b353825a39319f1312f3e77886cd063ffab222b395d168b280367cdf76608", "response_model": "deepseek-v4-flash", "row_id": "2698e78f__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 43, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 953, "prompt_tokens": 953, "total_tokens": 996}}
{"attempts": 1, "capacity": 7, "case_id": "2698e78f", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_2698e78f", "d_2698e78f_5_09d032c9", "d_2698e78f_7_0edc2aef", "d_2698e78f_0_5d3d2817", "d_2698e78f_1_001be529", "d_2698e78f_2_06f04340", "d_2698e78f_3_07b6f563"], "policy": "hybrid-v1", "prediction": "every week", "question_type": "knowledge-update", "request_fingerprint": "58aeedab9af276503c47d3ef37226a790781e8efd427dfdb98f49a7af6d45ae2", "response_model": "deepseek-v4-flash", "row_id": "2698e78f__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 155, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 944, "prompt_tokens": 944, "total_tokens": 1099}}
{"attempts": 1, "capacity": 7, "case_id": "26bdc477", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_26bdc477", "d_26bdc477_0_4fd1909e", "d_26bdc477_1_505af2f5", "d_26bdc477_2_51a45a95", "d_26bdc477_3_54026fce", "d_26bdc477_4_545bd2b5", "d_26bdc477_5_577d4d32"], "policy": "confidence", "prediction": "5", "question_type": "knowledge-update", "request_fingerprint": "61f5df0f9ded1e832c68affb80564973ecd533d46b8e0b3878c5213e2749f92a", "response_model": "deepseek-v4-flash", "row_id": "26bdc477__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 92, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 975, "prompt_tokens": 975, "total_tokens": 1067}}
{"attempts": 1, "capacity": 7, "case_id": "26bdc477", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_26bdc477_1_505af2f5", "d_26bdc477_3_54026fce", "gold_26bdc477", "d_26bdc477_0_4fd1909e", "d_26bdc477_2_51a45a95", "d_26bdc477_4_545bd2b5", "d_26bdc477_5_577d4d32"], "policy": "hybrid-v1", "prediction": "5", "question_type": "knowledge-update", "request_fingerprint": "61f5df0f9ded1e832c68affb80564973ecd533d46b8e0b3878c5213e2749f92a", "response_model": "deepseek-v4-flash", "row_id": "26bdc477__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 122, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 975, "prompt_tokens": 975, "total_tokens": 1097}}
{"attempts": 1, "capacity": 7, "case_id": "2a1811e2", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2a1811e2_0_1d4e3b97", "d_2a1811e2_1_1da05512", "d_2a1811e2_2_1e043500", "d_2a1811e2_3_1faac195", "d_2a1811e2_4_21436231", "d_2a1811e2_5_25e5aa4f", "d_2a1811e2_6_29f2956b"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "8d1af72bddde9ae2d1899b3878f0899580835d02f5e656549e6eaedc85836f0a", "response_model": "deepseek-v4-flash", "row_id": "2a1811e2__confidence", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 46, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 948, "prompt_tokens": 948, "total_tokens": 994}}
{"attempts": 1, "capacity": 7, "case_id": "2a1811e2", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_2a1811e2", "d_2a1811e2_0_1d4e3b97", "d_2a1811e2_1_1da05512", "d_2a1811e2_2_1e043500", "d_2a1811e2_3_1faac195", "d_2a1811e2_4_21436231", "d_2a1811e2_5_25e5aa4f"], "policy": "hybrid-v1", "prediction": "21", "question_type": "temporal-reasoning", "request_fingerprint": "c760b478dd68e3a26ed12e6f916539a78d420f37836e1f8de3c6f00023a80da7", "response_model": "deepseek-v4-flash", "row_id": "2a1811e2__hybrid-v1", "scenario": "confirmation_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 259, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1046, "prompt_tokens": 1046, "total_tokens": 1305}}
{"attempts": 1, "capacity": 7, "case_id": "2c63a862", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2c63a862_0_4fd1909e", "d_2c63a862_1_505af2f5", "d_2c63a862_2_51a45a95", "d_2c63a862_3_54026fce", "d_2c63a862_4_545bd2b5", "d_2c63a862_5_577d4d32", "d_2c63a862_6_57f827a0"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "02080e021aad29164681ab0230977f3cecb13458c4b986a83d7b1d6e13b7ce9d", "response_model": "deepseek-v4-flash", "row_id": "2c63a862__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 75, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 861, "prompt_tokens": 861, "total_tokens": 936}}
{"attempts": 1, "capacity": 7, "case_id": "2c63a862", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2c63a862_0_4fd1909e", "d_2c63a862_1_505af2f5", "d_2c63a862_2_51a45a95", "d_2c63a862_3_54026fce", "d_2c63a862_4_545bd2b5", "gold_2c63a862", "d_2c63a862_5_577d4d32"], "policy": "hybrid-v1", "prediction": "14 days.", "question_type": "temporal-reasoning", "request_fingerprint": "48df4fd951dc466fa94277f53b91cf091b7bf7cac66b6b249ba84b55dfa825df", "response_model": "deepseek-v4-flash", "row_id": "2c63a862__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 144, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 958, "prompt_tokens": 958, "total_tokens": 1102}}
{"attempts": 1, "capacity": 7, "case_id": "2ebe6c90", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2ebe6c90_0_3f1e9474", "d_2ebe6c90_1_4100d0a0", "d_2ebe6c90_2_4fd1909e", "d_2ebe6c90_3_505af2f5", "d_2ebe6c90_4_51a45a95", "d_2ebe6c90_5_54026fce", "d_2ebe6c90_6_545bd2b5"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "e7f7350fec2933fc5a7af210ab606175e20eb1305b9a050ffe694685ef6a3b7a", "response_model": "deepseek-v4-flash", "row_id": "2ebe6c90__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 44, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 889, "prompt_tokens": 889, "total_tokens": 933}}
{"attempts": 1, "capacity": 7, "case_id": "2ebe6c90", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2ebe6c90_0_3f1e9474", "d_2ebe6c90_1_4100d0a0", "d_2ebe6c90_2_4fd1909e", "d_2ebe6c90_3_505af2f5", "d_2ebe6c90_4_51a45a95", "gold_2ebe6c90", "d_2ebe6c90_5_54026fce"], "policy": "hybrid-v1", "prediction": "21", "question_type": "temporal-reasoning", "request_fingerprint": "fc452048eca185f90922f80a178f8119b1b90fbb19f1ac50721fcf22174628f4", "response_model": "deepseek-v4-flash", "row_id": "2ebe6c90__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 190, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 959, "prompt_tokens": 959, "total_tokens": 1149}}
{"attempts": 1, "capacity": 7, "case_id": "2ebe6c92", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2ebe6c92_0_57f827a0", "d_2ebe6c92_1_58bf7951", "d_2ebe6c92_2_58ef2f1c", "d_2ebe6c92_3_5d3d2817", "d_2ebe6c92_4_001be529", "d_2ebe6c92_5_06f04340", "d_2ebe6c92_6_07b6f563"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "33cd01df533e69ade4d675807ce9301cf1719332e239f66f00c2d42d2d345e80", "response_model": "deepseek-v4-flash", "row_id": "2ebe6c92__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 28, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 846, "prompt_tokens": 846, "total_tokens": 874}}
{"attempts": 1, "capacity": 7, "case_id": "2ebe6c92", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_2ebe6c92_0_57f827a0", "d_2ebe6c92_1_58bf7951", "d_2ebe6c92_2_58ef2f1c", "d_2ebe6c92_3_5d3d2817", "d_2ebe6c92_4_001be529", "gold_2ebe6c92", "d_2ebe6c92_5_06f04340"], "policy": "hybrid-v1", "prediction": "The Nightingale by Kristin Hannah", "question_type": "temporal-reasoning", "request_fingerprint": "ea5f03ddd295af803f796b888098343d2a04ed6b65313ea1cd625ead85d7416d", "response_model": "deepseek-v4-flash", "row_id": "2ebe6c92__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 173, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 902, "prompt_tokens": 902, "total_tokens": 1075}}
{"attempts": 1, "capacity": 7, "case_id": "370a8ff4", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_370a8ff4_0_25e5aa4f", "d_370a8ff4_1_29f2956b", "d_370a8ff4_2_311778f1", "d_370a8ff4_3_32260d93", "d_370a8ff4_4_35a27287", "d_370a8ff4_5_36580ce8", "d_370a8ff4_6_37d43f65"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "c7b0f3e0714b21406edf56f064752851bfd4189d1e2a8ccfa79a02cac11a643a", "response_model": "deepseek-v4-flash", "row_id": "370a8ff4__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 26, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 957, "prompt_tokens": 957, "total_tokens": 983}}
{"attempts": 1, "capacity": 7, "case_id": "370a8ff4", "grade_correct": false, "grade_rule": "numeric-conflict", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_370a8ff4_0_25e5aa4f", "d_370a8ff4_1_29f2956b", "d_370a8ff4_2_311778f1", "d_370a8ff4_3_32260d93", "d_370a8ff4_4_35a27287", "gold_370a8ff4", "d_370a8ff4_5_36580ce8"], "policy": "hybrid-v1", "prediction": "11 weeks and 4 days.", "question_type": "temporal-reasoning", "request_fingerprint": "6424bb99203fdc08460a6b4e629483e325bc2d839ff57de4c21fecb6ab5be25d", "response_model": "deepseek-v4-flash", "row_id": "370a8ff4__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 312, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1007, "prompt_tokens": 1007, "total_tokens": 1319}}
{"attempts": 1, "capacity": 7, "case_id": "3ba21379", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_3ba21379", "d_3ba21379_0_36580ce8", "d_3ba21379_1_37d43f65", "d_3ba21379_2_38146c39", "d_3ba21379_3_3b6f954b", "d_3ba21379_4_3d86fd0a", "d_3ba21379_5_3f1e9474"], "policy": "confidence", "prediction": "Ford F-150 pickup truck", "question_type": "knowledge-update", "request_fingerprint": "a07361f80fb2e1a348df2e474ed6e144a82b6713bc21c992c2ae16fac5b48a27", "response_model": "deepseek-v4-flash", "row_id": "3ba21379__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 175, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 948, "prompt_tokens": 948, "total_tokens": 1123}}
{"attempts": 1, "capacity": 7, "case_id": "3ba21379", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_3ba21379_1_37d43f65", "d_3ba21379_3_3b6f954b", "gold_3ba21379", "d_3ba21379_0_36580ce8", "d_3ba21379_2_38146c39", "d_3ba21379_4_3d86fd0a", "d_3ba21379_5_3f1e9474"], "policy": "hybrid-v1", "prediction": "Ford F-150 pickup truck.", "question_type": "knowledge-update", "request_fingerprint": "a07361f80fb2e1a348df2e474ed6e144a82b6713bc21c992c2ae16fac5b48a27", "response_model": "deepseek-v4-flash", "row_id": "3ba21379__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 174, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 948, "prompt_tokens": 948, "total_tokens": 1122}}
{"attempts": 1, "capacity": 7, "case_id": "41698283", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_41698283", "d_41698283_0_51a45a95", "d_41698283_1_54026fce", "d_41698283_2_545bd2b5", "d_41698283_3_577d4d32", "d_41698283_4_57f827a0", "d_41698283_5_58bf7951"], "policy": "confidence", "prediction": "70-200mm zoom lens", "question_type": "knowledge-update", "request_fingerprint": "0ce36f5a5503a35b0a9f1a6f9fb2c9be3d3f98b55426f10ba6b05471dbc44ad4", "response_model": "deepseek-v4-flash", "row_id": "41698283__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 117, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 921, "prompt_tokens": 921, "total_tokens": 1038}}
{"attempts": 1, "capacity": 7, "case_id": "41698283", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_41698283_1_54026fce", "d_41698283_3_577d4d32", "gold_41698283", "d_41698283_0_51a45a95", "d_41698283_2_545bd2b5", "d_41698283_4_57f827a0", "d_41698283_5_58bf7951"], "policy": "hybrid-v1", "prediction": "70-200mm zoom lens.", "question_type": "knowledge-update", "request_fingerprint": "0ce36f5a5503a35b0a9f1a6f9fb2c9be3d3f98b55426f10ba6b05471dbc44ad4", "response_model": "deepseek-v4-flash", "row_id": "41698283__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 119, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 921, "prompt_tokens": 921, "total_tokens": 1040}}
{"attempts": 1, "capacity": 7, "case_id": "42ec0761", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_42ec0761", "d_42ec0761_0_1a1907b4", "d_42ec0761_1_1c0ddc50", "d_42ec0761_2_1d4e3b97", "d_42ec0761_3_1da05512", "d_42ec0761_4_1e043500", "d_42ec0761_5_1faac195"], "policy": "confidence", "prediction": "YES", "question_type": "knowledge-update", "request_fingerprint": "ba076a5030afaedec604b53c204cda4a37a58b0bb03948afcc3c68292f2018ff", "response_model": "deepseek-v4-flash", "row_id": "42ec0761__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 75, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1000, "prompt_tokens": 1000, "total_tokens": 1075}}
{"attempts": 1, "capacity": 7, "case_id": "42ec0761", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_42ec0761_1_1c0ddc50", "d_42ec0761_3_1da05512", "d_42ec0761_5_1faac195", "gold_42ec0761", "d_42ec0761_0_1a1907b4", "d_42ec0761_2_1d4e3b97", "d_42ec0761_4_1e043500"], "policy": "hybrid-v1", "prediction": "YES", "question_type": "knowledge-update", "request_fingerprint": "ba076a5030afaedec604b53c204cda4a37a58b0bb03948afcc3c68292f2018ff", "response_model": "deepseek-v4-flash", "row_id": "42ec0761__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 81, "prompt_cache_hit_tokens": 896, "prompt_cache_miss_tokens": 104, "prompt_tokens": 1000, "total_tokens": 1081}}
{"attempts": 1, "capacity": 7, "case_id": "45dc21b6", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_45dc21b6", "d_45dc21b6_0_505af2f5", "d_45dc21b6_1_51a45a95", "d_45dc21b6_2_54026fce", "d_45dc21b6_3_545bd2b5", "d_45dc21b6_4_577d4d32", "d_45dc21b6_5_57f827a0"], "policy": "confidence", "prediction": "3", "question_type": "knowledge-update", "request_fingerprint": "14a22b004876dc226d0f47bd8cba1abb30d16dc4e353d30085c9930eb4c6e96d", "response_model": "deepseek-v4-flash", "row_id": "45dc21b6__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 76, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 899, "prompt_tokens": 899, "total_tokens": 975}}
{"attempts": 1, "capacity": 7, "case_id": "45dc21b6", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_45dc21b6_1_51a45a95", "d_45dc21b6_3_545bd2b5", "d_45dc21b6_5_57f827a0", "gold_45dc21b6", "d_45dc21b6_0_505af2f5", "d_45dc21b6_2_54026fce", "d_45dc21b6_4_577d4d32"], "policy": "hybrid-v1", "prediction": "3", "question_type": "knowledge-update", "request_fingerprint": "14a22b004876dc226d0f47bd8cba1abb30d16dc4e353d30085c9930eb4c6e96d", "response_model": "deepseek-v4-flash", "row_id": "45dc21b6__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 369, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 899, "prompt_tokens": 899, "total_tokens": 1268}}
{"attempts": 1, "capacity": 7, "case_id": "4dfccbf7", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_4dfccbf7_0_38146c39", "d_4dfccbf7_1_3b6f954b", "d_4dfccbf7_2_3d86fd0a", "d_4dfccbf7_3_3f1e9474", "d_4dfccbf7_4_4100d0a0", "d_4dfccbf7_5_4fd1909e", "d_4dfccbf7_6_505af2f5"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "fa2c8b305196e3c358d546396fb3f919709a41362cb68a968bf55978d02b4ef8", "response_model": "deepseek-v4-flash", "row_id": "4dfccbf7__confidence", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 65, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 941, "prompt_tokens": 941, "total_tokens": 1006}}
{"attempts": 1, "capacity": 7, "case_id": "4dfccbf7", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_4dfccbf7_0_38146c39", "d_4dfccbf7_1_3b6f954b", "d_4dfccbf7_2_3d86fd0a", "d_4dfccbf7_3_3f1e9474", "d_4dfccbf7_4_4100d0a0", "gold_4dfccbf7", "d_4dfccbf7_5_4fd1909e"], "policy": "hybrid-v1", "prediction": "24 days.", "question_type": "temporal-reasoning", "request_fingerprint": "f2c693ad33f5c985368dfe651d62580febc0fb8aa2bd184a0145df846226eb4f", "response_model": "deepseek-v4-flash", "row_id": "4dfccbf7__hybrid-v1", "scenario": "access_help", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 298, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1092, "prompt_tokens": 1092, "total_tokens": 1390}}
{"attempts": 1, "capacity": 7, "case_id": "5e1b23de", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_5e1b23de", "d_5e1b23de_0_5d3d2817", "d_5e1b23de_1_001be529", "d_5e1b23de_2_06f04340", "d_5e1b23de_3_07b6f563", "d_5e1b23de_4_0862e8bf", "d_5e1b23de_5_09d032c9"], "policy": "confidence", "prediction": "3 months ago", "question_type": "temporal-reasoning", "request_fingerprint": "ced953a9a02fa038320dd85df0ffbff1a010690923cb9c510de15e6ccd8af6b8", "response_model": "deepseek-v4-flash", "row_id": "5e1b23de__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 91, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 864, "prompt_tokens": 864, "total_tokens": 955}}
{"attempts": 1, "capacity": 7, "case_id": "5e1b23de", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_5e1b23de", "d_5e1b23de_5_09d032c9", "d_5e1b23de_7_0edc2aef", "d_5e1b23de_0_5d3d2817", "d_5e1b23de_1_001be529", "d_5e1b23de_2_06f04340", "d_5e1b23de_3_07b6f563"], "policy": "hybrid-v1", "prediction": "3", "question_type": "temporal-reasoning", "request_fingerprint": "53ce3c0629aa2f0bf912f38bdb69e16718bd0dbb2263f35b8e1674907b679ff9", "response_model": "deepseek-v4-flash", "row_id": "5e1b23de__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 112, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 855, "prompt_tokens": 855, "total_tokens": 967}}
{"attempts": 1, "capacity": 7, "case_id": "6e984301", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_6e984301", "d_6e984301_0_35a27287", "d_6e984301_1_36580ce8", "d_6e984301_2_37d43f65", "d_6e984301_3_38146c39", "d_6e984301_4_3b6f954b", "d_6e984301_5_3d86fd0a"], "policy": "confidence", "prediction": "3", "question_type": "temporal-reasoning", "request_fingerprint": "9d1daac71a2a71f95d3317729f5cbef1f83d112e9841d86d6894da42f94be52c", "response_model": "deepseek-v4-flash", "row_id": "6e984301__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 356, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1037, "prompt_tokens": 1037, "total_tokens": 1393}}
{"attempts": 1, "capacity": 7, "case_id": "6e984301", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_6e984301", "d_6e984301_5_3d86fd0a", "d_6e984301_7_4100d0a0", "d_6e984301_0_35a27287", "d_6e984301_1_36580ce8", "d_6e984301_2_37d43f65", "d_6e984301_3_38146c39"], "policy": "hybrid-v1", "prediction": "3 weeks", "question_type": "temporal-reasoning", "request_fingerprint": "b4d2e22bc94104afdf4b593cb177ca527aabb88bed245b7b95f2605a09471113", "response_model": "deepseek-v4-flash", "row_id": "6e984301__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 164, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1017, "prompt_tokens": 1017, "total_tokens": 1181}}
{"attempts": 1, "capacity": 7, "case_id": "71017276", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_71017276", "d_71017276_0_07b6f563", "d_71017276_1_0862e8bf", "d_71017276_2_09d032c9", "d_71017276_3_0a34ad58", "d_71017276_4_0edc2aef", "d_71017276_5_118b2229"], "policy": "confidence", "prediction": "4 weeks ago.", "question_type": "temporal-reasoning", "request_fingerprint": "54e094adc43011c4bcff194549262b1cb34d1a950b472678cb71218ac2df3096", "response_model": "deepseek-v4-flash", "row_id": "71017276__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 173, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 848, "prompt_tokens": 848, "total_tokens": 1021}}
{"attempts": 1, "capacity": 7, "case_id": "71017276", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_71017276", "d_71017276_5_118b2229", "d_71017276_7_195a1a1b", "d_71017276_0_07b6f563", "d_71017276_1_0862e8bf", "d_71017276_2_09d032c9", "d_71017276_3_0a34ad58"], "policy": "hybrid-v1", "prediction": "4", "question_type": "temporal-reasoning", "request_fingerprint": "58bbe558057fd5c8992016575ecf137b1c230a708b88c71ddf2c69d9c288d757", "response_model": "deepseek-v4-flash", "row_id": "71017276__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 160, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 844, "prompt_tokens": 844, "total_tokens": 1004}}
{"attempts": 1, "capacity": 7, "case_id": "71017277", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_71017277", "d_71017277_0_3b6f954b", "d_71017277_1_3d86fd0a", "d_71017277_2_3f1e9474", "d_71017277_3_4100d0a0", "d_71017277_4_4fd1909e", "d_71017277_5_505af2f5"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "b8b91ade69edd50770b34329a47d7fc6ddb12d2b785b28a0ca0d340ee6e21c7b", "response_model": "deepseek-v4-flash", "row_id": "71017277__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 69, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 968, "prompt_tokens": 968, "total_tokens": 1037}}
{"attempts": 1, "capacity": 7, "case_id": "71017277", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_71017277", "d_71017277_5_505af2f5", "d_71017277_7_54026fce", "d_71017277_0_3b6f954b", "d_71017277_1_3d86fd0a", "d_71017277_2_3f1e9474", "d_71017277_3_4100d0a0"], "policy": "hybrid-v1", "prediction": "INSUFFICIENT", "question_type": "temporal-reasoning", "request_fingerprint": "8788560ff8387fa1a242ef64f6fd4a7de0830b4d775aac3246eeb3e609de0726", "response_model": "deepseek-v4-flash", "row_id": "71017277__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 94, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 912, "prompt_tokens": 912, "total_tokens": 1006}}
{"attempts": 1, "capacity": 7, "case_id": "8c18457d", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_8c18457d", "d_8c18457d_0_35a27287", "d_8c18457d_1_36580ce8", "d_8c18457d_2_37d43f65", "d_8c18457d_3_38146c39", "d_8c18457d_4_3b6f954b", "d_8c18457d_5_3d86fd0a"], "policy": "confidence", "prediction": "7", "question_type": "temporal-reasoning", "request_fingerprint": "942f7bd0f859e4b65271780d9112f0f42eaab140884164c84315a3100ff4bf80", "response_model": "deepseek-v4-flash", "row_id": "8c18457d__confidence", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 193, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1083, "prompt_tokens": 1083, "total_tokens": 1276}}
{"attempts": 1, "capacity": 7, "case_id": "8c18457d", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_8c18457d", "d_8c18457d_5_3d86fd0a", "d_8c18457d_7_4100d0a0", "d_8c18457d_0_35a27287", "d_8c18457d_1_36580ce8", "d_8c18457d_2_37d43f65", "d_8c18457d_3_38146c39"], "policy": "hybrid-v1", "prediction": "7 days", "question_type": "temporal-reasoning", "request_fingerprint": "b3bd594527ee2c44dec1666ff70db8d909f65281ff328f748bde65cf1487a6a8", "response_model": "deepseek-v4-flash", "row_id": "8c18457d__hybrid-v1", "scenario": "confidence_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 133, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1063, "prompt_tokens": 1063, "total_tokens": 1196}}
{"attempts": 1, "capacity": 7, "case_id": "982b5123", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_982b5123", "d_982b5123_0_29f2956b", "d_982b5123_1_311778f1", "d_982b5123_2_32260d93", "d_982b5123_3_35a27287", "d_982b5123_4_36580ce8", "d_982b5123_5_37d43f65"], "policy": "confidence", "prediction": "5", "question_type": "temporal-reasoning", "request_fingerprint": "2545aafde5db0a288fb60824839324f9c298182def02bfe7e56854e165875306", "response_model": "deepseek-v4-flash", "row_id": "982b5123__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 181, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1013, "prompt_tokens": 1013, "total_tokens": 1194}}
{"attempts": 1, "capacity": 7, "case_id": "982b5123", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_982b5123_1_311778f1", "d_982b5123_3_35a27287", "d_982b5123_5_37d43f65", "gold_982b5123", "d_982b5123_0_29f2956b", "d_982b5123_2_32260d93", "d_982b5123_4_36580ce8"], "policy": "hybrid-v1", "prediction": "5", "question_type": "temporal-reasoning", "request_fingerprint": "2545aafde5db0a288fb60824839324f9c298182def02bfe7e56854e165875306", "response_model": "deepseek-v4-flash", "row_id": "982b5123__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 546, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1013, "prompt_tokens": 1013, "total_tokens": 1559}}
{"attempts": 1, "capacity": 7, "case_id": "993da5e2", "grade_correct": false, "grade_rule": "numeric-conflict", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_993da5e2", "d_993da5e2_0_35a27287", "d_993da5e2_1_36580ce8", "d_993da5e2_2_37d43f65", "d_993da5e2_3_38146c39", "d_993da5e2_4_3b6f954b", "d_993da5e2_5_3d86fd0a"], "policy": "confidence", "prediction": "About one week.", "question_type": "temporal-reasoning", "request_fingerprint": "30a2595db36c0951be904cb7b92c047d3596c4b6dfd62d07ce597059c137df7c", "response_model": "deepseek-v4-flash", "row_id": "993da5e2__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 257, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1038, "prompt_tokens": 1038, "total_tokens": 1295}}
{"attempts": 1, "capacity": 7, "case_id": "993da5e2", "grade_correct": false, "grade_rule": "numeric-conflict", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_993da5e2_1_36580ce8", "d_993da5e2_3_38146c39", "d_993da5e2_5_3d86fd0a", "d_993da5e2_7_4100d0a0", "gold_993da5e2", "d_993da5e2_0_35a27287", "d_993da5e2_2_37d43f65"], "policy": "hybrid-v1", "prediction": "About one week.", "question_type": "temporal-reasoning", "request_fingerprint": "1cdbcdfe739a7f437a21b41010d7c95dd9ad6cdb2bec60abaef435239f7036e2", "response_model": "deepseek-v4-flash", "row_id": "993da5e2__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 397, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1018, "prompt_tokens": 1018, "total_tokens": 1415}}
{"attempts": 1, "capacity": 7, "case_id": "9a707b81", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_9a707b81", "d_9a707b81_0_1d4e3b97", "d_9a707b81_1_1da05512", "d_9a707b81_2_1e043500", "d_9a707b81_3_1faac195", "d_9a707b81_4_21436231", "d_9a707b81_5_25e5aa4f"], "policy": "confidence", "prediction": "21", "question_type": "temporal-reasoning", "request_fingerprint": "3577dc107fff4f2ffa6caaa949d71eca5f6d64178590d07ca2dafb627ce001f5", "response_model": "deepseek-v4-flash", "row_id": "9a707b81__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 240, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1051, "prompt_tokens": 1051, "total_tokens": 1291}}
{"attempts": 1, "capacity": 7, "case_id": "9a707b81", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_9a707b81_1_1da05512", "d_9a707b81_3_1faac195", "d_9a707b81_5_25e5aa4f", "d_9a707b81_7_311778f1", "gold_9a707b81", "d_9a707b81_0_1d4e3b97", "d_9a707b81_2_1e043500"], "policy": "hybrid-v1", "prediction": "21 days", "question_type": "temporal-reasoning", "request_fingerprint": "456a7b42724e44a08a7abdda235c8e47e5c2b2f8b10b55eb72205fdf4e4a3d38", "response_model": "deepseek-v4-flash", "row_id": "9a707b81__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 316, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1041, "prompt_tokens": 1041, "total_tokens": 1357}}
{"attempts": 1, "capacity": 7, "case_id": "a3045048", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_a3045048", "d_a3045048_0_19b5f2b3", "d_a3045048_1_1a1907b4", "d_a3045048_2_1c0ddc50", "d_a3045048_3_1d4e3b97", "d_a3045048_4_1da05512", "d_a3045048_5_1e043500"], "policy": "confidence", "prediction": "7 days", "question_type": "temporal-reasoning", "request_fingerprint": "edabeafc2a0f43b211fe427078db27566c8232e84374f39383430526e0b867cc", "response_model": "deepseek-v4-flash", "row_id": "a3045048__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 58, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1057, "prompt_tokens": 1057, "total_tokens": 1115}}
{"attempts": 1, "capacity": 7, "case_id": "a3045048", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_a3045048_1_1a1907b4", "d_a3045048_3_1d4e3b97", "d_a3045048_5_1e043500", "d_a3045048_7_21436231", "gold_a3045048", "d_a3045048_0_19b5f2b3", "d_a3045048_2_1c0ddc50"], "policy": "hybrid-v1", "prediction": "7 days", "question_type": "temporal-reasoning", "request_fingerprint": "e861b6891607aeb9cac5c6c1b480df07fb63dfa917a949f6dc329c15b9adf379", "response_model": "deepseek-v4-flash", "row_id": "a3045048__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 64, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1044, "prompt_tokens": 1044, "total_tokens": 1108}}
{"attempts": 1, "capacity": 7, "case_id": "af082822", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["gold_af082822", "d_af082822_0_36580ce8", "d_af082822_1_37d43f65", "d_af082822_2_38146c39", "d_af082822_3_3b6f954b", "d_af082822_4_3d86fd0a", "d_af082822_5_3f1e9474"], "policy": "confidence", "prediction": "2 weeks ago", "question_type": "temporal-reasoning", "request_fingerprint": "9c0002f3e4b625bba5e3eb2781cdab209dab0463e7fcce45ba23aa8322cae6c5", "response_model": "deepseek-v4-flash", "row_id": "af082822__confidence", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 126, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 952, "prompt_tokens": 952, "total_tokens": 1078}}
{"attempts": 1, "capacity": 7, "case_id": "af082822", "grade_correct": true, "grade_rule": "substring", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_af082822_1_37d43f65", "d_af082822_3_3b6f954b", "d_af082822_5_3f1e9474", "d_af082822_7_4fd1909e", "gold_af082822", "d_af082822_0_36580ce8", "d_af082822_2_38146c39"], "policy": "hybrid-v1", "prediction": "2 weeks ago.", "question_type": "temporal-reasoning", "request_fingerprint": "58029895f9ce33e293e543207610f9040c9ec180267bb295f7870236543633e7", "response_model": "deepseek-v4-flash", "row_id": "af082822__hybrid-v1", "scenario": "noisy_signal_control", "schema_version": 1, "source": "longmemeval", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 338, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 1011, "prompt_tokens": 1011, "total_tokens": 1349}}
{"attempts": 1, "capacity": 7, "case_id": "correction_language", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_language_0_195a1a1b", "d_correction_language_1_19b5f2b3", "d_correction_language_2_1a1907b4", "d_correction_language_3_1c0ddc50", "d_correction_language_4_1d4e3b97", "d_correction_language_5_1da05512", "d_correction_language_6_1e043500"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "synthetic-correction", "request_fingerprint": "a12285e002c878cda2d667d0fba7d68ddc4c2e269db0e38674eaf66e213c9ebc", "response_model": "deepseek-v4-flash", "row_id": "correction_language__confidence", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 105, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 900, "prompt_tokens": 900, "total_tokens": 1005}}
{"attempts": 1, "capacity": 7, "case_id": "correction_language", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_language_0_195a1a1b", "d_correction_language_1_19b5f2b3", "d_correction_language_2_1a1907b4", "d_correction_language_3_1c0ddc50", "d_correction_language_4_1d4e3b97", "d_correction_language_5_1da05512", "gold_correction_language"], "policy": "hybrid-v1", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "9e0af54dbccd04ec66840afd9d5e1e62d52fb30f9cad85f0d1aca50355724faf", "response_model": "deepseek-v4-flash", "row_id": "correction_language__hybrid-v1", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 71, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 816, "prompt_tokens": 816, "total_tokens": 887}}
{"attempts": 1, "capacity": 7, "case_id": "correction_peanut_allergy", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_peanut_allergy_0_0a34ad58", "d_correction_peanut_allergy_1_0edc2aef", "d_correction_peanut_allergy_2_118b2229", "d_correction_peanut_allergy_3_15745da0", "d_correction_peanut_allergy_4_195a1a1b", "gold_correction_peanut_allergy", "d_correction_peanut_allergy_5_19b5f2b3"], "policy": "confidence", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "5c250ac833307e3593c15813f85b25992761e49e25839b63230f40d88aae38eb", "response_model": "deepseek-v4-flash", "row_id": "correction_peanut_allergy__confidence", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 60, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 726, "prompt_tokens": 726, "total_tokens": 786}}
{"attempts": 1, "capacity": 7, "case_id": "correction_peanut_allergy", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_peanut_allergy_0_0a34ad58", "d_correction_peanut_allergy_1_0edc2aef", "d_correction_peanut_allergy_2_118b2229", "d_correction_peanut_allergy_3_15745da0", "d_correction_peanut_allergy_4_195a1a1b", "gold_correction_peanut_allergy", "d_correction_peanut_allergy_5_19b5f2b3"], "policy": "hybrid-v1", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "5c250ac833307e3593c15813f85b25992761e49e25839b63230f40d88aae38eb", "response_model": "deepseek-v4-flash", "row_id": "correction_peanut_allergy__hybrid-v1", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 35, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 726, "prompt_tokens": 726, "total_tokens": 761}}
{"attempts": 1, "capacity": 7, "case_id": "correction_review_resolution", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_review_resolution_0_38146c39", "d_correction_review_resolution_1_3b6f954b", "d_correction_review_resolution_2_3d86fd0a", "d_correction_review_resolution_3_3f1e9474", "d_correction_review_resolution_4_4100d0a0", "d_correction_review_resolution_5_4fd1909e", "d_correction_review_resolution_6_505af2f5"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "synthetic-correction", "request_fingerprint": "c6d58ea81b83dfd9a9f531411476fb624371f0ee77de3ee6e2a4c63b7e22e48f", "response_model": "deepseek-v4-flash", "row_id": "correction_review_resolution__confidence", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 63, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 918, "prompt_tokens": 918, "total_tokens": 981}}
{"attempts": 1, "capacity": 7, "case_id": "correction_review_resolution", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_review_resolution_0_38146c39", "d_correction_review_resolution_1_3b6f954b", "d_correction_review_resolution_2_3d86fd0a", "d_correction_review_resolution_3_3f1e9474", "d_correction_review_resolution_4_4100d0a0", "d_correction_review_resolution_5_4fd1909e", "gold_correction_review_resolution"], "policy": "hybrid-v1", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "030a6ec74832d1e977769b0f36d9421133f95fefc76899079f3d690cb88c7bfd", "response_model": "deepseek-v4-flash", "row_id": "correction_review_resolution__hybrid-v1", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 39, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 860, "prompt_tokens": 860, "total_tokens": 899}}
{"attempts": 1, "capacity": 7, "case_id": "correction_shipping_address", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_shipping_address_0_5d3d2817", "d_correction_shipping_address_1_001be529", "d_correction_shipping_address_2_06f04340", "d_correction_shipping_address_3_07b6f563", "d_correction_shipping_address_4_0862e8bf", "d_correction_shipping_address_5_09d032c9", "d_correction_shipping_address_6_0a34ad58"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "synthetic-correction", "request_fingerprint": "382c8b764e59a05d0321a5c04ca1920bd38270f3eca0c7f6863af7655c041d82", "response_model": "deepseek-v4-flash", "row_id": "correction_shipping_address__confidence", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 74, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 859, "prompt_tokens": 859, "total_tokens": 933}}
{"attempts": 1, "capacity": 7, "case_id": "correction_shipping_address", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_shipping_address_0_5d3d2817", "d_correction_shipping_address_1_001be529", "d_correction_shipping_address_2_06f04340", "d_correction_shipping_address_3_07b6f563", "d_correction_shipping_address_4_0862e8bf", "d_correction_shipping_address_5_09d032c9", "gold_correction_shipping_address"], "policy": "hybrid-v1", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "d9fd5fc6a9bcaa71a77cd9aeed1661c0963293071f2a21635d5c4591dab9e23d", "response_model": "deepseek-v4-flash", "row_id": "correction_shipping_address__hybrid-v1", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 50, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 742, "prompt_tokens": 742, "total_tokens": 792}}
{"attempts": 1, "capacity": 7, "case_id": "correction_timezone", "grade_correct": false, "grade_rule": "insufficient", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_timezone_0_1c0ddc50", "d_correction_timezone_1_1d4e3b97", "d_correction_timezone_2_1da05512", "d_correction_timezone_3_1e043500", "d_correction_timezone_4_1faac195", "d_correction_timezone_5_21436231", "d_correction_timezone_6_25e5aa4f"], "policy": "confidence", "prediction": "INSUFFICIENT", "question_type": "synthetic-correction", "request_fingerprint": "0527b101ae55eb93b54f015971cce20fc4af79f5f5aa7217793fbc789f3858f4", "response_model": "deepseek-v4-flash", "row_id": "correction_timezone__confidence", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": false, "support_recall": 0.0, "usage": {"completion_tokens": 61, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 935, "prompt_tokens": 935, "total_tokens": 996}}
{"attempts": 1, "capacity": 7, "case_id": "correction_timezone", "grade_correct": true, "grade_rule": "exact", "grader_version": "deterministic-overlap-v1", "kept_fact_ids": ["d_correction_timezone_0_1c0ddc50", "d_correction_timezone_1_1d4e3b97", "d_correction_timezone_2_1da05512", "d_correction_timezone_3_1e043500", "d_correction_timezone_4_1faac195", "d_correction_timezone_5_21436231", "gold_correction_timezone"], "policy": "hybrid-v1", "prediction": "NO", "question_type": "synthetic-correction", "request_fingerprint": "1d94488a9c66d5acc5fa16bfff3738644d26491e85692acc4dfed54526132a45", "response_model": "deepseek-v4-flash", "row_id": "correction_timezone__hybrid-v1", "scenario": "correction_reserve", "schema_version": 1, "source": "synthetic", "support_all_retained": true, "support_recall": 1.0, "usage": {"completion_tokens": 71, "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 840, "prompt_tokens": 840, "total_tokens": 911}}

65
backend/scripts/benchmark/deermem_eviction/results/pr4789-reproduction-v1/qa.stats.json Normal file
View File

@ -0,0 +1,65 @@
{
"grader_version": "deterministic-overlap-v1",
"policies": [
"confidence",
"hybrid-v1"
],
"protocol_id": "deermem-hybrid-v1-pr4789-reproduction-v1",
"schema_version": 1,
"suites": {
"official": {
"bootstrap": {
"alpha": 0.05,
"iterations": 10000,
"lower": 0.15000000000000002,
"mean_difference": 0.30000000000000004,
"seed": 4789,
"upper": 0.45000000000000007
},
"cases": 40,
"mcnemar": {
"both_correct": 22,
"both_wrong": 4,
"only_first_correct": 1,
"only_second_correct": 13,
"p_value": 0.0018310546875
}
},
"overall": {
"bootstrap": {
"alpha": 0.05,
"iterations": 10000,
"lower": 0.20000000000000007,
"mean_difference": 0.3555555555555555,
"seed": 4789,
"upper": 0.5111111111111111
},
"cases": 45,
"mcnemar": {
"both_correct": 23,
"both_wrong": 4,
"only_first_correct": 1,
"only_second_correct": 17,
"p_value": 0.00014495849609375
}
},
"synthetic": {
"bootstrap": {
"alpha": 0.05,
"iterations": 10000,
"lower": 0.4,
"mean_difference": 0.8,
"seed": 4789,
"upper": 1.0
},
"cases": 5,
"mcnemar": {
"both_correct": 1,
"both_wrong": 0,
"only_first_correct": 0,
"only_second_correct": 4,
"p_value": 0.125
}
}
}
}

88
backend/scripts/benchmark/deermem_eviction/results/pr4789-reproduction-v1/qa.summary.json Normal file
View File

@ -0,0 +1,88 @@
{
"grader_version": "deterministic-overlap-v1",
"groups": [
{
"accuracy": 0.3,
"cases": 10,
"correct": 3,
"policy": "confidence",
"scenario": "access_help",
"source": "longmemeval"
},
{
"accuracy": 0.9,
"cases": 10,
"correct": 9,
"policy": "hybrid-v1",
"scenario": "access_help",
"source": "longmemeval"
},
{
"accuracy": 0.8,
"cases": 10,
"correct": 8,
"policy": "confidence",
"scenario": "confidence_control",
"source": "longmemeval"
},
{
"accuracy": 0.7,
"cases": 10,
"correct": 7,
"policy": "hybrid-v1",
"scenario": "confidence_control",
"source": "longmemeval"
},
{
"accuracy": 0.3,
"cases": 10,
"correct": 3,
"policy": "confidence",
"scenario": "confirmation_help",
"source": "longmemeval"
},
{
"accuracy": 1.0,
"cases": 10,
"correct": 10,
"policy": "hybrid-v1",
"scenario": "confirmation_help",
"source": "longmemeval"
},
{
"accuracy": 0.9,
"cases": 10,
"correct": 9,
"policy": "confidence",
"scenario": "noisy_signal_control",
"source": "longmemeval"
},
{
"accuracy": 0.9,
"cases": 10,
"correct": 9,
"policy": "hybrid-v1",
"scenario": "noisy_signal_control",
"source": "longmemeval"
},
{
"accuracy": 0.2,
"cases": 5,
"correct": 1,
"policy": "confidence",
"scenario": "correction_reserve",
"source": "synthetic"
},
{
"accuracy": 1.0,
"cases": 5,
"correct": 5,
"policy": "hybrid-v1",
"scenario": "correction_reserve",
"source": "synthetic"
}
],
"protocol_id": "deermem-hybrid-v1-pr4789-reproduction-v1",
"qa_capacity": 7,
"schema_version": 1
}

35
backend/scripts/benchmark/deermem_eviction/results/pr4789-reproduction-v1/qa_run.json Normal file
View File

@ -0,0 +1,35 @@
{
"artifacts": {
"answer_prompt_sha256": "2bd4f166219946ccff7ac86f52a053c6dbf8be6d27e178c398d6bf70cf8a7423",
"config_sha256": "e0bcd1581be8457c614a4159b87304e54501d48a7e07990b02f1f6f8627fa026",
"dataset_sha256": "821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c",
"official_manifest_sha256": "7f9b26f3bfb7a19c75af53af77ac5ebaa02eeb05c7824c5b5c08be0d9e43db4b",
"synthetic_manifest_sha256": "2f0c7f9bc05f87322bf3e2d8c33aeb9ec3bf27fb12772cda43915e4d75c2be1e"
},
"created_at": "2026-08-18T04:12:21.258451Z",
"dataset": {
"filename": "longmemeval_oracle.json",
"repository": "xiaowu0162/longmemeval-cleaned",
"revision": "98d7416c24c778c2fee6e6f3006e7a073259d48f",
"sha256": "821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c"
},
"git": {
"dirty": false,
"head": "01f99d61d3e068083978c72c43f5de67d2543c92"
},
"protocol_id": "deermem-hybrid-v1-pr4789-reproduction-v1",
"qa": {
"api_key_env": "DEERMEM_EVAL_ANSWER_API_KEY",
"base_url_env": "DEERMEM_EVAL_ANSWER_BASE_URL",
"capacity": 7,
"grader_version": "deterministic-overlap-v1",
"max_attempts": 3,
"max_tokens": 2048,
"model": "deepseek-v4-flash",
"stream": false,
"temperature": 0.0,
"timeout_seconds": 120,
"workers": 3
},
"schema_version": 2
}

196
backend/scripts/benchmark/deermem_eviction/runner.py Normal file
View File

@ -0,0 +1,196 @@
"""Resumable orchestration for the live answer calls.
Every row (case x policy at the QA capacity) is persisted as its own JSON file
as soon as its provider call succeeds, so a partial paid run can be resumed
without repeating completed calls. Row files contain the prediction and
non-secret metadata only — never questions, reference answers, memory content,
credentials, or response headers. A run directory is bound to the full
protocol identity — config, official and synthetic manifests, answer prompt,
and dataset — and resuming with any changed artifact is rejected. A stored
row is reused only when its identity, kept facts, and request fingerprint all
match the task recomputed from the current protocol; anything else is
re-called.
"""
from __future__ import annotations
import json
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
import httpx
from .config import EvaluationConfig
from .io import load_json, sha256_file
from .provider import ProviderCallError, request_answer, request_fingerprint
from .qa import AnswerTask
from .results import _atomic_write_text, _git_metadata
RESPONSES_DIRNAME = "responses"
ROW_SCHEMA_VERSION = 1
@dataclass(frozen=True)
class AnswerRunReport:
reused: int
called: int
failed: tuple[str, ...]
def response_path(output_dir: Path, row_id: str) -> Path:
return output_dir / RESPONSES_DIRNAME / f"{row_id}.json"
def load_completed_row(path: Path) -> dict[str, object] | None:
if not path.exists():
return None
try:
row = load_json(path)
except (OSError, ValueError):
return None
if not isinstance(row, dict) or row.get("schema_version") != ROW_SCHEMA_VERSION or not isinstance(row.get("prediction"), str):
return None
return row
def _write_row(path: Path, task: AnswerTask, prediction: str, *, attempts: int, request_fingerprint: str, response_model: str | None, usage: dict[str, int]) -> None:
row = {
"schema_version": ROW_SCHEMA_VERSION,
"row_id": task.row_id,
"case_id": task.case_id,
"source": task.source,
"scenario": task.scenario,
"policy": task.policy,
"capacity": task.capacity,
"kept_fact_ids": list(task.kept_fact_ids),
"prediction": prediction,
"attempts": attempts,
"request_fingerprint": request_fingerprint,
"response_model": response_model,
"usage": usage,
"created_at": datetime.now(UTC).isoformat().removesuffix("+00:00") + "Z",
}
_atomic_write_text(path, json.dumps(row, ensure_ascii=False, indent=2, sort_keys=True) + "\n")
def _protocol_artifact_hashes(*, config_path: Path, official_manifest_path: Path, synthetic_manifest_path: Path, prompt_path: Path, dataset_path: Path) -> dict[str, str]:
return {
"config_sha256": sha256_file(config_path),
"official_manifest_sha256": sha256_file(official_manifest_path),
"synthetic_manifest_sha256": sha256_file(synthetic_manifest_path),
"answer_prompt_sha256": sha256_file(prompt_path),
"dataset_sha256": sha256_file(dataset_path),
}
def _changed_artifacts(marker: dict[str, Any], artifacts: dict[str, str]) -> list[str]:
stored = marker.get("artifacts", {})
return sorted(name for name in artifacts if stored.get(name) != artifacts[name])
def verify_run_identity(output_dir: Path, *, config_path: Path, official_manifest_path: Path, synthetic_manifest_path: Path, prompt_path: Path, dataset_path: Path) -> None:
"""Read-only check that a completed run directory was produced by the current protocol artifacts."""
marker_path = output_dir / "qa_run.json"
if not marker_path.exists():
raise ValueError(f"{marker_path} is missing; grading requires the marker written by run-qa")
artifacts = _protocol_artifact_hashes(config_path=config_path, official_manifest_path=official_manifest_path, synthetic_manifest_path=synthetic_manifest_path, prompt_path=prompt_path, dataset_path=dataset_path)
changed = _changed_artifacts(load_json(marker_path), artifacts)
if changed:
raise ValueError(f"{marker_path} was produced with different protocol artifacts ({', '.join(changed)}); refusing to grade")
def ensure_run_config_identity(
output_dir: Path,
*,
config: EvaluationConfig,
config_path: Path,
official_manifest_path: Path,
synthetic_manifest_path: Path,
prompt_path: Path,
dataset_path: Path,
backend_root: Path,
) -> None:
marker_path = output_dir / "qa_run.json"
artifacts = _protocol_artifact_hashes(config_path=config_path, official_manifest_path=official_manifest_path, synthetic_manifest_path=synthetic_manifest_path, prompt_path=prompt_path, dataset_path=dataset_path)
if marker_path.exists():
changed = _changed_artifacts(load_json(marker_path), artifacts)
if changed:
raise ValueError(f"{marker_path} was produced with different protocol artifacts ({', '.join(changed)}); use a new output directory")
return
marker = {
"schema_version": 2,
"protocol_id": config.protocol_id,
"created_at": datetime.now(UTC).isoformat().removesuffix("+00:00") + "Z",
"git": _git_metadata(backend_root),
"dataset": {
"repository": config.dataset.repository,
"revision": config.dataset.revision,
"filename": config.dataset.filename,
"sha256": artifacts["dataset_sha256"],
},
"artifacts": artifacts,
"qa": {
"capacity": config.pool.qa_capacity,
"model": config.qa.model,
"temperature": config.qa.temperature,
"max_tokens": config.qa.max_tokens,
"stream": config.qa.stream,
"timeout_seconds": config.qa.timeout_seconds,
"max_attempts": config.qa.max_attempts,
"workers": config.qa.workers,
"grader_version": config.qa.grader_version,
"api_key_env": config.qa.api_key_env,
"base_url_env": config.qa.base_url_env,
},
}
_atomic_write_text(marker_path, json.dumps(marker, ensure_ascii=False, indent=2, sort_keys=True) + "\n")
def _row_matches_task(row: dict[str, Any], task: AnswerTask, expected_fingerprint: str) -> bool:
return (
row.get("row_id") == task.row_id
and row.get("case_id") == task.case_id
and row.get("source") == task.source
and row.get("scenario") == task.scenario
and row.get("policy") == task.policy
and row.get("capacity") == task.capacity
and tuple(row.get("kept_fact_ids", ())) == task.kept_fact_ids
and row.get("request_fingerprint") == expected_fingerprint
)
def run_answer_calls(tasks: list[AnswerTask], *, config: EvaluationConfig, client: httpx.Client, output_dir: Path, backoff_seconds: float | None = None) -> AnswerRunReport:
if len({task.row_id for task in tasks}) != len(tasks):
raise ValueError("answer tasks must have unique row IDs")
pending = []
for task in tasks:
row = load_completed_row(response_path(output_dir, task.row_id))
if row is None or not _row_matches_task(row, task, request_fingerprint(config.qa, task.messages)):
pending.append(task)
reused = len(tasks) - len(pending)
failed: list[str] = []
call_kwargs = {} if backoff_seconds is None else {"backoff_seconds": backoff_seconds}
def call(task: AnswerTask) -> str | None:
try:
answer = request_answer(client, config.qa, task.messages, **call_kwargs)
except ProviderCallError as error:
return f"{task.row_id}: {error}"
_write_row(
response_path(output_dir, task.row_id),
task,
answer.prediction,
attempts=answer.attempts,
request_fingerprint=answer.request_fingerprint,
response_model=answer.response_model,
usage=answer.usage,
)
return None
if pending:
with ThreadPoolExecutor(max_workers=config.qa.workers) as executor:
failed = [error for error in executor.map(call, pending) if error is not None]
return AnswerRunReport(reused=reused, called=len(pending) - len(failed), failed=tuple(sorted(failed)))

75
backend/scripts/benchmark/deermem_eviction/stats.py Normal file
View File

@ -0,0 +1,75 @@
"""Paired statistics pinned by the versioned config.
The exact McNemar test uses the two-sided exact binomial on discordant pairs.
The paired bootstrap resamples cases with replacement using the seeded
``random.Random`` stream from ``statistics.bootstrap_seed`` and reports the
percentile interval at ``statistics.alpha``; the percentile rule is pinned as
``sorted_diffs[floor((alpha / 2) * n)]`` and
``sorted_diffs[floor((1 - alpha / 2) * n) - 1]``.
"""
from __future__ import annotations
import math
import random
from dataclasses import dataclass
@dataclass(frozen=True)
class McNemarResult:
both_correct: int
both_wrong: int
only_first_correct: int
only_second_correct: int
p_value: float
@dataclass(frozen=True)
class BootstrapResult:
mean_difference: float
lower: float
upper: float
iterations: int
seed: int
alpha: float
def exact_mcnemar(pairs: list[tuple[bool, bool]]) -> McNemarResult:
if not pairs:
raise ValueError("McNemar requires at least one pair")
both_correct = sum(1 for first, second in pairs if first and second)
both_wrong = sum(1 for first, second in pairs if not first and not second)
only_first = sum(1 for first, second in pairs if first and not second)
only_second = sum(1 for first, second in pairs if not first and second)
discordant = only_first + only_second
if discordant == 0:
p_value = 1.0
else:
tail = sum(math.comb(discordant, k) for k in range(min(only_first, only_second) + 1)) * 0.5**discordant
p_value = min(1.0, 2.0 * tail)
return McNemarResult(both_correct=both_correct, both_wrong=both_wrong, only_first_correct=only_first, only_second_correct=only_second, p_value=p_value)
def paired_bootstrap_difference(pairs: list[tuple[bool, bool]], *, seed: int, iterations: int, alpha: float) -> BootstrapResult:
if not pairs:
raise ValueError("The paired bootstrap requires at least one pair")
if not 0.0 < alpha < 1.0:
raise ValueError("alpha must be between 0 and 1")
generator = random.Random(seed)
count = len(pairs)
differences: list[float] = []
for _ in range(iterations):
resample = [pairs[generator.randrange(count)] for _ in range(count)]
differences.append(sum(second for _, second in resample) / count - sum(first for first, _ in resample) / count)
differences.sort()
lower_index = math.floor((alpha / 2) * iterations)
upper_index = math.floor((1 - alpha / 2) * iterations) - 1
mean_difference = sum(second for _, second in pairs) / count - sum(first for first, _ in pairs) / count
return BootstrapResult(
mean_difference=mean_difference,
lower=differences[max(0, lower_index)],
upper=differences[min(iterations - 1, max(0, upper_index))],
iterations=iterations,
seed=seed,
alpha=alpha,
)

76
backend/tests/test_bench_deermem_eviction_contracts.py Normal file
View File

@ -0,0 +1,76 @@
from __future__ import annotations
import json
from pathlib import Path
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.io import sha256_file
from scripts.benchmark.deermem_eviction.manifest import load_official_manifest, load_synthetic_manifest
EVAL_ROOT = Path(__file__).parents[1] / "scripts" / "benchmark" / "deermem_eviction"
def _all_keys(value: object) -> set[str]:
if isinstance(value, dict):
return set(value) | {key for child in value.values() for key in _all_keys(child)}
if isinstance(value, list):
return {key for child in value for key in _all_keys(child)}
return set()
def test_committed_protocol_is_pinned_and_does_not_copy_longmemeval_text() -> None:
config = load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
official = load_official_manifest(EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json")
synthetic = load_synthetic_manifest(EVAL_ROOT / "manifests" / "synthetic-corrections-pr4789-v1.json")
assert config.schema_version == 1
assert config.protocol_id == "deermem-hybrid-v1-pr4789-reproduction-v1"
assert config.dataset.revision == "98d7416c24c778c2fee6e6f3006e7a073259d48f"
assert config.dataset.sha256 == "821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c"
assert config.pool.size == 10
assert config.pool.capacities == [5, 7, 9]
assert config.pool.qa_capacity == 7
assert sum(config.policies.hybrid_v1.weights.values()) == 1.0
official_ids = [question_id for question_ids in official.scenarios.values() for question_id in question_ids]
assert len(official_ids) == 40
assert len(set(official_ids)) == 40
assert set(official.scenarios) == {
"confirmation_help",
"access_help",
"confidence_control",
"noisy_signal_control",
}
assert official.scenario_order == [
"confirmation_help",
"access_help",
"confidence_control",
"noisy_signal_control",
]
assert len(synthetic.cases) == 5
raw_official = json.loads((EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json").read_text())
assert not ({"question", "answer", "content", "haystack_sessions"} & _all_keys(raw_official))
prompt_path = EVAL_ROOT / config.qa.answer_prompt.path
assert sha256_file(prompt_path) == config.qa.answer_prompt.sha256
def test_cli_contract_validation_is_offline(capsys) -> None:
from scripts.benchmark.deermem_eviction.cli import main
assert main(["validate-contracts"]) == 0
assert "validated 40 official and 5 synthetic cases" in capsys.readouterr().out
def test_required_policy_version_is_checked_against_production() -> None:
import pytest
from deerflow.agents.memory.backends.deermem.deermem.core.eviction import EVICTION_POLICY_HYBRID_V1
from scripts.benchmark.deermem_eviction.policy import require_production_policy
config = load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
assert config.required_policy_version == EVICTION_POLICY_HYBRID_V1
require_production_policy(config.required_policy_version)
with pytest.raises(ValueError, match="production implements"):
require_production_policy("hybrid-v2")

73
backend/tests/test_bench_deermem_eviction_dataset.py Normal file
View File

@ -0,0 +1,73 @@
from __future__ import annotations
import hashlib
import json
from pathlib import Path
import pytest
from scripts.benchmark.deermem_eviction.dataset import DatasetIntegrityError, build_distractor_bank, extract_evidence, load_longmemeval
def _row(question_id: str, question_type: str, content: str) -> dict[str, object]:
return {
"question_id": question_id,
"question_type": question_type,
"question": f"Question for {question_id}?",
"answer": "answer",
"question_date": "2026/08/13 (Thu) 10:00",
"haystack_session_ids": [f"session-{question_id}"],
"haystack_dates": ["2026/08/01 (Sat) 09:00"],
"haystack_sessions": [[{"role": "user", "content": content, "has_answer": True}]],
}
def test_load_longmemeval_rejects_hash_mismatch(tmp_path: Path) -> None:
dataset_path = tmp_path / "oracle.json"
dataset_path.write_text("[]")
with pytest.raises(DatasetIntegrityError, match="SHA-256 mismatch"):
load_longmemeval(dataset_path, expected_sha256="0" * 64)
def test_load_longmemeval_indexes_unique_rows(tmp_path: Path) -> None:
rows = [_row("b", "knowledge-update", "b" * 50), _row("a", "temporal-reasoning", "a" * 50)]
payload = json.dumps(rows).encode()
dataset_path = tmp_path / "oracle.json"
dataset_path.write_bytes(payload)
dataset = load_longmemeval(dataset_path, expected_sha256=hashlib.sha256(payload).hexdigest())
assert sorted(dataset.rows_by_id) == ["a", "b"]
def test_extract_evidence_uses_marked_turns_and_per_session_user_fallback() -> None:
row = {
"haystack_session_ids": ["s1", "s2"],
"haystack_dates": ["2026/01/01", "2026/01/02"],
"haystack_sessions": [
[
{"role": "user", "content": "ignore unmarked", "has_answer": False},
{"role": "assistant", "content": "marked answer", "has_answer": True},
],
[
{"role": "user", "content": "fallback user"},
{"role": "assistant", "content": "ignore fallback assistant"},
],
],
}
assert extract_evidence(row) == ("SESSION s1 AT 2026/01/01\nASSISTANT: marked answer\n\nSESSION s2 AT 2026/01/02\nUSER: fallback user")
def test_distractor_bank_is_filtered_sorted_and_bounded() -> None:
rows = [
_row("z", "single-session-user", "z" * 60),
_row("a", "single-session-preference", "a" * 60),
_row("wrong-type", "knowledge-update", "x" * 60),
_row("too-long", "single-session-user", "x" * 800),
]
bank = build_distractor_bank(rows, allowed_types={"single-session-user", "single-session-preference"}, min_evidence_chars=40, max_evidence_chars=700, limit=40)
assert [item.question_id for item in bank] == ["a", "z"]

94
backend/tests/test_bench_deermem_eviction_grading.py Normal file
View File

@ -0,0 +1,94 @@
from __future__ import annotations
import inspect
from pathlib import Path
import pytest
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.grading import GRADER_VERSION, OVERLAP_THRESHOLD, grade_answer, normalize_answer
EVAL_ROOT = Path(__file__).parents[1] / "scripts" / "benchmark" / "deermem_eviction"
def test_config_pins_the_committed_grader_version() -> None:
config = load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
assert config.qa.grader_version == GRADER_VERSION == "deterministic-overlap-v1"
def test_grader_is_blind_by_construction() -> None:
parameters = inspect.signature(grade_answer).parameters
assert list(parameters) == ["prediction", "reference"]
def test_normalization_lowercases_strips_and_maps_number_words() -> None:
assert normalize_answer("Seven WEEKS!") == ["7", "weeks"]
assert normalize_answer("fifteen") == ["15"]
assert normalize_answer("eleven") == ["eleven"]
assert normalize_answer("70-200mm zoom lens") == ["70", "200mm", "zoom", "lens"]
assert normalize_answer(" \t\n ") == []
def test_empty_and_insufficient_predictions_are_rejected() -> None:
empty = grade_answer("", "3 weeks")
assert not empty.correct
assert empty.rule == "empty-prediction"
assert not grade_answer(" ", "3 weeks").correct
result = grade_answer("INSUFFICIENT.", "3 weeks")
assert not result.correct
assert result.rule == "insufficient"
assert not grade_answer("insufficient", "3 weeks").correct
def test_empty_reference_is_a_contract_error() -> None:
with pytest.raises(ValueError):
grade_answer("3 weeks", " . ")
def test_exact_match_ignores_case_punctuation_and_number_words() -> None:
assert grade_answer("Every week.", "every week").rule == "exact"
assert grade_answer("NO", "no").rule == "exact"
assert grade_answer("seven", "7").rule == "exact"
def test_substring_matches_are_token_level_and_bidirectional() -> None:
assert grade_answer("132 points", "132").rule == "substring"
assert grade_answer("132", "132 points").rule == "substring"
assert grade_answer("Ford F-150 pickup truck.", "Ford F-150").rule == "substring"
assert grade_answer("Ford F-150 pickup truck.", "a Ford F-150").correct
assert not grade_answer("5", "25").correct
assert not grade_answer("no", "north").correct
def test_conflicting_numeric_answers_are_rejected() -> None:
result = grade_answer("5", "3 weeks")
assert not result.correct
assert result.rule == "numeric-conflict"
assert not grade_answer("12 weeks", "8 weeks").correct
def test_numbers_inside_an_explicit_reference_range_are_accepted() -> None:
reference = "ranging from 5 to 10 hours"
assert grade_answer("7", reference).rule == "numeric-range"
assert grade_answer("5 hours", reference).rule == "numeric-range"
assert grade_answer("10", reference).correct
assert not grade_answer("4", reference).correct
assert not grade_answer("11", reference).correct
assert grade_answer("7", "ranging from 5 dollars to 10 dollars").rule == "numeric-range"
def test_overlap_requires_sixty_percent_in_both_directions() -> None:
assert OVERLAP_THRESHOLD == 0.6
accepted = grade_answer("under my bed", "under the bed")
assert accepted.correct
assert accepted.rule == "overlap-accept"
assert grade_answer("red kite string", "red kite ribbon").correct
assert not grade_answer("red kite", "red balloon ribbon string flag").correct
assert not grade_answer("50mm prime lens", "70-200mm zoom lens").correct
assert not grade_answer("YES", "NO").correct
def test_all_stopword_predictions_cannot_pass_overlap() -> None:
result = grade_answer("of the", "3 weeks")
assert not result.correct
assert result.rule == "overlap-reject"

90
backend/tests/test_bench_deermem_eviction_policy.py Normal file
View File

@ -0,0 +1,90 @@
from __future__ import annotations
from datetime import UTC, datetime
from scripts.benchmark.deermem_eviction.dataset import EvidenceRecord
from scripts.benchmark.deermem_eviction.policy import evaluate_case
from scripts.benchmark.deermem_eviction.pool import build_case
NOW = datetime(2026, 8, 13, tzinfo=UTC)
def _evidence(question_id: str, question_type: str = "single-session-user") -> EvidenceRecord:
return EvidenceRecord(
question_id=question_id,
question_type=question_type,
question=f"Question for {question_id}?",
answer="answer",
question_date="2026/08/13 (Thu) 10:00",
content=f"Evidence for {question_id}",
)
def test_confirmation_scenario_is_deterministic_and_uses_production_policies() -> None:
support = _evidence("support", "knowledge-update")
distractors = [_evidence(f"d{i}") for i in range(9)]
first = build_case(
support=support,
distractors=distractors,
scenario="confirmation_help",
loss_rank=8,
evaluation_time=NOW,
)
second = build_case(
support=support,
distractors=distractors,
scenario="confirmation_help",
loss_rank=8,
evaluation_time=NOW,
)
assert first == second
assert len(first.facts) == 10
assert [fact["id"] for fact in first.facts] == sorted(fact["id"] for fact in first.facts)
confidence = evaluate_case(first, policy_name="confidence", capacity=7)
hybrid = evaluate_case(first, policy_name="hybrid-v1", capacity=7)
assert confidence.support_all_retained is False
assert hybrid.support_all_retained is True
assert confidence.kept_fact_ids != hybrid.kept_fact_ids
assert all("content" not in evicted for evicted in hybrid.evicted)
def test_correction_reserve_retains_a_low_confidence_correction() -> None:
support = _evidence("correction_case", "synthetic-correction")
distractors = [_evidence(f"d{i}") for i in range(9)]
case = build_case(
support=support,
distractors=distractors,
scenario="correction_reserve",
loss_rank=8,
evaluation_time=NOW,
)
confidence = evaluate_case(case, policy_name="confidence", capacity=7)
hybrid = evaluate_case(case, policy_name="hybrid-v1", capacity=7)
assert confidence.support_all_retained is False
assert hybrid.support_all_retained is True
assert hybrid.reserved_correction_slots == 1
def test_policy_result_contains_ids_and_scores_but_not_dataset_text() -> None:
case = build_case(
support=_evidence("support", "knowledge-update"),
distractors=[_evidence(f"d{i}") for i in range(9)],
scenario="access_help",
loss_rank=6,
evaluation_time=NOW,
)
result = evaluate_case(case, policy_name="hybrid-v1", capacity=7).to_public_dict()
assert result["case_id"] == "support"
assert result["support_fact_ids"] == ["gold_support"]
assert all(fact_id.startswith("d_support_") or fact_id == "gold_support" for fact_id in result["kept_fact_ids"])
assert "question" not in result
assert "answer" not in result
assert "facts" not in result

122
backend/tests/test_bench_deermem_eviction_protocol.py Normal file
View File

@ -0,0 +1,122 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
import pytest
from scripts.benchmark.deermem_eviction.dataset import DatasetIntegrityError, EvidenceRecord, LongMemEvalDataset
from scripts.benchmark.deermem_eviction.manifest import OfficialManifest
from scripts.benchmark.deermem_eviction.protocol import _distractors, validate_official_selection
SCENARIO_ORDER = ["confirmation_help", "access_help", "confidence_control", "noisy_signal_control"]
def _row(question_id: str, question_type: str, *, answer: str = "short answer", evidence_chars: int = 120) -> dict[str, Any]:
return {
"question_id": question_id,
"question_type": question_type,
"question": f"question for {question_id}?",
"answer": answer,
"haystack_session_ids": ["s1"],
"haystack_dates": ["2023/05/20"],
"haystack_sessions": [[{"role": "user", "content": "x" * evidence_chars, "has_answer": True}]],
}
def _eligible_ids(prefix: str) -> list[str]:
return [f"{prefix}-{index:03d}" for index in range(1, 21)]
def _dataset(rows: list[dict[str, Any]]) -> LongMemEvalDataset:
return LongMemEvalDataset(path=Path("synthetic"), sha256="0" * 64, rows=tuple(rows), rows_by_id={row["question_id"]: row for row in rows})
def _rows_with_exclusions() -> list[dict[str, Any]]:
rows = [_row(question_id, "knowledge-update") for question_id in _eligible_ids("ku")]
rows += [_row(question_id, "temporal-reasoning") for question_id in _eligible_ids("tr")]
# Every excluded row sorts before the eligible IDs, so a broken exclusion changes the recomputed selection.
rows.append(_row("ku-000-pilot", "knowledge-update"))
rows.append(_row("ku-000_abs", "knowledge-update"))
rows.append(_row("ku-000-long-answer", "knowledge-update", answer="a" * 150))
rows.append(_row("ku-000-refusal", "knowledge-update", answer="there is not enough information"))
rows.append(_row("ku-000-evidence", "knowledge-update", evidence_chars=2500))
return rows
def _manifest(*, excluded_pilot_ids: list[str] | None = None, scenarios: dict[str, list[str]] | None = None) -> OfficialManifest:
if scenarios is None:
ku, tr = _eligible_ids("ku"), _eligible_ids("tr")
scenarios = {scenario: ku[index * 5 : (index + 1) * 5] + tr[index * 5 : (index + 1) * 5] for index, scenario in enumerate(SCENARIO_ORDER)}
return OfficialManifest.model_validate(
{
"schema_version": 1,
"protocol_id": "synthetic-protocol",
"selection": {
"eligible_question_types": ["knowledge-update", "temporal-reasoning"],
"excluded_pilot_ids": ["ku-000-pilot"] if excluded_pilot_ids is None else excluded_pilot_ids,
"exclude_abstention_suffix": "_abs",
"answer_min_chars": 1,
"answer_max_chars": 100,
"answer_excluded_substrings": ["not enough", "only mentioned"],
"evidence_min_chars": 1,
"evidence_max_chars": 2000,
"take_per_question_type": 20,
"cases_per_type_per_scenario": 5,
},
"scenario_order": SCENARIO_ORDER,
"loss_ranks": [6, 6, 6, 8, 8, 8, 10, 10, 10, 10],
"scenarios": scenarios,
}
)
def test_selection_recomputation_accepts_a_manifest_matching_the_published_rule() -> None:
validate_official_selection(_dataset(_rows_with_exclusions()), _manifest())
def test_selection_recomputation_rejects_ids_that_break_the_rule() -> None:
ku, tr = _eligible_ids("ku"), _eligible_ids("tr")
scenarios = {scenario: ku[index * 5 : (index + 1) * 5] + tr[index * 5 : (index + 1) * 5] for index, scenario in enumerate(SCENARIO_ORDER)}
scenarios["confirmation_help"], scenarios["access_help"] = (
scenarios["confirmation_help"][:9] + [scenarios["access_help"][9]],
scenarios["access_help"][:9] + [scenarios["confirmation_help"][9]],
)
with pytest.raises(DatasetIntegrityError, match="do not match the declared selection rule"):
validate_official_selection(_dataset(_rows_with_exclusions()), _manifest(scenarios=scenarios))
def test_selection_recomputation_applies_every_published_exclusion() -> None:
manifest = _manifest()
excluded = {"ku-000-pilot", "ku-000_abs", "ku-000-long-answer", "ku-000-refusal", "ku-000-evidence"}
pinned = {question_id for question_ids in manifest.scenarios.values() for question_id in question_ids}
assert not (excluded & pinned)
# Dropping the pilot exclusion changes the recomputed selection, so validation must fail against the pinned IDs.
with pytest.raises(DatasetIntegrityError):
validate_official_selection(_dataset(_rows_with_exclusions()), _manifest(excluded_pilot_ids=[]))
def test_selection_recomputation_requires_enough_eligible_rows() -> None:
rows = [row for row in _rows_with_exclusions() if row["question_id"] != "tr-020"]
with pytest.raises(DatasetIntegrityError, match="not enough eligible"):
validate_official_selection(_dataset(rows), _manifest())
def _bank(size: int = 5) -> list[EvidenceRecord]:
return [EvidenceRecord(question_id=f"b{index}", question_type="single-session-user", question="q?", answer="a", question_date=None, content=f"evidence {index}") for index in range(size)]
def test_distractor_offset_derivation_and_wraparound_are_pinned() -> None:
# Hardcoded expectations for sha256(f"offset-ns:{case_id}").digest()[:4] % 5;
# a regression in the digest slice, modulus, or wraparound changes these indices.
bank = _bank()
no_wrap = _distractors("case-a", bank, count=3, namespace="offset-ns")
assert [record.question_id for record in no_wrap] == ["b0", "b1", "b2"]
wraps = _distractors("case-c", bank, count=3, namespace="offset-ns")
assert [record.question_id for record in wraps] == ["b3", "b4", "b0"]
assert [record.question_id for record in _distractors("case-a", bank, count=3, namespace="alt-ns")] == ["b1", "b2", "b3"]
def test_distractor_selection_requires_a_large_enough_bank() -> None:
with pytest.raises(DatasetIntegrityError, match="required"):
_distractors("case-a", _bank(size=2), count=3, namespace="offset-ns")

94
backend/tests/test_bench_deermem_eviction_published_results.py Normal file
View File

@ -0,0 +1,94 @@
from __future__ import annotations
import json
from pathlib import Path
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.grading import GRADER_VERSION
from scripts.benchmark.deermem_eviction.stats import exact_mcnemar, paired_bootstrap_difference
EVAL_ROOT = Path(__file__).parents[1] / "scripts" / "benchmark" / "deermem_eviction"
RESULTS_ROOT = EVAL_ROOT / "results" / "pr4789-reproduction-v1"
ALLOWED_ROW_KEYS = {
"schema_version",
"row_id",
"case_id",
"source",
"scenario",
"question_type",
"policy",
"capacity",
"kept_fact_ids",
"support_all_retained",
"support_recall",
"prediction",
"grade_correct",
"grade_rule",
"grader_version",
"attempts",
"request_fingerprint",
"response_model",
"usage",
}
def _rows() -> list[dict]:
return [json.loads(line) for line in (RESULTS_ROOT / "qa.rows.jsonl").read_text(encoding="utf-8").splitlines()]
def test_published_rows_contain_only_allowed_metadata() -> None:
rows = _rows()
config = load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
assert len(rows) == 90
assert {row["policy"] for row in rows} == {"confidence", "hybrid-v1"}
for row in rows:
assert set(row) == ALLOWED_ROW_KEYS
assert row["capacity"] == config.pool.qa_capacity
assert len(row["kept_fact_ids"]) == config.pool.qa_capacity
assert row["grader_version"] == GRADER_VERSION
assert isinstance(row["prediction"], str)
def test_published_statistics_are_recomputable_from_the_rows() -> None:
rows = _rows()
config = load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
statistics = json.loads((RESULTS_ROOT / "qa.stats.json").read_text(encoding="utf-8"))
grades: dict[str, dict[str, bool]] = {}
sources: dict[str, str] = {}
for row in rows:
grades.setdefault(row["case_id"], {})[row["policy"]] = row["grade_correct"]
sources[row["case_id"]] = row["source"]
suites = {
"official": [case_id for case_id in sorted(grades) if sources[case_id] == "longmemeval"],
"synthetic": [case_id for case_id in sorted(grades) if sources[case_id] == "synthetic"],
"overall": sorted(grades),
}
for suite, case_ids in suites.items():
pairs = [(grades[case_id]["confidence"], grades[case_id]["hybrid-v1"]) for case_id in case_ids]
expected = exact_mcnemar(pairs)
published = statistics["suites"][suite]["mcnemar"]
assert published["p_value"] == expected.p_value
assert published["only_first_correct"] == expected.only_first_correct
assert published["only_second_correct"] == expected.only_second_correct
assert statistics["suites"][suite]["cases"] == len(pairs)
expected_bootstrap = paired_bootstrap_difference(pairs, seed=config.statistics.bootstrap_seed, iterations=config.statistics.bootstrap_iterations, alpha=config.statistics.alpha)
published_bootstrap = statistics["suites"][suite]["bootstrap"]
assert published_bootstrap["mean_difference"] == expected_bootstrap.mean_difference
assert published_bootstrap["lower"] == expected_bootstrap.lower
assert published_bootstrap["upper"] == expected_bootstrap.upper
assert (published_bootstrap["seed"], published_bootstrap["iterations"], published_bootstrap["alpha"]) == (expected_bootstrap.seed, expected_bootstrap.iterations, expected_bootstrap.alpha)
def test_published_summary_matches_the_rows_and_run_provenance_is_secret_free() -> None:
rows = _rows()
summary = json.loads((RESULTS_ROOT / "qa.summary.json").read_text(encoding="utf-8"))
for group in summary["groups"]:
matching = [row for row in rows if (row["source"], row["scenario"], row["policy"]) == (group["source"], group["scenario"], group["policy"])]
assert len(matching) == group["cases"]
assert sum(1 for row in matching if row["grade_correct"]) == group["correct"]
run = json.loads((RESULTS_ROOT / "qa_run.json").read_text(encoding="utf-8"))
serialized = json.dumps(run)
assert "sk-" not in serialized
assert run["qa"]["api_key_env"] == "DEERMEM_EVAL_ANSWER_API_KEY"
assert run["dataset"]["sha256"] == load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml").dataset.sha256

349
backend/tests/test_bench_deermem_eviction_qa.py Normal file
View File

@ -0,0 +1,349 @@
from __future__ import annotations
import json
from dataclasses import replace
from pathlib import Path
import httpx
import pytest
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.policy import PolicyResult
from scripts.benchmark.deermem_eviction.pool import PreparedCase
from scripts.benchmark.deermem_eviction.provider import ProviderCallError, ProviderConfigurationError, ProviderSettings, build_client, request_answer, resolve_provider_settings
from scripts.benchmark.deermem_eviction.qa import build_answer_task, render_answer_messages
from scripts.benchmark.deermem_eviction.runner import ensure_run_config_identity, response_path, run_answer_calls, verify_run_identity
EVAL_ROOT = Path(__file__).parents[1] / "scripts" / "benchmark" / "deermem_eviction"
def _load_config():
return load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
def _template() -> str:
return (EVAL_ROOT / "prompts" / "answer-v1.txt").read_text(encoding="utf-8")
def _case(case_id: str = "case-1", question_date: str | None = "2023/05/20 (Sat) 02:21") -> PreparedCase:
config = _load_config()
facts = [
{"id": "fact-b", "content": "The cat sleeps in the study.", "category": "context", "confidence": 0.7, "createdAt": "2026-02-14T00:00:00Z", "source": "synthetic"},
{"id": "fact-a", "content": "The user moved to Lyon.", "category": "context", "confidence": 0.9, "createdAt": "2026-02-14T00:00:00Z", "source": "synthetic"},
{"id": "fact-c", "content": "The user has two bikes.", "category": "context", "confidence": 0.8, "createdAt": "2026-02-14T00:00:00Z", "source": "synthetic"},
]
return PreparedCase(
case_id=case_id,
source="synthetic",
scenario="correction_reserve",
question_type="synthetic-correction",
question="Where does the user live?",
answer="Lyon",
question_date=question_date,
evaluation_time=config.evaluation_time,
facts=facts,
usage={},
support_fact_ids=("fact-a",),
)
def _policy_result(case: PreparedCase, kept: tuple[str, ...], policy: str = "hybrid-v1") -> PolicyResult:
return PolicyResult(
case_id=case.case_id,
source=case.source,
scenario=case.scenario,
question_type=case.question_type,
policy=policy, # type: ignore[arg-type]
capacity=7,
support_fact_ids=case.support_fact_ids,
kept_fact_ids=kept,
evicted=(),
scores={},
support_all_retained=True,
support_recall=1.0,
reserved_correction_slots=0,
)
def test_rendering_pins_fact_order_date_line_and_block_format() -> None:
messages = render_answer_messages(
_template(),
question="Where does the user live?",
question_date="2023/05/20 (Sat) 02:21",
retained_facts=[("fact-b", "The cat sleeps in the study."), ("fact-a", "The user moved to Lyon.")],
)
assert messages[0]["role"] == "system"
assert messages[0]["content"] == "Answer using only the stored memory below. If the answer is not supported, output exactly INSUFFICIENT. For a YES/NO question, output only YES or NO. Otherwise give only the shortest direct answer."
assert messages[1]["role"] == "user"
assert messages[1]["content"] == ("CURRENT DATE: 2023/05/20 (Sat) 02:21\nSTORED MEMORY:\n[fact-a]\nThe user moved to Lyon.\n\n[fact-b]\nThe cat sleeps in the study.\n\nQUESTION: Where does the user live?")
def test_rendering_omits_the_date_line_when_absent() -> None:
messages = render_answer_messages(_template(), question="Q?", question_date=None, retained_facts=[("fact-a", "content")])
assert messages[1]["content"].startswith("STORED MEMORY:\n")
assert "CURRENT DATE" not in messages[1]["content"]
def test_rendering_rejects_empty_or_duplicate_facts() -> None:
with pytest.raises(ValueError):
render_answer_messages(_template(), question="Q?", question_date=None, retained_facts=[])
with pytest.raises(ValueError):
render_answer_messages(_template(), question="Q?", question_date=None, retained_facts=[("fact-a", "x"), ("fact-a", "y")])
def test_build_answer_task_renders_only_kept_facts() -> None:
case = _case()
task = build_answer_task(case, _policy_result(case, kept=("fact-a", "fact-c")), _template())
assert task.row_id == "case-1__hybrid-v1"
assert task.kept_fact_ids == ("fact-a", "fact-c")
user = task.messages[1]["content"]
assert "[fact-a]" in user and "[fact-c]" in user
assert "fact-b" not in user
with pytest.raises(ValueError):
build_answer_task(_case(case_id="other"), _policy_result(case, kept=("fact-a",)), _template())
def test_provider_settings_errors_name_the_missing_variables(monkeypatch: pytest.MonkeyPatch) -> None:
config = _load_config()
monkeypatch.delenv(config.qa.api_key_env, raising=False)
monkeypatch.delenv(config.qa.base_url_env, raising=False)
with pytest.raises(ProviderConfigurationError) as excinfo:
resolve_provider_settings(config.qa)
assert config.qa.api_key_env in str(excinfo.value)
assert config.qa.base_url_env in str(excinfo.value)
monkeypatch.setenv(config.qa.api_key_env, "secret-key")
monkeypatch.setenv(config.qa.base_url_env, "https://example.invalid/v1")
settings = resolve_provider_settings(config.qa)
assert settings.api_key == "secret-key"
assert settings.base_url == "https://example.invalid/v1"
def _mock_client(handler, qa) -> httpx.Client:
return httpx.Client(base_url="https://example.invalid/v1", headers={"Authorization": "Bearer secret-key"}, transport=httpx.MockTransport(handler), timeout=qa.timeout_seconds)
def _success_body(prediction: str = "Lyon") -> dict:
return {"model": "deepseek-v4-flash", "choices": [{"message": {"role": "assistant", "content": prediction}}], "usage": {"prompt_tokens": 100, "completion_tokens": 3, "detail": "ignored"}}
def test_request_answer_parses_prediction_and_non_secret_metadata() -> None:
qa = _load_config().qa
seen: list[httpx.Request] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(request)
return httpx.Response(200, json=_success_body())
answer = request_answer(_mock_client(handler, qa), qa, ({"role": "system", "content": "s"}, {"role": "user", "content": "u"}), backoff_seconds=0)
assert answer.prediction == "Lyon"
assert answer.attempts == 1
assert answer.usage == {"prompt_tokens": 100, "completion_tokens": 3}
assert answer.response_model == "deepseek-v4-flash"
payload = json.loads(seen[0].content)
assert payload["model"] == qa.model
assert payload["temperature"] == qa.temperature
assert payload["max_tokens"] == qa.max_tokens
assert payload["stream"] is qa.stream
assert seen[0].url.path.endswith("/chat/completions")
def test_request_answer_retries_retryable_failures_and_gives_up() -> None:
qa = _load_config().qa
statuses = [429, 500]
def flaky(request: httpx.Request) -> httpx.Response:
if statuses:
return httpx.Response(statuses.pop(0), json={})
return httpx.Response(200, json=_success_body())
answer = request_answer(_mock_client(flaky, qa), qa, ({"role": "user", "content": "u"},), backoff_seconds=0)
assert answer.attempts == 3
def always_broken(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("boom", request=request)
with pytest.raises(ProviderCallError, match="failed after 3 attempts"):
request_answer(_mock_client(always_broken, qa), qa, ({"role": "user", "content": "u"},), backoff_seconds=0)
def test_request_answer_rejects_non_retryable_and_malformed_responses() -> None:
qa = _load_config().qa
with pytest.raises(ProviderCallError, match="non-retryable status 401"):
request_answer(_mock_client(lambda request: httpx.Response(401, json={}), qa), qa, ({"role": "user", "content": "u"},), backoff_seconds=0)
with pytest.raises(ProviderCallError, match="choices"):
request_answer(_mock_client(lambda request: httpx.Response(200, json={"choices": []}), qa), qa, ({"role": "user", "content": "u"},), backoff_seconds=0)
def test_build_client_uses_configured_timeout_and_bearer_header() -> None:
config = _load_config()
with build_client(ProviderSettings(base_url="https://example.invalid/v1", api_key="secret-key"), config.qa) as client:
assert client.headers["Authorization"] == "Bearer secret-key"
assert client.timeout.read == config.qa.timeout_seconds
def _tasks(count: int = 2) -> list:
template = _template()
tasks = []
for index in range(count):
case = _case(case_id=f"case-{index}")
tasks.append(build_answer_task(case, _policy_result(case, kept=("fact-a", "fact-b")), template))
return tasks
def test_run_answer_calls_persists_rows_and_resumes_without_new_calls(tmp_path: Path) -> None:
config = _load_config()
calls = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(request)
return httpx.Response(200, json=_success_body())
tasks = _tasks()
with _mock_client(handler, config.qa) as client:
first = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (first.reused, first.called, first.failed) == (0, 2, ())
assert len(calls) == 2
row = json.loads(response_path(tmp_path, tasks[0].row_id).read_text(encoding="utf-8"))
assert set(row) == {"schema_version", "row_id", "case_id", "source", "scenario", "policy", "capacity", "kept_fact_ids", "prediction", "attempts", "request_fingerprint", "response_model", "usage", "created_at"}
assert row["prediction"] == "Lyon"
serialized = json.dumps(row)
assert "Where does the user live" not in serialized
assert "The user moved to Lyon." not in serialized
assert "Bearer" not in serialized
second = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (second.reused, second.called, second.failed) == (2, 0, ())
assert len(calls) == 2
response_path(tmp_path, tasks[0].row_id).write_text("{not json", encoding="utf-8")
third = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (third.reused, third.called, third.failed) == (1, 1, ())
assert len(calls) == 3
def test_run_answer_calls_refuses_to_reuse_a_row_bound_to_another_case(tmp_path: Path) -> None:
config = _load_config()
calls = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(request)
return httpx.Response(200, json=_success_body())
tasks = _tasks()
with _mock_client(handler, config.qa) as client:
run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert len(calls) == 2
path = response_path(tmp_path, tasks[0].row_id)
original = json.loads(path.read_text(encoding="utf-8"))
for field, value in (("case_id", tasks[1].case_id), ("source", f"not-{tasks[0].source}"), ("scenario", f"not-{tasks[0].scenario}")):
path.write_text(json.dumps(dict(original, **{field: value})), encoding="utf-8")
report = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (report.reused, report.called, report.failed) == (1, 1, ())
assert len(calls) == 5
def test_run_answer_calls_reports_failures_without_writing_rows(tmp_path: Path) -> None:
config = _load_config()
def broken(request: httpx.Request) -> httpx.Response:
return httpx.Response(500, json={})
tasks = _tasks(count=1)
with _mock_client(broken, config.qa) as client:
report = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert report.called == 0
assert len(report.failed) == 1
assert tasks[0].row_id in report.failed[0]
assert not response_path(tmp_path, tasks[0].row_id).exists()
def test_run_directory_is_bound_to_the_full_protocol_identity(tmp_path: Path) -> None:
config = _load_config()
paths = {
"config_path": EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml",
"official_manifest_path": EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json",
"synthetic_manifest_path": EVAL_ROOT / "manifests" / "synthetic-corrections-pr4789-v1.json",
"prompt_path": EVAL_ROOT / "prompts" / "answer-v1.txt",
}
dataset_path = tmp_path / "dataset.json"
dataset_path.write_text("[]", encoding="utf-8")
backend_root = Path(__file__).parents[1]
output_dir = tmp_path / "run"
ensure_run_config_identity(output_dir, config=config, dataset_path=dataset_path, backend_root=backend_root, **paths)
marker = json.loads((output_dir / "qa_run.json").read_text(encoding="utf-8"))
assert marker["qa"]["model"] == config.qa.model
assert set(marker["artifacts"]) == {"config_sha256", "official_manifest_sha256", "synthetic_manifest_sha256", "answer_prompt_sha256", "dataset_sha256"}
assert "secret" not in json.dumps(marker).lower()
ensure_run_config_identity(output_dir, config=config, dataset_path=dataset_path, backend_root=backend_root, **paths)
for changed_key, marker_field in (("config_path", "config_sha256"), ("synthetic_manifest_path", "synthetic_manifest_sha256"), ("prompt_path", "answer_prompt_sha256")):
changed_paths = dict(paths)
changed_file = tmp_path / f"changed-{changed_key}"
changed_file.write_text(paths[changed_key].read_text(encoding="utf-8") + "\n", encoding="utf-8")
changed_paths[changed_key] = changed_file
with pytest.raises(ValueError, match=f"different protocol artifacts.*{marker_field}"):
ensure_run_config_identity(output_dir, config=config, dataset_path=dataset_path, backend_root=backend_root, **changed_paths)
def test_verify_run_identity_is_read_only_and_names_the_changed_artifact(tmp_path: Path) -> None:
config = _load_config()
paths = {
"config_path": EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml",
"official_manifest_path": EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json",
"synthetic_manifest_path": EVAL_ROOT / "manifests" / "synthetic-corrections-pr4789-v1.json",
"prompt_path": EVAL_ROOT / "prompts" / "answer-v1.txt",
}
dataset_path = tmp_path / "dataset.json"
dataset_path.write_text("[]", encoding="utf-8")
output_dir = tmp_path / "run"
with pytest.raises(ValueError, match="missing"):
verify_run_identity(output_dir, dataset_path=dataset_path, **paths)
ensure_run_config_identity(output_dir, config=config, dataset_path=dataset_path, backend_root=Path(__file__).parents[1], **paths)
verify_run_identity(output_dir, dataset_path=dataset_path, **paths)
changed_prompt = tmp_path / "changed-prompt.txt"
changed_prompt.write_text(paths["prompt_path"].read_text(encoding="utf-8") + "\n", encoding="utf-8")
with pytest.raises(ValueError, match="refusing to grade.*|answer_prompt_sha256"):
verify_run_identity(output_dir, dataset_path=dataset_path, **{**paths, "prompt_path": changed_prompt})
def test_resume_revalidates_stored_rows_against_the_current_task(tmp_path: Path) -> None:
config = _load_config()
calls: list[httpx.Request] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(request)
return httpx.Response(200, json=_success_body())
tasks = _tasks(count=1)
with _mock_client(handler, config.qa) as client:
run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert len(calls) == 1
unchanged = run_answer_calls(tasks, config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (unchanged.reused, unchanged.called) == (1, 0)
assert len(calls) == 1
changed_message = replace(tasks[0], messages=(tasks[0].messages[0], {"role": "user", "content": "a different question"}))
after_message_change = run_answer_calls([changed_message], config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (after_message_change.reused, after_message_change.called) == (0, 1)
assert len(calls) == 2
changed_kept = replace(changed_message, kept_fact_ids=("fact-a",))
after_kept_change = run_answer_calls([changed_kept], config=config, client=client, output_dir=tmp_path, backoff_seconds=0)
assert (after_kept_change.reused, after_kept_change.called) == (0, 1)
assert len(calls) == 3
def test_cli_run_qa_fails_fast_without_provider_environment(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
from scripts.benchmark.deermem_eviction.cli import main
config = _load_config()
monkeypatch.delenv(config.qa.api_key_env, raising=False)
monkeypatch.delenv(config.qa.base_url_env, raising=False)
with pytest.raises(ProviderConfigurationError) as excinfo:
main(["run-qa", "--dataset", str(tmp_path / "missing.json"), "--output-dir", str(tmp_path / "out")])
assert config.qa.api_key_env in str(excinfo.value)

196
backend/tests/test_bench_deermem_eviction_report.py Normal file
View File

@ -0,0 +1,196 @@
from __future__ import annotations
import json
from pathlib import Path
import pytest
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.policy import PolicyResult
from scripts.benchmark.deermem_eviction.pool import PreparedCase
from scripts.benchmark.deermem_eviction.report import AnswerRowIntegrityError, collect_answer_rows, compute_qa_statistics, grade_answer_rows, summarize_qa_rows, write_qa_report
from scripts.benchmark.deermem_eviction.runner import response_path
from scripts.benchmark.deermem_eviction.stats import exact_mcnemar, paired_bootstrap_difference
EVAL_ROOT = Path(__file__).parents[1] / "scripts" / "benchmark" / "deermem_eviction"
def _load_config():
return load_evaluation_config(EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml")
def _case(case_id: str, *, source: str, scenario: str, answer: str) -> PreparedCase:
config = _load_config()
facts = [{"id": f"{case_id}-fact", "content": "Secret memory content.", "category": "context", "confidence": 0.9, "createdAt": "2026-02-14T00:00:00Z", "source": "synthetic"}]
return PreparedCase(
case_id=case_id,
source=source, # type: ignore[arg-type]
scenario=scenario, # type: ignore[arg-type]
question_type="synthetic-shape",
question="Secret question text?",
answer=answer,
question_date=None,
evaluation_time=config.evaluation_time,
facts=facts,
usage={},
support_fact_ids=(f"{case_id}-fact",),
)
def _policy_result(case: PreparedCase, policy: str) -> PolicyResult:
return PolicyResult(
case_id=case.case_id,
source=case.source,
scenario=case.scenario,
question_type=case.question_type,
policy=policy, # type: ignore[arg-type]
capacity=7,
support_fact_ids=case.support_fact_ids,
kept_fact_ids=case.support_fact_ids,
evicted=(),
scores={},
support_all_retained=True,
support_recall=1.0,
reserved_correction_slots=0,
)
def _write_response(output_dir: Path, case: PreparedCase, policy: str, prediction: str, *, capacity: int = 7) -> None:
row_id = f"{case.case_id}__{policy}"
path = response_path(output_dir, row_id)
path.parent.mkdir(parents=True, exist_ok=True)
row = {
"schema_version": 1,
"row_id": row_id,
"case_id": case.case_id,
"source": case.source,
"scenario": case.scenario,
"policy": policy,
"capacity": capacity,
"kept_fact_ids": list(case.support_fact_ids),
"prediction": prediction,
"attempts": 1,
"request_fingerprint": "f" * 64,
"response_model": "deepseek-v4-flash",
"usage": {"prompt_tokens": 10, "completion_tokens": 2},
"created_at": "2026-08-17T00:00:00Z",
}
path.write_text(json.dumps(row) + "\n", encoding="utf-8")
def _setup(tmp_path: Path):
official = _case("case-off", source="longmemeval", scenario="access_help", answer="Lyon-Reference-Secret")
synthetic = _case("case-syn", source="synthetic", scenario="correction_reserve", answer="NO")
cases = [official, synthetic]
_write_response(tmp_path, official, "confidence", "Paris")
_write_response(tmp_path, official, "hybrid-v1", "Lyon-Reference-Secret")
_write_response(tmp_path, synthetic, "confidence", "NO")
_write_response(tmp_path, synthetic, "hybrid-v1", "NO")
results = {f"{case.case_id}__{policy}": _policy_result(case, policy) for case in cases for policy in ("confidence", "hybrid-v1")}
fingerprints = {row_id: "f" * 64 for row_id in results}
return cases, results, fingerprints
def test_exact_mcnemar_matches_the_two_sided_exact_binomial() -> None:
concordant = exact_mcnemar([(True, True), (False, False)])
assert concordant.p_value == 1.0
one_sided_shift = exact_mcnemar([(False, True)] * 5)
assert one_sided_shift.only_second_correct == 5
assert one_sided_shift.p_value == pytest.approx(2 * 0.5**5)
mixed = exact_mcnemar([(False, True)] * 8 + [(True, False)] + [(True, True)] * 3)
assert (mixed.only_first_correct, mixed.only_second_correct, mixed.both_correct) == (1, 8, 3)
assert mixed.p_value == pytest.approx(2 * (0.5**9) * (1 + 9), rel=1e-12)
def test_paired_bootstrap_is_seed_deterministic_and_signed_second_minus_first() -> None:
config = _load_config()
pairs = [(False, True)] * 4 + [(True, True)] * 4
first = paired_bootstrap_difference(pairs, seed=config.statistics.bootstrap_seed, iterations=1000, alpha=config.statistics.alpha)
second = paired_bootstrap_difference(pairs, seed=config.statistics.bootstrap_seed, iterations=1000, alpha=config.statistics.alpha)
assert first == second
assert first.mean_difference == pytest.approx(0.5)
degenerate = paired_bootstrap_difference([(False, True)] * 3, seed=1, iterations=100, alpha=0.05)
assert (degenerate.mean_difference, degenerate.lower, degenerate.upper) == (1.0, 1.0, 1.0)
def test_grading_is_blind_and_joined_by_row_id(tmp_path: Path) -> None:
cases, results, fingerprints = _setup(tmp_path)
rows = collect_answer_rows(tmp_path, cases)
graded = grade_answer_rows(cases, results, rows, expected_fingerprints=fingerprints)
by_row = {row["row_id"]: row for row in graded}
assert not by_row["case-off__confidence"]["grade_correct"]
assert by_row["case-off__hybrid-v1"]["grade_correct"]
assert by_row["case-syn__confidence"]["grade_correct"]
assert by_row["case-syn__hybrid-v1"]["grade_correct"]
assert all(row["grader_version"] == "deterministic-overlap-v1" for row in graded)
def test_collect_and_integrity_checks_reject_incomplete_or_tampered_rows(tmp_path: Path) -> None:
cases, results, fingerprints = _setup(tmp_path)
response_path(tmp_path, "case-syn__hybrid-v1").unlink()
with pytest.raises(AnswerRowIntegrityError, match="case-syn__hybrid-v1"):
collect_answer_rows(tmp_path, cases)
_write_response(tmp_path, cases[1], "hybrid-v1", "NO", capacity=5)
rows = collect_answer_rows(tmp_path, cases)
with pytest.raises(AnswerRowIntegrityError, match="capacity/policy"):
grade_answer_rows(cases, results, rows, expected_fingerprints=fingerprints)
_write_response(tmp_path, cases[1], "hybrid-v1", "NO")
rows = collect_answer_rows(tmp_path, cases)
tampered = dict(results)
tampered["case-syn__hybrid-v1"] = _policy_result(_case("case-syn", source="synthetic", scenario="correction_reserve", answer="NO"), "hybrid-v1")
object.__setattr__(tampered["case-syn__hybrid-v1"], "kept_fact_ids", ("other-fact",))
with pytest.raises(AnswerRowIntegrityError, match="kept facts"):
grade_answer_rows(cases, tampered, rows, expected_fingerprints=fingerprints)
stale = dict(fingerprints)
stale["case-syn__hybrid-v1"] = "0" * 64
with pytest.raises(AnswerRowIntegrityError, match="request fingerprint"):
grade_answer_rows(cases, results, rows, expected_fingerprints=stale)
def test_grading_rejects_a_row_reassigned_to_another_valid_case(tmp_path: Path) -> None:
cases, results, fingerprints = _setup(tmp_path)
path = response_path(tmp_path, "case-off__hybrid-v1")
original = json.loads(path.read_text(encoding="utf-8"))
for field, value in (("case_id", "case-syn"), ("source", "synthetic"), ("scenario", "correction_reserve"), ("row_id", "case-syn__hybrid-v1")):
path.write_text(json.dumps(dict(original, **{field: value})) + "\n", encoding="utf-8")
rows = collect_answer_rows(tmp_path, cases)
with pytest.raises(AnswerRowIntegrityError, match="case-off__hybrid-v1.*case identity"):
grade_answer_rows(cases, results, rows, expected_fingerprints=fingerprints)
def test_summary_and_statistics_keep_suites_separate(tmp_path: Path) -> None:
cases, results, fingerprints = _setup(tmp_path)
config = _load_config()
graded = grade_answer_rows(cases, results, collect_answer_rows(tmp_path, cases), expected_fingerprints=fingerprints)
summary = summarize_qa_rows(graded)
assert {(group["source"], group["scenario"], group["policy"]): group["accuracy"] for group in summary} == {
("longmemeval", "access_help", "confidence"): 0.0,
("longmemeval", "access_help", "hybrid-v1"): 1.0,
("synthetic", "correction_reserve", "confidence"): 1.0,
("synthetic", "correction_reserve", "hybrid-v1"): 1.0,
}
statistics = compute_qa_statistics(graded, config)
assert statistics["suites"]["official"]["cases"] == 1
assert statistics["suites"]["synthetic"]["cases"] == 1
assert statistics["suites"]["overall"]["cases"] == 2
assert statistics["suites"]["official"]["mcnemar"]["only_second_correct"] == 1
assert statistics["suites"]["official"]["bootstrap"]["seed"] == config.statistics.bootstrap_seed
def test_written_report_redacts_dataset_text_and_refuses_overwrite(tmp_path: Path) -> None:
cases, results, fingerprints = _setup(tmp_path)
config = _load_config()
graded = grade_answer_rows(cases, results, collect_answer_rows(tmp_path, cases), expected_fingerprints=fingerprints)
write_qa_report(tmp_path, graded=graded, summary=summarize_qa_rows(graded), statistics=compute_qa_statistics(graded, config), config=config)
rows_text = (tmp_path / "qa.rows.jsonl").read_text(encoding="utf-8")
assert "Secret question text" not in rows_text
assert "Secret memory content" not in rows_text
confidence_official = json.loads(next(line for line in rows_text.splitlines() if '"case-off__confidence"' in line))
assert "Lyon-Reference-Secret" not in json.dumps(confidence_official)
assert (tmp_path / "qa.summary.json").exists()
assert (tmp_path / "qa.stats.json").exists()
with pytest.raises(FileExistsError):
write_qa_report(tmp_path, graded=graded, summary=summarize_qa_rows(graded), statistics=compute_qa_statistics(graded, config), config=config)

89
backend/tests/test_bench_deermem_eviction_results.py Normal file
View File

@ -0,0 +1,89 @@
from __future__ import annotations
import json
from datetime import UTC, datetime
from pathlib import Path
import pytest
from scripts.benchmark.deermem_eviction.config import load_evaluation_config
from scripts.benchmark.deermem_eviction.dataset import EvidenceRecord
from scripts.benchmark.deermem_eviction.policy import evaluate_case
from scripts.benchmark.deermem_eviction.pool import PreparedCase, build_case
from scripts.benchmark.deermem_eviction.results import summarize_policy_results, write_policy_run
BACKEND_ROOT = Path(__file__).parents[1]
EVAL_ROOT = BACKEND_ROOT / "scripts" / "benchmark" / "deermem_eviction"
def _case() -> PreparedCase:
support = EvidenceRecord("support", "knowledge-update", "private question", "private answer", None, "private support text")
distractors = [EvidenceRecord(f"d{i}", "single-session-user", "q", "a", None, f"private distractor {i}") for i in range(9)]
return build_case(
support=support,
distractors=distractors,
scenario="confirmation_help",
loss_rank=8,
evaluation_time=datetime(2026, 8, 13, tzinfo=UTC),
)
def test_summary_keeps_policy_and_scenario_groups_separate() -> None:
case = _case()
rows = [
evaluate_case(case, policy_name="confidence", capacity=7),
evaluate_case(case, policy_name="hybrid-v1", capacity=7),
]
summary = summarize_policy_results(rows)
assert [(item["policy"], item["support_all_retained"]) for item in summary] == [
("confidence", 0),
("hybrid-v1", 1),
]
def test_public_policy_run_omits_dataset_text_and_refuses_overwrite(tmp_path: Path) -> None:
config_path = EVAL_ROOT / "configs" / "pr4789-reproduction-v1.yaml"
official_manifest_path = EVAL_ROOT / "manifests" / "longmemeval-pr4789-v1.json"
synthetic_manifest_path = EVAL_ROOT / "manifests" / "synthetic-corrections-pr4789-v1.json"
prompt_path = EVAL_ROOT / "prompts" / "answer-v1.txt"
config = load_evaluation_config(config_path)
case = _case()
rows = [evaluate_case(case, policy_name="confidence", capacity=7), evaluate_case(case, policy_name="hybrid-v1", capacity=7)]
dataset_path = tmp_path / "private-dataset.json"
dataset_path.write_text("[]")
output_dir = tmp_path / "run"
write_policy_run(
output_dir,
results=rows,
config=config,
config_path=config_path,
official_manifest_path=official_manifest_path,
synthetic_manifest_path=synthetic_manifest_path,
prompt_path=prompt_path,
dataset_path=dataset_path,
backend_root=BACKEND_ROOT,
)
raw_text = (output_dir / "policy.raw.jsonl").read_text()
assert "private question" not in raw_text
assert "private answer" not in raw_text
assert "private support text" not in raw_text
assert len([json.loads(line) for line in raw_text.splitlines()]) == 2
assert json.loads((output_dir / "summary.json").read_text())["protocol_id"] == config.protocol_id
assert json.loads((output_dir / "run.json").read_text())["dataset"]["filename"] == config.dataset.filename
with pytest.raises(FileExistsError, match="refusing to overwrite"):
write_policy_run(
output_dir,
results=rows,
config=config,
config_path=config_path,
official_manifest_path=official_manifest_path,
synthetic_manifest_path=synthetic_manifest_path,
prompt_path=prompt_path,
dataset_path=dataset_path,
backend_root=BACKEND_ROOT,
)