From d700ad9edf6f3d03f87a0a33d528c8421d799d39 Mon Sep 17 00:00:00 2001 From: Zeren Wang <53075619+Vanzeren@users.noreply.github.com> Date: Sun, 27 Sep 2026 14:28:05 +0200 Subject: [PATCH] feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client - refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its configuration: one connection-precedence rule, credential fingerprint, wire-size counting and error taxonomy, shared by the guardrail gate and the memory hooks. - feat(memory): wire `memory.prescreen` and `memory.signal_classification` into DeerMem through one injected judge hook (both off by default, `enforce` gated by the shadow evaluation). - config: add the top-level `typesafe:` block and the two judging slots to `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync the Helm README example with it. - test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with per-stratum gates, censoring and the no-network baseline. Review follow-ups folded in: - `mutations_accepted` counts only facts that survive capacity enforcement and the upload-event scrub that precedes persistence, only removals/reinforcements of ids that existed, and persisted near-duplicate merges (matched by id, or by identity for legacy id-less facts); it is published only after persistence succeeds. - The cached manager hot-reloads its judge when a memory slot or the shared `typesafe` block changes (including a credential rotation); publication is serialized and revalidates the config generation, so a build started before an `enforce` -> `off` rollback cannot land after it. - Cache provenance is per answer: the `cached` flag and the reported model follow the answers a side actually consumed, across partial merges and answerless retries. - An oversized noul integer stays a per-question error instead of raising. - Incremental call savings use the shared scored denominator. - The sensitive-stratum gate requires evidence for each of identity, preference and correction. * fix(typesafe): resolve one credential per layer and refuse unusable connection values Review follow-ups on the shared TypeSafe client (#5906): - A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the shared `typesafe:` block: the first layer that sets either credential setting decides it, and inside one layer the literal key wins (the standalone semantics). A consumer whose endpoint does not accept the block's key no longer sends it on every call, and the missing-credential message keeps naming the variable that layer configured. - `base_url` is validated where it is configured: `http(s)` only, with no query, fragment, embedded credentials, whitespace or control characters, and a malformed port fails at construction instead of on every request (httpx raises `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been swallowed into the appended endpoint path). - A credential that cannot travel as a header value — surrounding whitespace from a secret mounted from a file, a non-printable character — is refused when the connection resolves. It used to construct fine and then put the whole `Bearer ` value into h11's `LocalProtocolError` message, which the guardrail middleware's `logger.exception` prints. - The client raises transport failures without chaining the original exception, so that message can no longer carry the credential through the cause chain; the exception type stays in the message for diagnosis. - `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config` path: `max_attempts: true` was silently read as `1`. An integer stays a usable float, so `timeout: 5` keeps working. Tests: 20 cases across `tests/test_typesafe_config.py` and `tests/test_typesafe_client.py`, each red on the previous revision and green now. 367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean. Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`, `config.example.yaml`. * fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence Second review round on #5906: the remaining items from the review thread and from the follow-up pass. Typesafe client (the reply is treated as hostile input too): - Send `Accept-Encoding: identity`, refuse a body that carries a non-identity `Content-Encoding`, and read through a bounded `iter_bytes` that stops past `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be decoded in full and, on the gate path, allowed the tool call unexamined; it is now `invalid_response` (a denial with `fail_closed: true`). - Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting escaped the module's taxonomy, so the memory consumers lost the round's record instead of recording a fallback. Memory judge: - A request-level failure is recorded as `request_failed`, never `no_verdict` (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable endpoint was indistinguishable from a provider with no usable answer). The adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record and log; `no_verdict` stays for a question-level failure, and an ineligible side keeps its own reason even when the round's request failed. Shadow evaluation (the enablement gate reads the evidence, not one number): - The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to meet the 1% target; a sample that only meets the point estimate is `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound). - New `baseline_incremental_savings` gate: a baseline that saves as many calls as the model — the shipped all-trivial sample — fails instead of enabling. - Negative counts (`mutations_accepted`, message / batch / token counts) are treated as absent rather than scored: `-3` read as "nothing lost", adding a phantom saving to the numerator and the denominator. Docs: the design documents cited but absent from the repository are replaced with in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`, `GUARDRAILS.md`), and the gate and limit wording now matches what the code enforces. Tests: 31 new or updated cases, each red on the previous revision (compressed body refused even when it decodes to a valid envelope, body over the cap refused, body at the cap answered, deep nesting mapped, `request_failed` on each request-level failure, the bound and baseline gates, the negative counter). 454 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` and `ruff format --check` are clean. * fix(memory-eval): pair the baseline comparison on the same records Review follow-up on the incremental-savings gate added in ed930478 (P2): both sides of the comparison are now the records the no-network baseline can score (`trivial_only` plus a readable signals list). Subtracting that subset's heuristic savings from model savings over *all* scorable skips credited the model with batches the baseline never saw: 300 reviewed skips with a single annotated record computed 300-1=299 "incremental", passed the gate and set `enable_recommended`, while the paired comparison was model=1 against heuristic=1 (incremental 0). - `Evidence` carries the paired numbers (`baseline_paired_skips`, `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one heuristic count, filled by `build_report` from the baseline summary. - The gate reports `INSUFFICIENT` when no scored skip carries the annotation and when the annotation covers only a subset of the scored skips (the message carries the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive paired gain, and `PASS` only on a positive gain over the full paired set. - Docs state the pairing/coverage requirement in `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring. Tests: the reviewer's counterexample as a report-level regression (it failed with `assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true), a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields. 344 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` / `ruff format --check` clean. * fix(typesafe): bound the sync body read by the deadline `deadline_seconds` only covered the sync path's headers, the completed body and the parse, so an endpoint that drips bytes just under the read timeout held the caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget. The read loop now checks the budget before every transport read, which bounds the overshoot to one read. The pieces are the transport's own reads, not re-chunked output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk, so a check at that granularity would never see a drip. `tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24 pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now (~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track the new granularity. * fix(memory): read the YAML off spelling, and veto a skip over the whole batch Two review findings on the memory judging path. `memory.prescreen.mode` / `memory.signal_classification.mode` rejected the `mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1 parses the unquoted value as a boolean, so both fields saw `False`. The failed reload left the previous judge running, so both fields now normalize `False` to `off` before the `Literal` runs. `True` stays a validation error (no mode it could mean). The examples quote the value, and the docs record why. The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch it answers for is the whole post-watermark feed, so an explicit correction in an earlier turn of an eight-message batch did not stop the skip that then advanced the watermark past it. `detect_signals` gains a `window` parameter (default unchanged) and the updater scans the full feed for the veto; the extraction hint and the reinforcement gate keep the window. Both regressions are red on the previous head: the config test reproduced the `input_value=False` rejection through `AppConfig.from_file`, and the batch test reproduced the prescreen judging (and skipping) a batch whose first turn carries the correction. --- backend/docs/GUARDRAILS.md | 59 +- backend/docs/MEMORY_IMPROVEMENTS.md | 68 + .../harness/deerflow/agents/memory/AGENTS.md | 132 ++ .../memory/backends/deermem/deer_mem.py | 12 +- .../memory/backends/deermem/deermem/config.py | 11 + .../deermem/core/message_processing.py | 22 +- .../backends/deermem/deermem/core/queue.py | 5 + .../backends/deermem/deermem/core/updater.py | 191 ++- .../harness/deerflow/agents/memory/judging.py | 132 ++ .../harness/deerflow/agents/memory/manager.py | 165 +- .../agents/memory/prescreen/__init__.py | 38 + .../agents/memory/prescreen/contract.py | 140 ++ .../agents/memory/prescreen/typesafe.py | 185 ++ .../agents/memory/signals/__init__.py | 61 + .../agents/memory/signals/contract.py | 158 ++ .../agents/memory/signals/coordinator.py | 600 +++++++ .../agents/memory/signals/typesafe.py | 248 +++ .../harness/deerflow/config/app_config.py | 6 + .../harness/deerflow/config/memory_config.py | 86 +- .../deerflow/config/typesafe_config.py | 67 + .../harness/deerflow/guardrails/typesafe.py | 420 ++--- .../harness/deerflow/typesafe/AGENTS.md | 52 + .../harness/deerflow/typesafe/__init__.py | 118 ++ .../harness/deerflow/typesafe/client.py | 556 ++++++ .../harness/deerflow/typesafe/connection.py | 279 ++++ .../harness/deerflow/typesafe/errors.py | 58 + .../harness/deerflow/typesafe/validation.py | 91 + backend/scripts/eval_memory_prescreen.py | 1089 ++++++++++++ backend/tests/test_agent_guidance_check.py | 1 + backend/tests/test_app_config_reload.py | 50 + backend/tests/test_memory_prescreen.py | 1487 +++++++++++++++++ .../test_memory_prescreen_eval_script.py | 624 +++++++ backend/tests/test_memory_queue.py | 9 +- backend/tests/test_typesafe_client.py | 446 +++++ backend/tests/test_typesafe_config.py | 318 ++++ .../tests/test_typesafe_guardrail_provider.py | 16 + config.example.yaml | 94 +- deploy/helm/deer-flow/README.md | 2 +- deploy/helm/deer-flow/values.yaml | 2 +- .../deerflow-extension-jev-classify/README.md | 9 + .../deerflow-extension-jev-context/README.md | 11 + 41 files changed, 7793 insertions(+), 325 deletions(-) create mode 100644 backend/packages/harness/deerflow/agents/memory/judging.py create mode 100644 backend/packages/harness/deerflow/agents/memory/prescreen/__init__.py create mode 100644 backend/packages/harness/deerflow/agents/memory/prescreen/contract.py create mode 100644 backend/packages/harness/deerflow/agents/memory/prescreen/typesafe.py create mode 100644 backend/packages/harness/deerflow/agents/memory/signals/__init__.py create mode 100644 backend/packages/harness/deerflow/agents/memory/signals/contract.py create mode 100644 backend/packages/harness/deerflow/agents/memory/signals/coordinator.py create mode 100644 backend/packages/harness/deerflow/agents/memory/signals/typesafe.py create mode 100644 backend/packages/harness/deerflow/config/typesafe_config.py create mode 100644 backend/packages/harness/deerflow/typesafe/AGENTS.md create mode 100644 backend/packages/harness/deerflow/typesafe/__init__.py create mode 100644 backend/packages/harness/deerflow/typesafe/client.py create mode 100644 backend/packages/harness/deerflow/typesafe/connection.py create mode 100644 backend/packages/harness/deerflow/typesafe/errors.py create mode 100644 backend/packages/harness/deerflow/typesafe/validation.py create mode 100644 backend/scripts/eval_memory_prescreen.py create mode 100644 backend/tests/test_memory_prescreen.py create mode 100644 backend/tests/test_memory_prescreen_eval_script.py create mode 100644 backend/tests/test_typesafe_client.py create mode 100644 backend/tests/test_typesafe_config.py diff --git a/backend/docs/GUARDRAILS.md b/backend/docs/GUARDRAILS.md index 8bf0d93f96..48a1be8d75 100644 --- a/backend/docs/GUARDRAILS.md +++ b/backend/docs/GUARDRAILS.md @@ -468,11 +468,35 @@ guardrails: | `max_state_chars` | `4000` | Argument text above this is denied locally. | | `cache_size` / `cache_ttl_seconds` | `256` / `300` | FIFO cache keyed by `(tool, arguments)`; `0` disables it. | +The eight connection settings (`api_key` / `api_key_env` / `base_url` / `model` / `timeout` / `deadline_seconds` / `max_attempts` / `retry_backoff`) may instead live once in a top-level `typesafe:` block and be shared by every TypeSafe consumer; a value written here still wins (see "Shared client" below). The two credential settings are resolved per *layer*: the first layer that sets either one decides, and inside a layer a literal `api_key` beats that layer's own `api_key_env` — so a consumer configured with `api_key_env` keeps reading its own variable even when the block also carries a literal key. A connection value that no request could use is refused where it is configured: a `base_url` that is not `http(s)`, or that carries a query, fragment or embedded credentials, and a credential that cannot travel as a header value (surrounding whitespace from a mounted secret, a non-printable character) all fail at construction instead of on every call — the last one used to put the whole `Bearer ` value into h11's error message, which reaches the guardrail's exception log. The block itself is validated in pydantic's strict mode, so `max_attempts: true` is an error rather than a silent `1`. + +**Shared client.** The transport half of this provider — authentication, client lifecycle, retry and backoff, the deadline budget, response parsing and the error taxonomy, UTF-8 wire-size counting — lives in `packages/harness/deerflow/typesafe/` and is shared with the other host-side TypeSafe consumers (memory pre-screening and signal classification, planned). Everything that makes this a *gate* stays in the provider: the state it builds, its question and rubric, its threshold and direction, its local preflight, its cache, and the rule that an error denies. The same rule holds in the other direction: nothing in the shared client knows what a "risky tool call" is, and a change that would move a failure policy, a threshold or a cache into it is a design change, not a refactor. + +**Effective configuration.** Connection settings resolve with the precedence consumer `config` > top-level `typesafe:` > built-in defaults, so an existing `guardrails.provider.config` keeps working unchanged while a deployment with several TypeSafe consumers shares one block: + +```yaml +typesafe: # optional; only the connection/model/timeout defaults + api_key_env: TYPESAFE_API_KEY + model: jev-latest + timeout: 5.0 + +guardrails: + enabled: true + fail_closed: true + provider: + use: deerflow.guardrails.typesafe:TypeSafeGuardrailProvider + config: + threshold: 0.5 # consumer policy stays here + tools: ["bash"] +``` + +A question-level failure is reported per question by the shared parser and mapped back to this provider's own error by the adapter, so the gate's behavior is unchanged: a response whose envelope is usable but whose answer for `risky_tool_call` is missing or malformed still raises `TypeSafeGuardrailError` (`invalid_response`) and, with `fail_closed: true`, still denies. `max_state_chars` remains a **character** count of the canonical argument JSON — the shared client can report a payload's UTF-8 byte size for consumers that need it, but it changes no limit here, because one CJK character is three bytes and a byte limit would move this fallback boundary. + **Allowed tools (`allowed_tools`).** A hard permission list, not an exemption from evaluation. Omitted, no list is enforced (the probe scope decides what is evaluated). Set to `[]`, no tool may run. Set to a list, only those tools may run — a call to any other tool is refused locally (`typesafe.tool_not_allowed`) with no state built, no request and no cache entry, and the refusal is a guardrail decision, so `fail_closed: false` cannot reopen it. A listed tool is **still** probed and still denied when its risk probability reaches `threshold`; the list answers "may this tool run at all", the risk gate answers "is this particular call safe". It is checked before `tools`, so a tool outside the list is refused even when `tools` would have skipped probing it — an unlisted tool must not inherit an allow from being out of probe scope. The provider has no denylist: express deny rules and per-role limits in `authorization.*`. **What is sent.** The tool name plus the call's full argument JSON, under `state.tool_call`. The provider does **not** redact: arguments can carry user content, file paths, shell commands or secrets, and `pii_redaction_middleware` does not apply on this path. Narrow `tools` to the tools that can cause side effects, and clear the egress with your data-protection owner before enabling. -**Limits are enforced locally.** Arguments that cannot be serialised as strict JSON (bytes, non-string keys, `NaN`/`Infinity`, lone surrogates that UTF-8 cannot encode) and argument text above `max_state_chars` are **denied without a request** (`typesafe.state_unusable`). A local refusal is a guardrail decision, so `fail_closed: false` does not turn it into an unevaluated tool run. Nothing is truncated and sent -- a prefix could hide the dangerous half of a `write_file` payload. The trade-off is real: long heredocs, inline scripts and large `write_file` bodies can trip the limit, and those refusals count toward the deployment's false-positive rate. Raising `max_state_chars` increases the data leaving the process. +**Limits are enforced locally.** Arguments that cannot be serialised as strict JSON (bytes, non-string keys, `NaN`/`Infinity`, lone surrogates that UTF-8 cannot encode) and argument text above `max_state_chars` are **denied without a request** (`typesafe.state_unusable`). A local refusal is a guardrail decision, so `fail_closed: false` does not turn it into an unevaluated tool run. Nothing is truncated and sent -- a prefix could hide the dangerous half of a `write_file` payload. The trade-off is real: long heredocs, inline scripts and large `write_file` bodies can trip the limit, and those refusals count toward the deployment's false-positive rate. Raising `max_state_chars` increases the data leaving the process. The *reply* is bounded in the same spirit: the client asks for an identity encoding, refuses a body that carries a non-identity `Content-Encoding`, and stops past 64 KiB (`typesafe.client.MAX_RESPONSE_BYTES`), so a compressed or oversized response is a failed evaluation — `invalid_response`, and with `fail_closed: true` a denial — rather than an unexamined allow. **Deadlines.** The async path cancels an in-flight request through `asyncio.timeout`, which bounds the request duration but not the wall-clock cost of cleanup. The synchronous path cannot preempt a blocking call: it checks the deadline after the response headers arrive, around the body read, and before the decision is accepted, and drops results that arrived late. **No total return-time bound is promised on the sync path.** @@ -631,6 +655,7 @@ guardrails: cd backend uv run python -m pytest tests/test_guardrail_middleware.py -v uv run python -m pytest tests/test_typesafe_guardrail_provider.py -v +uv run python -m pytest tests/test_typesafe_client.py tests/test_typesafe_config.py -v uv run python -m pytest tests/blocking_io/test_typesafe_guardrail_provider.py -v ``` @@ -654,6 +679,24 @@ uv run python -m pytest tests/blocking_io/test_typesafe_guardrail_provider.py -v `tests/blocking_io/test_typesafe_guardrail_provider.py` -- anchor that drives `aevaluate` against a real loopback HTTP server, with a meta-check proving the sync path on the loop trips the Blockbuster gate. +`tests/test_typesafe_client.py` / `tests/test_typesafe_config.py` -- the shared client the provider is now built on: +- Per-question partial success: a valid answer survives an invalid or missing one, each failure category is reported + separately, an answer for a question that was never asked is ignored, and a missing *envelope* still fails the whole request +- The gate's own question-level failure still surfaces as `TypeSafeGuardrailError` / `invalid_response` +- `wire_size` matches what httpx sends, counts UTF-8 bytes, and leaves the gate's `max_state_chars` a character count +- Precedence consumer `config` > top-level `typesafe` > built-in defaults, and `mode: off` resolving no credential at all +- One credential source per layer: a consumer's `api_key_env` is not overridden by a literal `api_key` in the block, + while a layer that sets both uses its own literal key +- Unusable connection values fail at construction: non-`http(s)` / query / fragment / embedded-credential `base_url` + forms, a malformed port, and a credential that cannot be a header value (never echoed in the message) +- Strict types for the `typesafe:` block: `max_attempts: true` is rejected instead of silently becoming `1`, while an + integer stays a usable float (`timeout: 5`) +- A transport failure raises without chaining the original error, so a rejected header cannot print the credential +- The reply is bounded and never decoded: a compressed body is refused even when it decodes to a valid envelope, a body + over `MAX_RESPONSE_BYTES` is refused, one exactly at the cap is answered, and deep nesting is a request-level failure +- `sharing_key` (internal: credential fingerprint, connection settings, input limit, transport factory) versus + `release_policy_parameters()` (public: behaviour, never the credential) + ## Files ``` @@ -662,17 +705,27 @@ packages/harness/deerflow/guardrails/ provider.py # GuardrailProvider protocol, GuardrailRequest, GuardrailDecision middleware.py # GuardrailMiddleware (AgentMiddleware subclass) builtin.py # AllowlistProvider (zero deps) - typesafe.py # TypeSafeGuardrailProvider (System One HTTP risk gate) + typesafe.py # TypeSafeGuardrailProvider (state, question, threshold, cache, failure policy) + +packages/harness/deerflow/typesafe/ + __init__.py # Public exports + the shared/not-shared boundary + client.py # TypeSafeClient: request, retry, deadline, response parsing, wire_size + connection.py # Effective connection: precedence, credential fingerprint, sharing_key inputs + errors.py # One error taxonomy (transport / http_status / invalid_response / deadline) + validation.py # Eager config-value validation shared with consumers packages/harness/deerflow/config/ guardrails_config.py # GuardrailsConfig Pydantic model + singleton + typesafe_config.py # Top-level `typesafe:` defaults + singleton packages/harness/deerflow/agents/middlewares/ tool_error_handling_middleware.py # Registers GuardrailMiddleware in chain -config.example.yaml # Four provider options documented +config.example.yaml # Four provider options + the top-level `typesafe:` defaults tests/test_guardrail_middleware.py # 25 tests tests/test_typesafe_guardrail_provider.py # TypeSafe provider + middleware integration +tests/test_typesafe_client.py # Shared client: partial success, wire size +tests/test_typesafe_config.py # Precedence, mode off, the two identities tests/blocking_io/test_typesafe_guardrail_provider.py # Async path must stay off the loop docs/GUARDRAILS.md # This file ``` diff --git a/backend/docs/MEMORY_IMPROVEMENTS.md b/backend/docs/MEMORY_IMPROVEMENTS.md index 0b098c1501..5fd55f8344 100644 --- a/backend/docs/MEMORY_IMPROVEMENTS.md +++ b/backend/docs/MEMORY_IMPROVEMENTS.md @@ -34,6 +34,74 @@ Token counting: - Falls back to a network-free CJK-aware character estimate if tokenizer import or encoding load fails (CJK characters count as ~2 chars/token, other characters as ~4 chars/token) +## Extraction cost gate (opt-in, off by default) + +`memory.prescreen` decides whether a batch of new conversation is worth paying +for the extraction LLM call; `memory.signal_classification` adds model hint +labels (and, in one narrow case, vetoes a skip). Both are off by default, and +`off` resolves no class path, constructs no provider and validates no credential. + +Modes: + +| Setting | Modes | Effect | +|---|---|---| +| `memory.prescreen.mode` | `off` / `shadow` / `enforce` | `shadow` records the verdict and extracts as usual; `enforce` lets a low-probability batch skip the call **and advance the watermark** (that batch is then consumed as "nothing durable here") | +| `memory.signal_classification.mode` | `off` / `shadow` / `hints` | `hints` merges `reinforcement` / `correction` labels into the extraction hint text and, only while the pre-screen is `enforce`, vetoes a skip | +| `memory.signal_classification.combine` | `auto` / `always` / `never` | Whether the two sides share one request. `auto` shares only when every effective client setting matches (model, base URL, credential fingerprint, timeouts, retries, `max_state_chars`, cache settings); `always` fails at construction on a mismatch; `never` always splits | + +Both `mode` values are written quoted in `config.example.yaml` (`mode: "off"`). +Unquoted `off` is still accepted — YAML parses it as a boolean and both fields +normalize that back to `off` — but a quoted value says what an operator means. + +Failure direction is always extraction: an error, timeout, unusable response, +missing verdict, provider exception, or an over-limit batch extracts as usual. +Nothing is truncated to fit a limit. A skip changes **no** post-extraction gate +(confidence, scope, durability, authority, dedup, capacity eviction) and is not a +safety boundary. + +Batches that are never judged: no judge configured; the emergency +(pre-summarization) flush; the shutdown drain (`flush_sync`, whose budget belongs +to persistence); batches with a **deterministic** signal anywhere in the batch (a +positive signal outranks a model negative — the veto scans the whole batch, since +a skip consumes all of it, while the extraction hint keeps `detect_signals`' own +6-message window); and any batch while +`staleness_review_enabled` (default **true**) or `consolidation_enabled` is on, +because a skip would also skip that batch's maintenance review. With the shipped +defaults the pre-screen therefore sees no batches until `staleness_review_enabled` +is turned off — do that deliberately, with the evaluation below. + +**Egress.** The formatted conversation text (user plus final assistant turns, which +may contain file paths or credentials) is sent to the TypeSafe API: a second +destination beyond the deployment's own extraction model. Existing memory, fact +ids and tool arguments are never sent. `pii_redaction_middleware` does not cover +this path. `mode: "off"` is the rollback — no request, no behavior change, no data +migration; the unquoted `mode: off` a hand-edit may type is accepted too. + +**Before `enforce`.** Run the shadow evaluation and publish its gates: +`miss_rate` (a skip on a batch whose extraction would have been accepted) at or +below 1% **and** its one-sided 95% upper bound at or below that target — a sample +that only meets the point estimate is reported `INSUFFICIENT`, so the answer is +more skips rather than a smaller number to quote — zero misses on the identity / +preference / correction strata, at least 200 reviewed skips whose manual review +confirms none was worth remembering, and the recorded savings evidence — calls and +tokens saved, their p50/p95 verdict latency, and the no-network heuristic baseline +on the same data, with a positive incremental saving over that baseline measured on +the same records — the comparison must cover the scored skips, so a run that +annotates only a subset is reported `INSUFFICIENT` rather than crediting the model +for batches the heuristic never saw. If the heuristic already captures the savings +(it saves as many calls as the model), the third-party call is not justified and the +gate fails. `scripts/eval_memory_prescreen.py` reads +shadow records and reports those populations and gates, and reports a gate that +lacks evidence as `INSUFFICIENT` rather than passing it. `skip_threshold` (0.2) +and `hint_threshold` (0.5) are provisional and unmeasured. + +**Online `hints` is a different gate.** The signal-classification design §6 +requires its own independently human-reviewed dataset, a pre-registered maximum +allowed increase δ with a confidence upper bound and per-stratum sample sizes, +and a `veto_recovered_facts` benefit reading built on human-confirmed recovered +facts. The pre-screen evaluation above measures none of that, so its gates +approve `enforce` only; `hints` stays `off`/`shadow` until that evidence exists. + ## Known Gap Previous versions of this document described TF-IDF/context-aware retrieval as if it were already shipped. diff --git a/backend/packages/harness/deerflow/agents/memory/AGENTS.md b/backend/packages/harness/deerflow/agents/memory/AGENTS.md index a772b774c3..1bee3974b2 100644 --- a/backend/packages/harness/deerflow/agents/memory/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/memory/AGENTS.md @@ -368,3 +368,135 @@ runs by default. fact dicts are read-only inputs. Legacy fact normalization in DeerMem and `frontend/src/core/memory/import-memory.ts` uses neutral confidence `0.5` for missing or invalid values, clamps finite confidence to `[0, 1]`, trims content, and defaults blank or missing sources to `unknown`. Keep these compatibility defaults aligned. + +#### Extraction cost gate and model hints (opt-in) + +`memory.prescreen` and `memory.signal_classification` are host slots wired into +DeerMem through one injected `judge` hook (`judge(context) -> verdict`; a plain +mapping in, so the vendored backend imports no host judging types, exactly like +`extraction_callback`). Both default to `off`, and `off` resolves no class path, +constructs nothing and validates no credentials, so an unconfigured deployment +behaves exactly as before. Each `mode` accepts the quoted spelling +(`mode: "off"`) and the unquoted one a hand-edit tends to type (`mode: off`, which +YAML parses as a boolean and both fields normalize back to `off`) — a rollback to +off must load rather than fail the reload and leave the previous judge running. + +- The pre-screen is a **cost gate, never a safety boundary**. It gates no + execution and no write; it only decides whether the extraction call is worth + paying for. Every failure direction is extraction: error, timeout, unusable + response, missing verdict, provider exception, or an over-limit batch extract + as usual. A judge failure is logged and treated as "no opinion" — judging can + never lose a memory, and it can never block one either. +- **`max_state_chars` is a character count of the judged text, per side, and + nothing is truncated to fit.** The judged text is exactly + `format_conversation_for_update`'s output (including its own head/tail + retention for long single messages); over a side's limit that side falls back, + and each side enforces its own limit. The shared client counts UTF-8 bytes + (`wire_size`) but replaces no consumer's limit (shared-client design §2.4): a + byte count would trip the fallback three times early on CJK text. +- **A judging-config edit is hot-reloaded onto the cached manager.** The manager + singleton (and its judge) is built once, but `memory.prescreen` / + `memory.signal_classification` are documented as hot-reloadable + (config/AGENTS.md "Config Hot-Reload Boundary"). `get_memory_manager()` rebuilds + and re-injects the judge through `MemoryManager.refresh_judge` when either + slot's config changes, so `enforce` → `off` stops skipping extraction and + sending conversation text to the provider, and `off` → `shadow` starts + recording, without a restart. The invalidating signature covers the shared + top-level `typesafe:` connection block and each enabled side's *resolved* + connection identity (including the credential's fingerprint), so a judged side + that inherits its endpoint, model, credential, or deadlines from that block + picks up an edit there, and a rotated credential invalidates the judge, without + a slot edit. A config that cannot build a judge is logged and leaves the + previous judge in place, recorded so the rebuild is not retried on every call. + Publication is serialized on the manager lock and **revalidates the config + generation** before installing: a judge built just before a rollback + (`enforce` → `off`) is dropped rather than landing after it, so queued batches + cannot keep judging against a superseded mode. +- **The combined request's cache is consulted before eligibility narrows the + question set, and it is keyed by the *full* logical question set** (signal + classification design §2.2.3 / §2.2.6). A round in which only one side is still + eligible — L3, L8, or an over-limit side — reuses the answers already bucketed + for that digest instead of re-asking them, so `mode` and phase-A eligibility + never change what a cache hit already answers. +- **A partially answered bucket is not a hit.** Entries hold a per-question + answer mapping, and consumption is per question, never per side (S18): an + answer missing or malformed in a response falls back for that question this + round and is asked again on the next request with the same digest, while the + other direction in the same response is cached and consumed as usual. Coverage + is attributed per answer, not per round: a side's `cached` flag (and the model + it reports) reflects the answers it actually consumed, so a verdict reused from + the bucket stays a cache hit even when the round fetched the other side's + questions — otherwise the shadow evaluation counts it as a second network + sample and inflates the confidence-bound denominator. The bucket also keeps the + served model **per answer**, so filling it with a newer model does not + re-attribute an answer it already held: a later full cache hit still audits each + verdict under the model that produced it, and a retry that validates no answer + reports the cached answer's model rather than the empty response's. +- **An enabled side always leaves a record, including when it cannot judge.** + The judge emits the `prescreen` / `signal_classification` payload with a + `fallback_reason` for every round in which a side is enabled, so the + pre-screening design §5 "local fallbacks" population (L3 / L4 / L5 / L7 / L8 or + a failure fallback) is countable from the records. A round with no record at + all now means no side was enabled (or the drain path forbade judging) — not + "every enabled side was ineligible". +- **A failed request is `request_failed`, never `no_verdict`.** The two are + separate populations: an unreachable endpoint (transport, non-200, unusable or + unbounded body, deadline) must not read as "the provider answered with nothing + usable". The adapters' `decide()` therefore *propagates* `TypeSafeError` instead of + swallowing it, the coordinator records the reason and logs a warning, and the + updater still extracts as usual. `no_verdict` is reserved for a question-level + failure — the envelope arrived, this side's answer did not. Only a side that + actually asked can report `request_failed`; an ineligible one keeps its own reason + (L3 / L5 / L7 / L8) even when the round's request failed. +- **Only `enforce` + `skip` changes persistence:** it drops the extraction call + and **advances the watermark**, consuming the batch as "nothing durable here", + and emits a `prescreen` record with the digest. `shadow` records and extracts as + usual; the watermark advance on extraction success is unchanged. +- Batches excluded from judging, in this order: no judge injected (L1), + `bypass_watermark` (emergency flush), the shutdown drain (`judge=False`), + non-empty **deterministic** `detect_signals` anywhere in the batch (L3 — the + whole post-watermark feed is scanned for this veto, because a skip consumes all + of it; the extraction hint keeps `detect_signals`' default 6-message window), and + `staleness_review_enabled` / `consolidation_enabled` (L8 — a skip would also + skip that batch's maintenance review). Note the shipped default + `staleness_review_enabled: true` therefore makes the pre-screen inert until an + operator turns it off; that is the intended conservative Phase 1 policy. +- Model hints are **additive**: the hint text takes the union of deterministic + signals and model labels, but the reinforcement evidence gate reads the + **deterministic set only**, so a model verdict can never write a confirmation. + A model hint may veto a pre-screen skip (only `prescreen enforce` × + `classifier hints`); it can never cause one. Weakening hints only add text — + they never drive deletion or demotion. +- **Online `hints` is a separate gate from the pre-screen evaluation below.** The + signal-classification design §6 requires its own independently human-reviewed + dataset, a pre-registered δ with per-stratum sample sizes for + non-inferiority/superiority, and a `veto_recovered_facts` benefit reading. + `eval_memory_prescreen.py` measures none of those and does not stand in for that + dataset, so passing its gates approves `enforce` only; `hints` stays off or + `shadow` until the §6 evidence exists. +- `mutations_accepted` is counted at the real apply sites (accepted new facts, + accepted removals, accepted confirmations, accepted consolidations) and is the + definition of "worth remembering": a batch whose only effect was a summary + rewrite counts zero, so a skip that dropped it is not a missed memory. A new + fact counts only if it survives capacity enforcement (and any consolidation + that consumes it as a source); a near-duplicate merge counts only when it + actually raised the target's confidence (a no-op restatement changes nothing) + and the target survives (matched by id, or by identity for legacy id-less + facts); a removal or reinforcement counts only for an + id that actually existed — a proposal that passes its gate but changes no + persistent fact must not register as a lost memory. Scoring runs against the + scrubbed fact set (upload-event facts are dropped on the way to persistence), so + a proposal the scrub removes is not counted either. The counter is published + only after persistence succeeds: a failed apply (rejected commit, storage + error) omits it, so the evaluation censors that record as a missing sample + rather than reading it as a lost memory. +- **`enforce` stays gated by the shadow evaluation, and the evaluation's gates + read evidence, not counts alone.** `scripts/eval_memory_prescreen.py` needs + ≥200 reviewed skips *that the review confirms were none worth remembering* (a + reviewed skip found worth remembering fails gate 3), and recorded savings + evidence — saved calls, their tokens, p50/p95 verdict latency, and the + mandatory no-network baseline on the same records. Missing any of that is + `INSUFFICIENT`, never a pass; the exit code is the verdict. +- Signal classification is not admission control and not a write path: it never + participates in queue admission, never alters `ConversationContext.signals` or + the union merge, and neither hook touches the journal or the Gateway API. diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py index 150ddbea46..63bc9f17ad 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py @@ -180,7 +180,7 @@ class DeerMem(MemoryManager): ``model_post_init`` (shared with direct construction). """ config_dict = dict(backend_config or {}) - for key in ("should_keep_hidden_message", "trace_context_manager", "extraction_callback"): + for key in ("should_keep_hidden_message", "trace_context_manager", "extraction_callback", "judge"): if key not in config_dict and key in host_hooks: config_dict[key] = host_hooks[key] if "host_llm" not in config_dict: @@ -200,6 +200,16 @@ class DeerMem(MemoryManager): instance.backend_config = dict(backend_config or {}) return instance + def refresh_judge(self, judge: Any) -> None: + """Replace the injected memory judge after a host config hot-reload. + + ``self._config`` is shared with the updater and the update queue, so + mutating the parsed config reaches every judging call site without + rebuilding storage, the LLM, or the queue. ``judge=None`` (both sides + now off) disables judging on the next batch. + """ + self._config.judge = judge + # ── Write ──────────────────────────────────────────────────────────── def add( self, diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py index eaa3e6d1ad..193675f2da 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py @@ -293,6 +293,17 @@ class DeerMemConfig(BaseModel): "post-invoke observability. Set programmatically (not from YAML)." ), ) + # ── Memory judge (pre-screen + signal classification) ──────────────── + judge: Any = Field( + default=None, + description=( + "Optional host-injected memory judge ``judge(context) -> MemoryBatchVerdict``: " + "decides whether this batch is worth an extraction call (pre-screening) and " + "supplies model hint labels (signal classification). None (default) = no judging, " + "leaving the extraction path byte-identical to a deployment without this feature. " + "Set programmatically by the host factory (not from YAML)." + ), + ) # ── Watermark cache (in-memory, bounded LRU) ───────────────────────── watermark_max_keys: int = Field( default=4096, diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_processing.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_processing.py index 46a133c164..ce296b7c78 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_processing.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_processing.py @@ -262,20 +262,32 @@ SIGNAL_NAMES: tuple[str, ...] = ( "decision", ) +#: Trailing messages :func:`detect_signals` scans by default. This is the "recent +#: turns" window the extraction hints and the reinforcement gate are about; a caller +#: with a different question -- the pre-screen's L3 veto, which has to answer for a +#: whole batch -- passes ``window=None``. +_SIGNAL_WINDOW = 6 + def detect_signals( messages: list[Any], *, patterns_dir: str | None = None, + window: int | None = _SIGNAL_WINDOW, ) -> set[str]: """Detect signal classes in the recent conversation turns. - Returns the set of signal names whose patterns match a human message among - the last 6 filtered messages. This generalizes :func:`detect_correction` / - :func:`detect_reinforcement` (which remain for backward compatibility) to - the full signal set. The window stays ``messages[-6:]``. + Returns the set of signal names whose patterns match a human message among the + last ``window`` messages (the default 6, the "recent turns" window the + extraction hints and the reinforcement gate are about). ``window=None`` scans + every message instead: the pre-screen's L3 veto needs that, because a skip + consumes the whole post-watermark batch, so an explicit signal anywhere in that + batch must keep it out of judging -- not only one inside the trailing window. + This generalizes :func:`detect_correction` / :func:`detect_reinforcement` (which + remain for backward compatibility) to the full signal set. """ - recent_user_msgs = [msg for msg in messages[-6:] if getattr(msg, "type", None) == "human"] + candidates = messages if window is None else messages[-window:] + recent_user_msgs = [msg for msg in candidates if getattr(msg, "type", None) == "human"] if not recent_user_msgs: return set() diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/queue.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/queue.py index 9532b302fc..49dc8d9719 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/queue.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/queue.py @@ -267,6 +267,11 @@ class MemoryUpdateQueue: user_id=context.user_id, trace_id=context.trace_id, bypass_watermark=context.bypass_watermark, + # The shutdown drain never pre-screens (design L7): its + # budget belongs to persistence, and a judge request would + # spend part of a bounded shutdown window on a cost + # optimization instead of on saving queued work. + judge=not skip_inter_item_delay, ) if success: succeeded += 1 diff --git a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py index a62c92c9b7..0ea7481891 100644 --- a/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py +++ b/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py @@ -1337,14 +1337,21 @@ class MemoryUpdater: agent_name: str | None, signals: frozenset[str], user_id: str | None = None, + conversation_text: str | None = None, ) -> tuple[dict[str, Any], list[Any]] | None: - """Load memory and build the update prompt for a conversation.""" + """Load memory and build the update prompt for a conversation. + + ``conversation_text`` lets the caller hand over the formatted text it + already produced for the memory judge, so the conversation is formatted + once per batch instead of twice. + """ config = self._config if not messages: return None current_memory = self.get_memory_data(agent_name, user_id=user_id) - conversation_text = format_conversation_for_update(messages) + if conversation_text is None: + conversation_text = format_conversation_for_update(messages) if not conversation_text.strip(): return None @@ -1438,6 +1445,50 @@ class MemoryUpdater: except Exception: logger.warning("extraction_callback raised; ignoring", exc_info=True) + def _judge_batch( + self, + hook: Any, + *, + batch_text: str, + signals: frozenset[str], + thread_id: str | None, + agent_name: str | None, + user_id: str | None, + trace_id: str | None, + bypass_watermark: bool, + message_count: int, + ) -> Any: + """Ask an injected memory judge about this batch; ``None`` when unjudged. + + The hook is a plain callable over a mapping -- the same convention as + ``extraction_callback``'s payload -- so DeerMem stays host-agnostic (it + imports no host judging types) while the host owns what a verdict means. + + A judge failure is logged and treated as "no opinion": pre-screening must + never lose a memory and signal classification must never block one, so + this path can only ever fall back to today's behaviour (design L2 / S2). + """ + if hook is None: + return None + try: + return hook( + { + "batch_text": batch_text, + "signals": signals, + "staleness_review_enabled": bool(self._config.staleness_review_enabled), + "consolidation_enabled": bool(self._config.consolidation_enabled), + "bypass_watermark": bypass_watermark, + "thread_id": thread_id, + "user_id": user_id, + "agent_name": agent_name, + "trace_id": trace_id, + "message_count": message_count, + } + ) + except Exception: + logger.warning("Memory judge failed; extracting as usual", exc_info=True) + return None + def _finalize_update( self, current_memory: dict[str, Any], @@ -1476,7 +1527,6 @@ class MemoryUpdater: user_id=user_id, capacity_decisions=capacity_decisions, ) - updated_memory = _strip_upload_mentions_from_memory(updated_memory) current_by_id = {str(fact.get("id")): fact for fact in current_memory.get("facts", [])} updated_by_id = {str(fact.get("id")): fact for fact in updated_memory.get("facts", [])} change_set = { @@ -1525,7 +1575,6 @@ class MemoryUpdater: user_id=user_id, capacity_decisions=capacity_decisions, ) - updated_memory = _strip_upload_mentions_from_memory(updated_memory) saved = self._storage.save( updated_memory, agent_name, @@ -1552,6 +1601,7 @@ class MemoryUpdater: trace_id: str | None = None, *, bypass_watermark: bool = False, + judge: bool = True, ) -> bool: """Update memory asynchronously by delegating to the sync path. @@ -1570,6 +1620,7 @@ class MemoryUpdater: user_id=user_id, trace_id=trace_id, bypass_watermark=bypass_watermark, + judge=judge, ) def _do_update_memory_sync( @@ -1582,6 +1633,7 @@ class MemoryUpdater: trace_id: str | None = None, *, bypass_watermark: bool = False, + judge: bool = True, ) -> bool: """Pure-sync memory update; bind ``trace_id`` into the request-trace ContextVar for the worker thread, then delegate to the impl. @@ -1605,6 +1657,7 @@ class MemoryUpdater: user_id=user_id, trace_id=trace_id, bypass_watermark=bypass_watermark, + judge=judge, ) return self._do_update_memory_sync_impl( messages=messages, @@ -1614,6 +1667,7 @@ class MemoryUpdater: user_id=user_id, trace_id=trace_id, bypass_watermark=bypass_watermark, + judge=judge, ) def _watermark_get(self, key: tuple[str | None, str | None, str | None]) -> tuple[str, ...] | None: @@ -1677,6 +1731,7 @@ class MemoryUpdater: trace_id: str | None = None, *, bypass_watermark: bool = False, + judge: bool = True, ) -> bool: """Pure-sync memory update using ``model.invoke()``. @@ -1719,11 +1774,55 @@ class MemoryUpdater: # served their purpose (backpressure admission at enqueue); the hint # is a soft nudge and must not point at turns the watermark excluded. feed_signals = detect_signals(feed_messages, patterns_dir=self._config.patterns_dir) + # The judge's L3 veto reads the whole batch, not the hint window: a skip + # consumes every turn fed here (it advances the watermark), so an explicit + # signal anywhere in the batch has to keep that batch out of judging. The + # extraction hint above keeps the default 6-message window -- a hint points + # at recent turns, a veto answers for the batch. + batch_signals = detect_signals(feed_messages, patterns_dir=self._config.patterns_dir, window=None) + batch_text = format_conversation_for_update(feed_messages) + # The memory judge (when the host injected one) runs here: still off the + # turn path, before any LLM call, on exactly the text the extractor would + # see -- no second truncation. ``judge=False`` (the shutdown drain) and a + # ``None`` hook both leave the batch unjudged. + verdict = self._judge_batch( + self._config.judge if judge else None, + batch_text=batch_text, + signals=frozenset(batch_signals), + thread_id=thread_id, + agent_name=agent_name, + user_id=user_id, + trace_id=trace_id, + bypass_watermark=bypass_watermark, + message_count=len(feed_messages), + ) + verdict_payload = getattr(verdict, "payload", None) + if isinstance(verdict_payload, dict): + metrics.update({key: value for key, value in verdict_payload.items() if value is not None}) + if getattr(verdict, "skip", False): + # The pre-screen decided this batch is not worth a call: consume it + # and advance the watermark (nothing else runs -- no prompt, no + # invoke, no apply). Advancing is what makes the saving real; leaving + # the watermark put would re-judge a longer batch every turn and the + # digest-keyed cache could never hit. + if not bypass_watermark: + self._watermark_set(watermark_key, _message_identity(messages[-1])) + success = True + # The judge record is this batch's only trace, so emit it even though + # no LLM call happened (the finally-block gate reads this flag). + attempted = True + logger.info("Memory extraction skipped by the pre-screen (thread=%s)", thread_id) + return True + model_hints = getattr(verdict, "hints", None) prepared = self._prepare_update_prompt( messages=feed_messages, agent_name=agent_name, - signals=feed_signals, + # Hint text takes the union (deterministic + model hints); the + # reinforcement evidence gate below still reads the deterministic + # set only, so a model verdict can never confirm a fact (S4). + signals=frozenset(feed_signals) | (model_hints if isinstance(model_hints, frozenset) else frozenset()), user_id=user_id, + conversation_text=batch_text, ) if prepared is None: return False @@ -1805,6 +1904,14 @@ class MemoryUpdater: # pre-attempt early returns (no new messages, empty conversation, no # model) do not emit, matching the prior behavior. if attempted: + if not success: + # A failed extraction persisted nothing, so the apply-site + # counter must not be published. The evaluation script scores a + # present counter even on ``success=False`` (it is treated as an + # outcome, not a censored failure), which would register a + # rejected commit as a missed memory. Absent + failed is censored + # -- the honest reading (persistent-mutation contract). + metrics.pop("mutations_accepted", None) self._emit_extraction_metrics( metrics, thread_id=thread_id, @@ -1856,6 +1963,7 @@ class MemoryUpdater: trace_id: str | None = None, *, bypass_watermark: bool = False, + judge: bool = True, ) -> bool: """Synchronously update memory using the sync LLM path. @@ -1864,6 +1972,10 @@ class MemoryUpdater: the lead agent. This eliminates the cross-loop connection-reuse bug described in issue #2615. + ``judge=False`` forbids this batch from being judged by the injected + memory judge (the shutdown-drain path sets it: the drain budget belongs + to persistence, not to a cost optimization). + When called from within a running event loop (e.g. from a LangGraph node), the blocking sync call is offloaded to a thread pool so the caller's loop is not blocked. @@ -1898,6 +2010,7 @@ class MemoryUpdater: user_id=user_id, trace_id=trace_id, bypass_watermark=bypass_watermark, + judge=judge, ) return future.result() except Exception: @@ -1912,6 +2025,7 @@ class MemoryUpdater: user_id=user_id, trace_id=trace_id, bypass_watermark=bypass_watermark, + judge=judge, ) def _apply_updates( @@ -1941,6 +2055,13 @@ class MemoryUpdater: """ config = self._config now = utc_now_iso_z() + # Mutations that actually landed, counted at their real apply sites. New + # facts are counted only after they survive capacity enforcement (and any + # consolidation pass), not at the append. This is the pre-screen's "worth + # remembering" signal: a batch whose only effect was a summary rewrite + # counts zero, so a skip that dropped it is not a missed memory + # (pre-screening design §5). + mutations_accepted = 0 scope_gate_rejections: dict[str, dict[str, int]] = { "facts": {"missing": 0, "scope": 0, "durability": 0, "authority": 0}, "summaries": {"missing": 0, "scope": 0, "authority": 0}, @@ -1963,6 +2084,11 @@ class MemoryUpdater: for entry in update_data.get("factsToReinforce", []) if isinstance(entry, dict) and entry.get("scope") == "user" and isinstance(entry.get("reason"), str) and entry["reason"].strip() and isinstance(entry.get("id"), str) } + # Only an id that actually exists can be confirmed. A nonexistent + # target changes nothing, so counting it would report an accepted + # mutation that never landed (the pre-screen evaluation reads the + # counter as "worth remembering"). + reinforced_ids &= {fact.get("id") for fact in current_memory.get("facts", []) if isinstance(fact, dict)} if reinforced_ids: current_memory["facts"] = [ { @@ -1974,6 +2100,7 @@ class MemoryUpdater: else fact for fact in current_memory.get("facts", []) ] + mutations_accepted += len(reinforced_ids) # Update user sections user_updates = update_data.get("user", {}) @@ -2119,6 +2246,18 @@ class MemoryUpdater: # persisted-fact count. passed_threshold = 0 replacement_fact_keys: dict[int, str] = {} + # Fact IDs appended by this pass. They count as accepted mutations only + # if they survive to the final fact set (see the metrics block): capacity + # enforcement can evict one immediately, and a later consolidation can + # consume one as a source. + added_fact_ids: list[str] = [] + # Existing fact IDs a near-duplicate merge actually changed (confidence + # raised). Counted under the same survival rule: a no-op merge (proposed + # confidence not higher) changed nothing and must not count. + changed_merge_ids: set[str] = set() + # Legacy id-less facts cannot be matched by id, so they are tracked by + # object identity; the target dicts stay alive in the memory being edited. + changed_merge_facts: dict[int, dict[str, Any]] = {} # A correction must keep its proposed content, not inherit an older # paraphrase. Also avoid strengthening any proposed removal target, # including stale targets retained by candidate guards or the cap. @@ -2175,6 +2314,14 @@ class MemoryUpdater: if confidence > existing_confidence: merge_target["confidence"] = confidence merge_target["source"] = thread_id or "unknown" + # This merge persisted a change to an existing fact, so it + # is a mutation outcome too -- but only once we know the + # target survives (the count below checks the final set). + merge_target_id = merge_target.get("id") + if isinstance(merge_target_id, str): + changed_merge_ids.add(merge_target_id) + else: + changed_merge_facts[id(merge_target)] = merge_target if metrics is not None: metrics["facts_merged_dedup"] = metrics.get("facts_merged_dedup", 0) + 1 # New proposals have no durable ID yet. Log their batch @@ -2211,6 +2358,7 @@ class MemoryUpdater: fact_entry["expected_valid_days"] = min(evd, creation_cap) current_memory["facts"].append(fact_entry) existing_fact_keys.add(fact_key) + added_fact_ids.append(fact_entry["id"]) # Enforce one capacity policy across automatic, manual, and import # writes. Usage comes from a separate sidecar, so scoring never rewrites @@ -2252,7 +2400,14 @@ class MemoryUpdater: fact_ids_to_remove.add(fact_id) if fact_ids_to_remove: - current_memory["facts"] = [fact for fact in current_memory.get("facts", []) if fact.get("id") not in fact_ids_to_remove] + # Count only targets that actually existed: a scope-valid removal of + # a nonexistent id deletes nothing, so counting it would report an + # accepted mutation that never landed (the pre-screen evaluation + # reads the counter as "worth remembering"). + landed_removals = fact_ids_to_remove & {fact.get("id") for fact in current_memory.get("facts", []) if isinstance(fact, dict)} + if landed_removals: + current_memory["facts"] = [fact for fact in current_memory.get("facts", []) if fact.get("id") not in landed_removals] + mutations_accepted += len(landed_removals) # ── Memory consolidation ── # Runs after the max_facts trim so source facts that were just evicted @@ -2427,6 +2582,29 @@ class MemoryUpdater: if ids_consumed: current_memory["facts"] = [f for f in current_memory.get("facts", []) if f.get("id") not in ids_consumed] current_memory["facts"].extend(new_consolidated) + mutations_accepted += merge_count + + # Finalization scrubs upload-event facts (session-scoped) from the memory it + # persists; apply that here, before scoring survival, so a fact accepted by + # the apply but dropped by the scrub is not counted as a landed mutation + # (the shadow evaluation reads a positive count on a skip as a lost memory). + current_memory = _strip_upload_mentions_from_memory(current_memory) + + # Count new facts and changed near-duplicate targets only once they survive + # capacity enforcement. A proposed fact can pass the confidence gate and be + # appended, then be immediately evicted by _select_for_capacity; a merge + # target can be evicted the same way. Counting either would report an + # accepted mutation even though the persistent facts are unchanged, and the + # pre-screen evaluation reads any positive mutations_accepted on a skip as a + # lost memory (memory/AGENTS.md, "worth remembering"). Scoring against the + # final fact set also excludes a fact a later consolidation consumed as a + # source, whose merge is counted separately. + final_fact_ids = {fact.get("id") for fact in current_memory.get("facts", []) if isinstance(fact, dict)} + mutations_accepted += sum(1 for fact_id in added_fact_ids if fact_id in final_fact_ids) + mutations_accepted += sum(1 for fact_id in changed_merge_ids if fact_id in final_fact_ids) + if changed_merge_facts: + surviving_objects = {id(fact) for fact in current_memory.get("facts", []) if isinstance(fact, dict)} + mutations_accepted += sum(1 for object_id in changed_merge_facts if object_id in surviving_objects) if metrics is not None: metrics["facts_passed_confidence"] = passed_threshold @@ -2434,5 +2612,6 @@ class MemoryUpdater: metrics["facts_passed_scope_gate"] = len(new_facts) - sum(scope_gate_rejections["facts"].values()) metrics["rejected_by_scope_gate"] = sum(count for reasons in scope_gate_rejections.values() for count in reasons.values()) metrics["scope_gate_rejections"] = scope_gate_rejections + metrics["mutations_accepted"] = mutations_accepted return current_memory diff --git a/backend/packages/harness/deerflow/agents/memory/judging.py b/backend/packages/harness/deerflow/agents/memory/judging.py new file mode 100644 index 0000000000..a85456097d --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/judging.py @@ -0,0 +1,132 @@ +"""Shared pieces for the memory judging hooks: pre-screening and signal classification. + +Both hooks judge **the same batch text**, build **the same wire state** and need +**the same kind of answer cache**, so those three things live here instead of +three times over (two adapters plus the coordinator). + +What deliberately does **not** live here: the questions, rubrics, thresholds, +decision direction, modes, and failure policy. Pre-screening skips a call when the +text looks worthless (failure ⇒ extract as usual); signal classification only adds +hint text (failure ⇒ fall back to the deterministic signals). Those are adapter +policy, documented in: + +* the pre-screen: ``backend/docs/MEMORY_IMPROVEMENTS.md``, ``prescreen/contract.py`` +* the classifier: ``backend/docs/MEMORY_IMPROVEMENTS.md``, ``signals/contract.py`` +""" + +from __future__ import annotations + +import hashlib +import time +from collections import OrderedDict +from collections.abc import Iterable, Mapping +from dataclasses import dataclass + +from deerflow.typesafe.client import Answer + +_DIGEST_CHARS = 16 + + +def batch_digest(batch_text: str) -> str: + """The batch's identity: its formatted text hashed. Cache key and audit identifier.""" + return hashlib.sha256(batch_text.encode("utf-8")).hexdigest()[:_DIGEST_CHARS] + + +def conversation_tail_state(batch_text: str) -> dict[str, dict[str, str]]: + """The wire state both hooks send: the formatted batch, and nothing else. + + Neither hook sends existing memory, fact ids, tool arguments, or the signals — + "is this worth keeping / does this endorse something" is judged from the text + alone (pre-screening design L10, signal-classification design S8). + """ + return {"conversation_tail": {"text": batch_text}} + + +def batch_chars(batch_text: str) -> int: + """The judged batch's length, for a side's ``max_state_chars`` limit. + + A **character** count of the formatted text, never its UTF-8 byte count: the + shared client's ``wire_size`` reports bytes and replaces no consumer's limit + (shared-client design §2.4). Counting bytes would move the fallback boundary + — one CJK character is three of them (pre-screening design L5 / S16). + """ + return len(batch_text) + + +@dataclass(frozen=True) +class CachedVerdict: + """A judged state's validated answers, each tagged with the model that served it. + + ``models`` keeps provenance per answer rather than one model for the whole + bucket: a later partial response merges into the bucket without re-attributing + an earlier answer (served by a different model) to the new one, so a subsequent + full cache hit still reports the model that actually produced the verdict. + """ + + answers: Mapping[str, Answer] + models: Mapping[str, str] + + def model_for(self, question_ids: Iterable[str]) -> str: + """The one model that served every *answered* ``question_ids`` entry, or "" when mixed. + + Only ids with a recorded model count: a question the bucket never answered + is not consumed evidence, so a partially answered side attributes to the + model that actually served the answers it used. + """ + served = {self.models[question_id] for question_id in question_ids if question_id in self.models} + return served.pop() if len(served) == 1 else "" + + +class AnswerCache: + """FIFO + monotonic-TTL cache of validated answers, one entry per judged state. + + Cache semantics are the memory layer's own, never the shared transport's + (pre-screening design L11, signal-classification design §2.2.6): + + * only **validated** answers are written, so a missing or malformed answer is a + miss next time rather than a remembered failure; + * an entry's ``answers`` is a per-question mapping, so a partial response is a + partial entry: the consumer must subtract what the entry holds from the + questions it wants and re-ask the difference, never read a partial entry as a + full hit (signal-classification design §2.2.6 / S18); + * a whole-request failure writes nothing; + * ``size <= 0`` or ``ttl_seconds <= 0`` disables caching. + + ``key`` is opaque to this class; callers choose the namespace (a single side + uses the digest, the coordinator adds its configuration fingerprint and the + logical question set). + """ + + def __init__(self, *, size: int, ttl_seconds: float) -> None: + self._size = size + self._ttl_seconds = ttl_seconds + self._entries: OrderedDict[object, tuple[CachedVerdict, float]] = OrderedDict() + + @property + def enabled(self) -> bool: + return self._size > 0 and self._ttl_seconds > 0 + + def get(self, key: object) -> CachedVerdict | None: + if not self.enabled: + return None + entry = self._entries.get(key) + if entry is None: + return None + verdict, expires_at = entry + if expires_at <= time.monotonic(): + # pop(): the memory queue's Timer thread and an executor-thread flush + # can reach the same expired key, and a bare ``del`` would raise where + # this must simply be a miss. + self._entries.pop(key, None) + return None + return verdict + + def put(self, key: object, verdict: CachedVerdict) -> None: + if not self.enabled: + return + self._entries[key] = (verdict, time.monotonic() + self._ttl_seconds) + while len(self._entries) > self._size: + self._entries.popitem(last=False) + + +__all__ = ["AnswerCache", "CachedVerdict", "batch_chars", "batch_digest", "conversation_tail_state"] diff --git a/backend/packages/harness/deerflow/agents/memory/manager.py b/backend/packages/harness/deerflow/agents/memory/manager.py index 1efb38a52e..09ce744b68 100644 --- a/backend/packages/harness/deerflow/agents/memory/manager.py +++ b/backend/packages/harness/deerflow/agents/memory/manager.py @@ -18,6 +18,7 @@ from __future__ import annotations import importlib import inspect +import json import logging import os import sys @@ -44,6 +45,12 @@ _MANAGER_CLASS_ATTR = "MANAGER_CLASS" _memory_manager: MemoryManager | None = None _backends_cache: dict[str, type[MemoryManager]] | None = None _manager_lock = threading.Lock() +# Signature of the host judging config the cached manager's judge was built +# from. ``memory.*`` is documented as hot-reloadable, but the manager singleton +# (and its injected judge) is constructed once; this lets get_memory_manager() +# rebuild only the judge when memory.prescreen / memory.signal_classification +# changes. None = no manager built yet (or reset). +_memory_judge_signature: str | None = None def context_query_kwargs(get_context: Callable[..., str], query: str | None) -> dict[str, str | None]: @@ -533,6 +540,16 @@ class MemoryManager(BaseModel): """Turn-start nudge (future background review). Default: no-op.""" return None + def refresh_judge(self, judge: Any) -> None: + """Replace the injected memory judge after a judging-config hot-reload. + + Called by :func:`get_memory_manager` when ``memory.prescreen`` or + ``memory.signal_classification`` changes in ``config.yaml``, so toggling + a mode takes effect without restarting the process. Default: no-op (a + backend without a judge ignores it). + """ + return None + # ── Async (speculative) ────────────────────────────────────────────── # Interface placeholders so a future async LLM client can override without # changing the contract. Defaults delegate to the sync methods (no @@ -912,6 +929,34 @@ def _host_default_extraction_callback(payload: Any) -> None: fact_scope_rejected / extracted * 100, thread_id, ) + prescreen = payload.get("prescreen") + if isinstance(prescreen, dict): + # The pre-screen record is the only trace a *skip* leaves (no LLM call + # runs), so it carries the digest and the fallback reason rather than a + # probability for every batch. + logger.info( + "Memory pre-screen: thread=%s mode=%s verdict=%s p=%s cached=%s digest=%s fallback=%s", + thread_id, + prescreen.get("mode"), + prescreen.get("verdict"), + prescreen.get("probability"), + prescreen.get("cached"), + prescreen.get("digest"), + prescreen.get("fallback_reason"), + ) + signal_classification = payload.get("signal_classification") + if isinstance(signal_classification, dict): + logger.info( + "Memory signal classification: thread=%s mode=%s labels=%s cached=%s digest=%s fallback=%s", + thread_id, + signal_classification.get("mode"), + signal_classification.get("labels"), + signal_classification.get("cached"), + signal_classification.get("digest"), + signal_classification.get("fallback_reason"), + ) + if payload.get("skip_vetoed_by_model_signal"): + logger.info("Memory pre-screen skip vetoed by a model signal (thread=%s)", thread_id) def _collect_host_hooks() -> dict[str, Any]: @@ -924,6 +969,13 @@ def _collect_host_hooks() -> dict[str, Any]: builds the host default model when it actually needs one (i.e. has no model of its own) -- building an unused default on every startup would waste time. The others are direct values (cheap function refs). + + ``judge`` is built here from the host memory config (pre-screening and signal + classification) and is ``None`` when both sides are off, which leaves the + extraction path byte-identical to a deployment without the feature. A + configured-but-unusable judge fails *here* (agent build time) rather than + silently degrading on the first batch: it can decide whether a call happens, + so a half-configured one must not run. """ from deerflow.trace_context import ensure_trace_context @@ -933,9 +985,108 @@ def _collect_host_hooks() -> dict[str, Any]: "trace_context_manager": ensure_trace_context, "host_llm_factory": _host_default_llm, "extraction_callback": _host_default_extraction_callback, + "judge": _host_default_judge(), } +def _host_default_judge() -> Any: + """Build the memory judge from the host memory config (``None`` when every side is off).""" + from deerflow.agents.memory.signals.coordinator import build_memory_judge + + return build_memory_judge() + + +def _judging_config_signature(cfg: Any) -> str: + """Stable signature of the host judging config, for change detection. + + Covers both slots in full (mode, ``use``, and provider ``config``), the + shared top-level ``typesafe:`` block, and each enabled side's *resolved* + connection identity. A side that does not override a connection field + (endpoint, model, credential, deadlines) inherits it from that block, so + editing the block must invalidate the cached judge; the resolved identity + also catches a credential rotated through its environment variable, which + the block's own fields do not show. + """ + from deerflow.typesafe.connection import typesafe_defaults + + return json.dumps( + { + "prescreen": cfg.prescreen.model_dump(mode="json"), + "signal_classification": cfg.signal_classification.model_dump(mode="json"), + "typesafe": typesafe_defaults(), + "effective_connections": _effective_connection_signature(cfg), + }, + sort_keys=True, + ) + + +def _effective_connection_signature(cfg: Any) -> dict[str, Any]: + """Each judging side's resolved connection identity, keyed by slot. + + Uses the same ``resolve_connection_for_mode`` the providers use, so the + signature reflects the *effective* settings (slot override > shared block > + built-in defaults > environment credential) rather than config text alone. + It records the credential's fingerprint, never the key. A side that is off + resolves to ``None``; a side whose config cannot resolve is marked + ``unresolved`` instead of raising, so this signature can never break + ``get_memory_manager()`` — the rebuild that follows surfaces the same error. + """ + from deerflow.agents.memory.prescreen.contract import CONFIGURATION_SOURCE as PRESCREEN_SOURCE + from deerflow.agents.memory.signals.contract import CONFIGURATION_SOURCE as SIGNAL_SOURCE + from deerflow.typesafe.connection import resolve_connection_for_mode, typesafe_defaults + + defaults = typesafe_defaults() + signature: dict[str, Any] = {} + for name, slot, source in ( + ("prescreen", cfg.prescreen, PRESCREEN_SOURCE), + ("signal_classification", cfg.signal_classification, SIGNAL_SOURCE), + ): + try: + connection = resolve_connection_for_mode(mode=slot.mode, settings=slot.config, defaults=defaults, configuration_source=source) + except Exception: + signature[name] = {"unresolved": True} + continue + signature[name] = None if connection is None else {**connection.public_parameters(), "credential": connection.credential_fingerprint()} + return signature + + +def _refresh_judge_for_reloaded_config(manager: MemoryManager) -> None: + """Rebuild and re-inject the judge when the host judging config changed. + + The manager singleton (and its judge) is built once, yet ``memory.*`` is + documented as hot-reloadable and a mode toggle must take effect: + ``enforce`` -> ``off`` must stop skipping extraction and sending the + conversation to the provider, and ``off`` -> ``shadow`` must start + recording. Rebuild here so storage / queue / LLM dependencies stay intact + and only the judging hook is swapped. + """ + global _memory_judge_signature + signature = _judging_config_signature(get_memory_config()) + if signature == _memory_judge_signature: + return + try: + judge = _host_default_judge() + except Exception: + # A transiently broken judging config must not fail every memory call. + # Keep the last-good judge, record the signature so the rebuild is not + # retried until the config is edited again, and surface it loudly. + logger.error("Failed to rebuild the memory judge from the reloaded config; keeping the previous judge", exc_info=True) + with _manager_lock: + _memory_judge_signature = signature + return + # Publish under the manager lock, revalidating the config generation first: + # another thread may have rolled the config back (e.g. enforce -> off) while + # this judge was being built, and installing it then would resurrect the + # superseded mode for already-queued batches. If the generation moved, drop + # this build and let the next lookup rebuild for the current config. + with _manager_lock: + if _judging_config_signature(get_memory_config()) != signature: + return + manager.refresh_judge(judge) + _memory_judge_signature = signature + logger.info("Memory judge refreshed after a memory judging-config change") + + # ── Singleton factory ───────────────────────────────────────────────────── def get_memory_manager() -> MemoryManager: """Return the singleton :class:`MemoryManager` for the active config. @@ -945,8 +1096,12 @@ def get_memory_manager() -> MemoryManager: :func:`reset_memory_manager` to force re-resolution (tests / runtime backend switching). """ - global _memory_manager + global _memory_manager, _memory_judge_signature if _memory_manager is not None: + # The manager is built once; refresh only the judge so a hot-reloaded + # memory.prescreen / memory.signal_classification takes effect (a mode + # switch must not keep a stale judge deciding or a stale skip persisting). + _refresh_judge_for_reloaded_config(_memory_manager) return _memory_manager # deer-flow is multi-threaded: memory injection runs via asyncio.to_thread, @@ -957,6 +1112,10 @@ def get_memory_manager() -> MemoryManager: # connections) constructed here in __init__. with _manager_lock: if _memory_manager is not None: + # Another thread built the singleton while we waited for the lock; it + # installed the matching judge signature, and the lock-free fast path + # above is what handles later config edits. (Refreshing here would + # re-enter _manager_lock.) return _memory_manager cfg = get_memory_config() @@ -990,6 +1149,7 @@ def get_memory_manager() -> MemoryManager: # ``mode`` mirrors host MemoryConfig.mode so the invariant validator # (mode=="tool" requires search) can fire on the factory path too. _memory_manager = cls.from_config(backend_config, mode=cfg.mode, **host_hooks) + _memory_judge_signature = _judging_config_signature(cfg) logger.info("Memory manager resolved: %s (manager_class=%r)", cls.__name__, manager_class) return _memory_manager @@ -1027,7 +1187,8 @@ def reset_memory_manager() -> None: The next :func:`get_memory_manager` call re-reads the config and re-scans backends. Use this in tests or when switching backends at runtime. """ - global _memory_manager, _backends_cache + global _memory_manager, _backends_cache, _memory_judge_signature with _manager_lock: _memory_manager = None _backends_cache = None + _memory_judge_signature = None diff --git a/backend/packages/harness/deerflow/agents/memory/prescreen/__init__.py b/backend/packages/harness/deerflow/agents/memory/prescreen/__init__.py new file mode 100644 index 0000000000..512693ae72 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/prescreen/__init__.py @@ -0,0 +1,38 @@ +"""Memory capture pre-screening: a pluggable, off-by-default cost gate. + +The hook lives in the host (this package + the DeerMem updater that calls it), +not in an extension: the extension API's only memory touchpoint is a read-only +post-hoc observer and its contributions are fail-open, which is the wrong shape +for a switch that can affect writes (design §2.6). + +See ``backend/docs/MEMORY_IMPROVEMENTS.md`` for the modes and the gates that +must pass before ``enforce``, and ``agents/memory/AGENTS.md`` for the invariants. +""" + +from deerflow.agents.memory.prescreen.contract import ( + CONFIGURATION_SOURCE, + MODE_ENFORCE, + MODE_OFF, + MODE_SHADOW, + MODES, + VERDICT_EXTRACT, + VERDICT_SKIP, + MemoryPrescreenDecision, + MemoryPrescreenProvider, + MemoryPrescreenRequest, + resolve_memory_prescreen, +) + +__all__ = [ + "CONFIGURATION_SOURCE", + "MODE_ENFORCE", + "MODE_OFF", + "MODE_SHADOW", + "MODES", + "VERDICT_EXTRACT", + "VERDICT_SKIP", + "MemoryPrescreenDecision", + "MemoryPrescreenProvider", + "MemoryPrescreenRequest", + "resolve_memory_prescreen", +] diff --git a/backend/packages/harness/deerflow/agents/memory/prescreen/contract.py b/backend/packages/harness/deerflow/agents/memory/prescreen/contract.py new file mode 100644 index 0000000000..7c3872e207 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/prescreen/contract.py @@ -0,0 +1,140 @@ +"""Contract for the memory capture pre-screen: a cost gate, never a safety gate. + +The pre-screen answers one question — "is this batch worth paying for an +extraction call?" — and nothing else. Its failure direction is therefore +**extraction**: an error, a deadline, an unusable response, an over-limit batch, or +no verdict at all means "extract as usual" (design L2). It gates no execution and +no write, and it never replaces the post-extraction confidence / scope / +durability / authority gates (L6). + +Host contract, consumed by the DeerMem updater through the memory-layer judge: + +* :class:`MemoryPrescreenRequest` — what the host knows about the batch; +* :class:`MemoryPrescreenDecision` — the verdict, or ``None`` for "no opinion"; +* :class:`MemoryPrescreenProvider` — duck-typed, resolved by class path. + +The operator-facing modes and the pre-``enforce`` gates are documented in +``backend/docs/MEMORY_IMPROVEMENTS.md``; the module invariants are in +``agents/memory/AGENTS.md``. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Literal, Protocol, runtime_checkable + +from deerflow.reflection import resolve_variable + +MODE_OFF = "off" +MODE_SHADOW = "shadow" +MODE_ENFORCE = "enforce" +MODES: tuple[str, ...] = (MODE_OFF, MODE_SHADOW, MODE_ENFORCE) + +VERDICT_EXTRACT: Literal["extract"] = "extract" +VERDICT_SKIP: Literal["skip"] = "skip" + +CONFIGURATION_SOURCE = "memory.prescreen.config" + + +@dataclass(frozen=True) +class MemoryPrescreenRequest: + """One batch, as the host sees it. + + ``batch_text`` is exactly what the extractor would be sent + (``format_conversation_for_update`` output, including its existing head/tail + retention for long single messages) — the judging side introduces no second + truncation (L5). Existing memory, tool-call arguments and dropped messages are + never part of the request (L10). + """ + + batch_text: str + digest: str + signals: frozenset[str] = frozenset() + thread_id: str | None = None + user_id: str | None = None + agent_name: str | None = None + trace_id: str | None = None + bypass_watermark: bool = False + message_count: int = 0 + + +@dataclass(frozen=True) +class MemoryPrescreenDecision: + """One verdict. ``model`` is the served version, already reduced to a recordable token.""" + + verdict: Literal["extract", "skip"] + probability: float + model: str + cached: bool = False + reason: str = "" + + +@runtime_checkable +class MemoryPrescreenProvider(Protocol): + """Contract for a pluggable memory pre-screen. + + ``decide`` is **synchronous**: the updater runs on the debounce Timer / + executor thread and must not touch the event loop. ``None`` is "no opinion" + (a question-level failure, or nothing to judge), which the caller treats as a + fallback — extract. A *request-level* failure must propagate as + ``deerflow.typesafe.errors.TypeSafeError`` instead of ``None``, so the round's + audit record can say ``request_failed`` rather than "no verdict"; any other + exception is a provider bug and the updater still extracts as usual. + """ + + name: str + + def decide(self, request: MemoryPrescreenRequest) -> MemoryPrescreenDecision | None: + """Return a verdict for this batch, ``None`` for no opinion, or raise ``TypeSafeError`` on a request failure.""" + ... + + def release_policy_parameters(self) -> dict[str, object]: + """Behaviour-affecting parameters for assembly identity (never the credential).""" + ... + + +def resolve_memory_prescreen( + *, + mode: str, + use: str | None, + config: Mapping[str, object] | None = None, + configuration_source: str = CONFIGURATION_SOURCE, +) -> MemoryPrescreenProvider | None: + """Resolve the configured pre-screen, or ``None`` when its mode is off. + + ``off`` resolves no class path, constructs nothing and validates no + credentials (design §3), so a deployment without a pre-screen pays nothing and + cannot fail at build time. Any other mode fails **loudly** when the class path + is unusable or the provider rejects its configuration — never a silent + fallback to "no pre-screen", which would hide a deployment mistake behind a + behavior nothing records. + """ + if mode == MODE_OFF: + return None + if mode not in MODES: + raise ValueError(f"{configuration_source} mode must be one of {list(MODES)}, got {mode!r}") + if not use: + raise ValueError(f"{configuration_source} needs 'use' when mode={mode!r}") + try: + provider_class = resolve_variable(use) + except (ImportError, ValueError) as exc: + raise ValueError(f"{configuration_source} use={use!r} could not be resolved: {exc}") from exc + if not callable(provider_class): + raise ValueError(f"{configuration_source} use={use!r} is not instantiable") + return provider_class(mode=mode, **dict(config or {})) + + +__all__ = [ + "CONFIGURATION_SOURCE", + "MODE_ENFORCE", + "MODE_OFF", + "MODE_SHADOW", + "MODES", + "VERDICT_EXTRACT", + "VERDICT_SKIP", + "MemoryPrescreenDecision", + "MemoryPrescreenProvider", + "MemoryPrescreenRequest", + "resolve_memory_prescreen", +] diff --git a/backend/packages/harness/deerflow/agents/memory/prescreen/typesafe.py b/backend/packages/harness/deerflow/agents/memory/prescreen/typesafe.py new file mode 100644 index 0000000000..6b4c59ad58 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/prescreen/typesafe.py @@ -0,0 +1,185 @@ +"""TypeSafe (Jev) memory pre-screen: one ``noul`` question per batch, judged off the turn path. + +This is a **cost gate**. It answers "is this batch worth paying for an extraction +call?" and can only ever save a call — it gates no execution and no write, and +every failure mode extracts as usual (design L2). The direction is therefore the +opposite of the tool gate: there, a high probability denies; here, a probability +**below** ``skip_threshold`` skips (design §2.5). + +What this adapter owns: the state (the formatted batch, nothing else), the +question and its rubric, ``skip_threshold``, the failure direction, its own +``digest``-keyed cache, and the audit policy for the served model version. +Transport, authentication, retry, the deadline budget and response validation are +the shared client's (``deerflow.typesafe``). +""" + +from __future__ import annotations + +from collections.abc import Mapping + +from deerflow.agents.memory.judging import AnswerCache, CachedVerdict, conversation_tail_state +from deerflow.agents.memory.prescreen.contract import ( + CONFIGURATION_SOURCE, + MODE_SHADOW, + MODES, + VERDICT_EXTRACT, + VERDICT_SKIP, + MemoryPrescreenDecision, + MemoryPrescreenRequest, +) +from deerflow.typesafe.client import QUESTION_NOUL, Answer, Question, TransportFactory, TypeSafeClient, recordable_model +from deerflow.typesafe.connection import TypeSafeConnection, resolve_connection, typesafe_defaults +from deerflow.typesafe.validation import criteria_entry, defaulted_text, finite_float, whole_number + +QUESTION_ID = "memory_worth_keeping" + +_DEFAULT_INSTRUCTIONS = "Does this conversation contain durable, cross-task, user-level information that should be extracted into long-term memory? Judge the text alone." +_DEFAULT_CRITERIA_TRUE = ( + "Any of: a durable identity fact (role, profession, background); a long-term preference, working style or stable constraint; " + "a long-term goal; a durable decision or working pattern that stays useful across unrelated tasks; an explicit correction of " + "existing user-level information. These take precedence over the false criteria." +) +_DEFAULT_CRITERIA_FALSE = ( + "Task-local progress, status or next steps; greetings, acknowledgments and restatements of what was already said; process chatter " + "about the current file, PR, thread or tool run; approval of the current result only; nothing new relative to what the user already stated." +) + +DEFAULT_SKIP_THRESHOLD = 0.2 +DEFAULT_MAX_STATE_CHARS = 6000 + + +class TypeSafeMemoryPrescreen: + """Skip the extraction call when a batch's durable-value probability is below ``skip_threshold``. + + ``mode`` is recorded for the policy identity but does not change this class's + behaviour: the mode decides how a verdict is *consumed* (shadow records it, + enforce acts on it) and that decision belongs to the caller (design §2.2.4). + """ + + name = "typesafe" + policy_id = "deerflow.memory.prescreen.typesafe" + policy_version = "1.0.0" + + def __init__( + self, + *, + mode: str = MODE_SHADOW, + api_key: str | None = None, + api_key_env: str | None = None, + base_url: str | None = None, + model: str | None = None, + skip_threshold: float = DEFAULT_SKIP_THRESHOLD, + instructions: str | None = None, + criteria: dict[object, object] | None = None, + max_state_chars: int = DEFAULT_MAX_STATE_CHARS, + timeout: float | None = None, + deadline_seconds: float | None = None, + max_attempts: int | None = None, + retry_backoff: float | None = None, + cache_size: int = 256, + cache_ttl_seconds: float = 300.0, + transport_factory: TransportFactory | None = None, + ) -> None: + if mode not in MODES: + raise ValueError(f"memory.prescreen.mode must be one of {list(MODES)}, got {mode!r}") + self._mode = mode + connection: TypeSafeConnection = resolve_connection( + settings={ + "api_key": api_key, + "api_key_env": api_key_env, + "base_url": base_url, + "model": model, + "timeout": timeout, + "deadline_seconds": deadline_seconds, + "max_attempts": max_attempts, + "retry_backoff": retry_backoff, + }, + defaults=typesafe_defaults(), + configuration_source=CONFIGURATION_SOURCE, + ) + self._connection = connection + self._client = TypeSafeClient(connection, transport_factory=transport_factory) + self.skip_threshold = finite_float("skip_threshold", skip_threshold, minimum=0.0, maximum=1.0) + self._instructions = defaulted_text("instructions", instructions, _DEFAULT_INSTRUCTIONS) + self._criteria_true = defaulted_text("criteria.true", criteria_entry(criteria, True), _DEFAULT_CRITERIA_TRUE) + self._criteria_false = defaulted_text("criteria.false", criteria_entry(criteria, False), _DEFAULT_CRITERIA_FALSE) + self.max_state_chars = whole_number("max_state_chars", max_state_chars, minimum=1) + self.cache_size = whole_number("cache_size", cache_size, minimum=0) + self.cache_ttl_seconds = finite_float("cache_ttl_seconds", cache_ttl_seconds, minimum=0.0) + self._questions: dict[str, Question] = { + QUESTION_ID: Question( + type=QUESTION_NOUL, + instructions=self._instructions, + criteria={"true": self._criteria_true, "false": self._criteria_false}, + ) + } + self._cache = AnswerCache(size=self.cache_size, ttl_seconds=self.cache_ttl_seconds) + + # --- side interface (shared with the coordinator) --------------------- + + def questions(self) -> Mapping[str, Question]: + """The questions this side asks. The batch is added by :meth:`ask`/the coordinator.""" + return self._questions + + def ask(self, batch_text: str, questions: Mapping[str, Question]): + """Send ``questions`` about ``batch_text`` through this side's client.""" + return self._client.ask(conversation_tail_state(batch_text), questions) + + def sharing_key(self, **dimensions: object) -> str: + """Internal sharing identity (credential fingerprint, connection, limits, cache).""" + return self._client.sharing_key(**dimensions) + + def interpret(self, answers: Mapping[str, Answer], *, model: str, cached: bool) -> MemoryPrescreenDecision | None: + """Map validated answers to a verdict, or ``None`` when this batch has no opinion. + + A question-level failure is "no opinion", never an error: the caller + extracts as usual (L2). + """ + answer = answers.get(QUESTION_ID) + probability = getattr(answer, "probability", None) + if not isinstance(probability, float): + return None + verdict = VERDICT_SKIP if probability < self.skip_threshold else VERDICT_EXTRACT + reason = f"{verdict}: p={probability!r} {'<' if verdict == VERDICT_SKIP else '>='} skip_threshold={self.skip_threshold!r}" + return MemoryPrescreenDecision(verdict=verdict, probability=probability, model=recordable_model(model), cached=cached, reason=reason) + + def release_policy_parameters(self) -> dict[str, object]: + """Declare behaviour-affecting parameters for assembly identity (never the key).""" + from deerflow_extension_api import canonical_hash + + return { + "mode": self._mode, + **self._connection.public_parameters(), + "skip_threshold": self.skip_threshold, + "instructions_hash": canonical_hash(self._instructions), + "criteria": {"true": self._criteria_true, "false": self._criteria_false}, + "max_state_chars": self.max_state_chars, + "cache_size": self.cache_size, + "cache_ttl_seconds": self.cache_ttl_seconds, + } + + # --- the contract's standalone entry point ---------------------------- + + def decide(self, request: MemoryPrescreenRequest) -> MemoryPrescreenDecision | None: + """Judge one batch with this side's own cache (the single-side path). + + Used when this side is the only one enabled, or when the deployment + cannot combine requests. ``None`` means "no opinion for this batch" — a + question-level failure, or nothing to judge. A *request-level* failure + propagates as ``deerflow.typesafe.errors.TypeSafeError`` so the coordinator can record + ``request_failed`` rather than "no verdict"; the caller still extracts as + usual, which is L2's "extract as usual". + """ + cached = self._cache.get(request.digest) + if cached is not None: + return self.interpret(cached.answers, model=cached.model_for(self._questions), cached=True) + # No try/except: the transport failure is the coordinator's to record, and it + # writes no bucket here either way (a failure must not be remembered). + answer_set = self._client.ask(conversation_tail_state(request.batch_text), self._questions) + decision = self.interpret(answer_set.answers, model=answer_set.model, cached=False) + if decision is not None: + self._cache.put(request.digest, CachedVerdict(answers=dict(answer_set.answers), models={question_id: answer_set.model for question_id in answer_set.answers})) + return decision + + +__all__ = ["DEFAULT_MAX_STATE_CHARS", "DEFAULT_SKIP_THRESHOLD", "QUESTION_ID", "TypeSafeMemoryPrescreen"] diff --git a/backend/packages/harness/deerflow/agents/memory/signals/__init__.py b/backend/packages/harness/deerflow/agents/memory/signals/__init__.py new file mode 100644 index 0000000000..7ade3888b3 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/signals/__init__.py @@ -0,0 +1,61 @@ +"""Jev memory signal classification: reinforcement / weakening hints over one batch. + +The classifier is a *hint* source and, in one narrow case (pre-screen ``enforce`` +× classifier ``hints``), a **veto** over a skip. It never decides extraction, never +drives deletion, and never participates in the reinforcement evidence gate. + +The memory-layer coordinator in this package owns request combination and the +combined cache; each adapter keeps its own cache when the sides are not combined. + +See ``backend/docs/MEMORY_IMPROVEMENTS.md`` and ``agents/memory/AGENTS.md``. +""" + +from deerflow.agents.memory.signals.contract import ( + COMBINE_ALWAYS, + COMBINE_AUTO, + COMBINE_NEVER, + COMBINES, + CONFIGURATION_SOURCE, + LABEL_CORRECTION, + LABEL_REINFORCEMENT, + LABELS, + MODE_HINTS, + MODE_OFF, + MODE_SHADOW, + MODES, + MemorySignalDecision, + MemorySignalProvider, + MemorySignalRequest, + direction_labels, + resolve_memory_signal_classifier, +) +from deerflow.agents.memory.signals.coordinator import ( + MemoryBatchContext, + MemoryBatchVerdict, + MemorySignalCoordinator, + build_memory_judge, +) + +__all__ = [ + "COMBINE_ALWAYS", + "COMBINE_AUTO", + "COMBINE_NEVER", + "COMBINES", + "CONFIGURATION_SOURCE", + "LABEL_CORRECTION", + "LABEL_REINFORCEMENT", + "LABELS", + "MODE_HINTS", + "MODE_OFF", + "MODE_SHADOW", + "MODES", + "MemoryBatchContext", + "MemoryBatchVerdict", + "MemorySignalCoordinator", + "MemorySignalDecision", + "MemorySignalProvider", + "MemorySignalRequest", + "build_memory_judge", + "direction_labels", + "resolve_memory_signal_classifier", +] diff --git a/backend/packages/harness/deerflow/agents/memory/signals/contract.py b/backend/packages/harness/deerflow/agents/memory/signals/contract.py new file mode 100644 index 0000000000..343c2a6504 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/signals/contract.py @@ -0,0 +1,158 @@ +"""Contract for Jev memory signal classification (reinforcement / weakening hints). + +This side is **additive only**: a model verdict can add hint text and — under +pre-screen ``enforce`` × classifier ``hints`` — veto a skip. It never decides +extraction on its own, never drives deletion, and never participates in the +reinforcement evidence gate (locks S2-S5, §3 of the design). + +Host contract, consumed through the memory-layer judge: + +* :class:`MemorySignalRequest` — what the host knows about the batch; +* :class:`MemorySignalDecision` — hint labels, or ``None`` for "no model result"; +* :class:`MemorySignalProvider` — duck-typed, resolved by class path. + +See ``backend/docs/MEMORY_IMPROVEMENTS.md`` for the modes, the separate +``hints`` evidence requirement and the pre-``enforce`` gates, and +``agents/memory/AGENTS.md`` for the invariants. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from typing import Protocol, runtime_checkable + +from deerflow.reflection import resolve_variable + +MODE_OFF = "off" +MODE_SHADOW = "shadow" +MODE_HINTS = "hints" +MODES: tuple[str, ...] = (MODE_OFF, MODE_SHADOW, MODE_HINTS) + +#: How this side's requests relate to the pre-screen's (§2.2.5). +COMBINE_AUTO = "auto" +COMBINE_ALWAYS = "always" +COMBINE_NEVER = "never" +COMBINES: tuple[str, ...] = (COMBINE_AUTO, COMBINE_ALWAYS, COMBINE_NEVER) + +#: Label names aligned with the deterministic signal classes (`SIGNAL_NAMES`). +LABEL_REINFORCEMENT = "reinforcement" +LABEL_CORRECTION = "correction" +LABELS: tuple[str, ...] = (LABEL_REINFORCEMENT, LABEL_CORRECTION) + +CONFIGURATION_SOURCE = "memory.signal_classification.config" + + +@dataclass(frozen=True) +class MemorySignalRequest: + """One batch, as this side sees it (same data plane as the pre-screen).""" + + batch_text: str + digest: str + signals: frozenset[str] = frozenset() + thread_id: str | None = None + user_id: str | None = None + agent_name: str | None = None + trace_id: str | None = None + bypass_watermark: bool = False + message_count: int = 0 + + +@dataclass(frozen=True) +class MemorySignalDecision: + """Hint labels whose direction's probability reached ``hint_threshold``. + + ``probabilities`` carries only the directions that produced a validated + answer, so a response with one usable direction still contributes that one + (failure is counted per question, never per side). + """ + + labels: frozenset[str] = field(default_factory=frozenset) + probabilities: Mapping[str, float] = field(default_factory=dict) + model: str = "" + cached: bool = False + + +@runtime_checkable +class MemorySignalProvider(Protocol): + """Contract for a pluggable signal classifier (synchronous; same thread rules as the pre-screen). + + ``None`` is "no model result" (a question-level failure, or nothing validated) and + means the deterministic signals stand. A *request-level* failure must propagate as + ``deerflow.typesafe.errors.TypeSafeError`` so the round's audit record can say + ``request_failed`` rather than "no verdict". + """ + + name: str + + def decide(self, request: MemorySignalRequest) -> MemorySignalDecision | None: + """Return the batch's hint labels, ``None`` for no model result, or raise ``TypeSafeError`` on a request failure.""" + ... + + def release_policy_parameters(self) -> dict[str, object]: + """Behaviour-affecting parameters for assembly identity (never the credential).""" + ... + + +def resolve_memory_signal_classifier( + *, + mode: str, + use: str | None, + config: Mapping[str, object] | None = None, + configuration_source: str = CONFIGURATION_SOURCE, +) -> MemorySignalProvider | None: + """Resolve the configured classifier, or ``None`` when its mode is off. + + Same rules as the pre-screen's resolver: ``off`` resolves nothing and + validates nothing; any other mode fails loudly rather than silently + degrading to "no model hints". + """ + if mode == MODE_OFF: + return None + if mode not in MODES: + raise ValueError(f"{configuration_source} mode must be one of {list(MODES)}, got {mode!r}") + if not use: + raise ValueError(f"{configuration_source} needs 'use' when mode={mode!r}") + try: + provider_class = resolve_variable(use) + except (ImportError, ValueError) as exc: + raise ValueError(f"{configuration_source} use={use!r} could not be resolved: {exc}") from exc + if not callable(provider_class): + raise ValueError(f"{configuration_source} use={use!r} is not instantiable") + return provider_class(mode=mode, **dict(config or {})) + + +def direction_labels(affirmation: float | None, negation: float | None, *, hint_threshold: float) -> frozenset[str]: + """Map the two directions' probabilities to labels (design §5's fixed mapping). + + ``affirmation >= hint_threshold`` ⇒ ``reinforcement``; + ``negation >= hint_threshold`` ⇒ ``correction``. Both may hold at once + ("keep using X, but stop using Y"). + """ + labels = set() + if affirmation is not None and affirmation >= hint_threshold: + labels.add(LABEL_REINFORCEMENT) + if negation is not None and negation >= hint_threshold: + labels.add(LABEL_CORRECTION) + return frozenset(labels) + + +__all__ = [ + "COMBINE_ALWAYS", + "COMBINE_AUTO", + "COMBINE_NEVER", + "COMBINES", + "CONFIGURATION_SOURCE", + "LABEL_CORRECTION", + "LABEL_REINFORCEMENT", + "LABELS", + "MODE_HINTS", + "MODE_OFF", + "MODE_SHADOW", + "MODES", + "MemorySignalDecision", + "MemorySignalProvider", + "MemorySignalRequest", + "direction_labels", + "resolve_memory_signal_classifier", +] diff --git a/backend/packages/harness/deerflow/agents/memory/signals/coordinator.py b/backend/packages/harness/deerflow/agents/memory/signals/coordinator.py new file mode 100644 index 0000000000..f4531164c4 --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/signals/coordinator.py @@ -0,0 +1,600 @@ +"""The memory layer's judge: pre-screening and signal classification over one batch. + +This is the single object the DeerMem updater calls. It exists because the two +sides must not each assemble their own request (design §2.2, sole authority): + +* **Phase A — per-side eligibility.** Each side independently decides whether it + can judge this round. The pre-screen can be ineligible for reasons the + classifier does not share (deterministic signals, staleness/consolidation), + while both are ineligible on the emergency-flush and shutdown-drain paths. +* **Phase B — cache and assembly.** Look the cache up under the group's sharing + identity and this round's full logical question set, subtract what the cache + already holds, then group: ``combine: always`` is one request, + ``combine: never`` is one request per side, ``combine: auto`` shares only when + every effective client setting matches. The lookup precedes phase A's narrowing + (§2.2.3): a round in which only one side is still eligible reuses the answers the + bucket already holds for that digest instead of asking for them again. A full + hit sends nothing. +* **Phase C — consumption.** ``mode`` decides only what a result *means*: + pre-screen ``shadow`` records, ``enforce`` skips; classifier ``shadow`` + records, ``hints`` merges. The one exception is the veto (design §2.2.7): + pre-screen ``enforce`` × classifier ``hints`` × a model hint at or above + ``hint_threshold`` extracts instead of skipping. + +Failure is always additive: a side with no result contributes nothing, the other +side's validated answers in the same response are still consumed, and the caller +falls back to its deterministic behaviour (pre-screening extracts as usual; signal +classification falls back to the regex union). +""" + +from __future__ import annotations + +import logging +import time +from collections.abc import Mapping +from dataclasses import dataclass, field +from typing import Protocol, runtime_checkable + +from deerflow.agents.memory.judging import AnswerCache, CachedVerdict, batch_chars, batch_digest +from deerflow.agents.memory.prescreen.contract import ( + MODE_ENFORCE, + VERDICT_SKIP, + MemoryPrescreenDecision, + MemoryPrescreenProvider, +) +from deerflow.agents.memory.prescreen.contract import ( + MODE_OFF as PRESCREEN_OFF, +) +from deerflow.agents.memory.signals.contract import ( + COMBINE_ALWAYS, + COMBINE_AUTO, + COMBINE_NEVER, + COMBINES, + MODE_HINTS, + MemorySignalDecision, + MemorySignalProvider, +) +from deerflow.agents.memory.signals.contract import ( + MODE_OFF as CLASSIFIER_OFF, +) +from deerflow.typesafe.client import Answer, AnswerSet, Question +from deerflow.typesafe.errors import TypeSafeError + +logger = logging.getLogger(__name__) + +REASON_DISABLED = "disabled" +REASON_DRAIN = "shutdown_drain" +REASON_EMERGENCY = "emergency_flush" +REASON_SIGNALS = "deterministic_signals" +REASON_MAINTENANCE = "staleness_or_consolidation" +REASON_OVER_LIMIT = "over_limit" +REASON_REQUEST_FAILED = "request_failed" +REASON_NO_VERDICT = "no_verdict" + + +@dataclass(frozen=True) +class MemoryBatchContext: + """What the updater knows about one batch when it calls the judge.""" + + batch_text: str + digest: str + signals: frozenset[str] = frozenset() + staleness_review_enabled: bool = False + consolidation_enabled: bool = False + bypass_watermark: bool = False + judged: bool = True + thread_id: str | None = None + user_id: str | None = None + agent_name: str | None = None + trace_id: str | None = None + message_count: int = 0 + + +@dataclass(frozen=True) +class MemoryBatchVerdict: + """What the updater should do with this batch, plus the audit payload. + + ``skip`` is already the **effective** decision: a vetoed skip arrives as + ``skip=False`` with ``vetoed_by_model_signal=True``. ``hints`` are the model's + hint labels for the hint union (empty when the classifier is not in ``hints`` + mode or produced no usable answer). + """ + + skip: bool = False + vetoed_by_model_signal: bool = False + hints: frozenset[str] = field(default_factory=frozenset) + payload: dict[str, object] = field(default_factory=dict) + + @property + def decided(self) -> bool: + """Whether any side actually produced a record this round.""" + return bool(self.payload) + + +@runtime_checkable +class CombinablePrescreen(MemoryPrescreenProvider, Protocol): + """A pre-screen that can share a request: it exposes its questions and interpretation.""" + + max_state_chars: int + cache_size: int + cache_ttl_seconds: float + + def questions(self) -> Mapping[str, Question]: ... + + def ask(self, batch_text: str, questions: Mapping[str, Question]) -> AnswerSet: ... + + def interpret(self, answers: Mapping[str, Answer], *, model: str, cached: bool) -> MemoryPrescreenDecision | None: ... + + def sharing_key(self, **dimensions: object) -> str: ... + + +@runtime_checkable +class CombinableClassifier(MemorySignalProvider, Protocol): + """A classifier that can share a request: it exposes its questions and interpretation.""" + + max_state_chars: int + cache_size: int + cache_ttl_seconds: float + + def questions(self) -> Mapping[str, Question]: ... + + def ask(self, batch_text: str, questions: Mapping[str, Question]) -> AnswerSet: ... + + def interpret(self, answers: Mapping[str, Answer], *, model: str, cached: bool) -> MemorySignalDecision | None: ... + + def sharing_key(self, **dimensions: object) -> str: ... + + +@dataclass(frozen=True) +class _Eligibility: + eligible: bool + reason: str | None = None + + +@dataclass(frozen=True) +class _SideOutcome: + """One side's result for this round: a decision, or why there is none. + + ``request_failed`` separates "the request never produced a usable answer" from + "the provider answered and had no verdict for this batch". The audit record and + the shadow evaluation count those two as different fallbacks, so the distinction + has to survive this far. + """ + + decision: MemoryPrescreenDecision | MemorySignalDecision | None = None + request_failed: bool = False + + +class MemorySignalCoordinator: + """Judge one batch with the enabled sides, combining their requests when allowed.""" + + def __init__( + self, + *, + prescreen: MemoryPrescreenProvider | None = None, + prescreen_mode: str = PRESCREEN_OFF, + classifier: MemorySignalProvider | None = None, + classifier_mode: str = CLASSIFIER_OFF, + combine: str = COMBINE_AUTO, + ) -> None: + if combine not in COMBINES: + raise ValueError(f"memory.signal_classification.combine must be one of {list(COMBINES)}, got {combine!r}") + self._prescreen = prescreen + self._prescreen_mode = prescreen_mode + self._classifier = classifier + self._classifier_mode = classifier_mode + self._combine = combine + self._combined = combine != COMBINE_NEVER and self._both_combinable() and self._configuration_matches() + if combine == COMBINE_ALWAYS and self._prescreen is not None and self._classifier is not None and not self._combined: + raise ValueError( + "memory.signal_classification.combine='always' requires both sides to share every effective client setting " + "(model, base_url, credential fingerprint, timeouts, retries, transport factory, max_state_chars, cache settings); " + "use combine='auto' or align the two configs" + ) + self._cache = AnswerCache(size=self._cache_size(), ttl_seconds=self._cache_ttl()) if self._combined else None + + # --- construction helpers -------------------------------------------- + + def _both_combinable(self) -> bool: + return isinstance(self._prescreen, CombinablePrescreen) and isinstance(self._classifier, CombinableClassifier) and self._combine != COMBINE_NEVER + + def _cache_size(self) -> int: + return max(0, int(getattr(self._prescreen, "cache_size", 0)), int(getattr(self._classifier, "cache_size", 0))) + + def _cache_ttl(self) -> float: + return min(float(getattr(self._prescreen, "cache_ttl_seconds", 0.0)), float(getattr(self._classifier, "cache_ttl_seconds", 0.0))) + + def _sharing_key(self, side: CombinablePrescreen | CombinableClassifier) -> str: + """The side's sharing identity including the dimensions the transport cannot see (§2.2.5).""" + return side.sharing_key(max_state_chars=side.max_state_chars, cache_size=side.cache_size, cache_ttl_seconds=side.cache_ttl_seconds) + + def _configuration_matches(self) -> bool: + if not self._both_combinable(): + return False + assert isinstance(self._prescreen, CombinablePrescreen) + assert isinstance(self._classifier, CombinableClassifier) + return self._sharing_key(self._prescreen) == self._sharing_key(self._classifier) + + # --- the judge ------------------------------------------------------- + + def __call__(self, context: Mapping[str, object]) -> MemoryBatchVerdict: + """Hook entry point: DeerMem hands over a plain mapping, not a host type. + + The vendored backend imports no host judging types (it uses the same + mapping convention as ``extraction_callback``), so this adapts its payload + to :class:`MemoryBatchContext`. The digest is derived here when the caller + does not supply one, keeping the hashing policy in one place. + """ + batch_text = str(context.get("batch_text") or "") + signals = context.get("signals") + return self.judge( + MemoryBatchContext( + batch_text=batch_text, + digest=str(context.get("digest") or batch_digest(batch_text)), + signals=frozenset(signals) if signals else frozenset(), + staleness_review_enabled=bool(context.get("staleness_review_enabled")), + consolidation_enabled=bool(context.get("consolidation_enabled")), + bypass_watermark=bool(context.get("bypass_watermark")), + judged=bool(context.get("judged", True)), + thread_id=_optional_str(context.get("thread_id")), + user_id=_optional_str(context.get("user_id")), + agent_name=_optional_str(context.get("agent_name")), + trace_id=_optional_str(context.get("trace_id")), + message_count=int(context.get("message_count") or 0), + ) + ) + + def judge(self, context: MemoryBatchContext) -> MemoryBatchVerdict: + """Judge one batch. Never raises for a request-level failure (locks S2 / L2). + + An ineligible side is a *fallback*, not silence: while any side is enabled + the round still emits its record with the fallback reason (design §5's + "local fallbacks"), so "why was this batch not judged" stays auditable. A + request that failed is recorded as ``request_failed``, which keeps an + unreachable endpoint distinguishable from a provider that answered with + nothing usable. Only a round where every side is off produces an empty + payload. + """ + started = time.monotonic() + batch_length = batch_chars(context.batch_text) + prescreen_eligibility = self._prescreen_eligibility(context, batch_length) + classifier_eligibility = self._classifier_eligibility(context, batch_length) + + if self._combined and (prescreen_eligibility.eligible or classifier_eligibility.eligible): + prescreen_outcome, classifier_outcome = self._judge_combined(context, prescreen_eligibility, classifier_eligibility) + else: + prescreen_outcome = self._decide_prescreen(context) if prescreen_eligibility.eligible else _SideOutcome() + classifier_outcome = self._decide_classifier(context) if classifier_eligibility.eligible else _SideOutcome() + duration_ms = (time.monotonic() - started) * 1000 + + return self._consume( + context, + prescreen_eligibility=prescreen_eligibility, + classifier_eligibility=classifier_eligibility, + prescreen_outcome=prescreen_outcome, + classifier_outcome=classifier_outcome, + duration_ms=duration_ms, + ) + + def _decide_prescreen(self, context: MemoryBatchContext) -> _SideOutcome: + """This side's own request (the single-side path); a failure is data for the record.""" + if self._prescreen is None: + return _SideOutcome() + try: + return _SideOutcome(decision=self._prescreen.decide(_prescreen_request(context))) + except TypeSafeError: + # The provider propagates a request-level failure (see its contract) so + # this round can record it; the caller still extracts as usual (L2). + return _request_failure(self._prescreen.name) + + def _decide_classifier(self, context: MemoryBatchContext) -> _SideOutcome: + """As :meth:`_decide_prescreen`, for the classifier side.""" + if self._classifier is None: + return _SideOutcome() + try: + return _SideOutcome(decision=self._classifier.decide(_signal_request(context))) + except TypeSafeError: + return _request_failure(self._classifier.name) + + def _judge_combined( + self, + context: MemoryBatchContext, + prescreen_eligibility: _Eligibility, + classifier_eligibility: _Eligibility, + ) -> tuple[_SideOutcome, _SideOutcome]: + """One shared deployment's request assembly (§2.2.3): cache first, then the missing questions. + + The bucket is keyed by the **full logical question set**, independent of what + is sent this round, so an ineligible side costs no repeat request: its held + answers are still found and its questions are simply not part of ``wanted``. + """ + assert isinstance(self._prescreen, CombinablePrescreen) + assert isinstance(self._classifier, CombinableClassifier) + assert self._cache is not None + logical_set = tuple(sorted({**dict(self._prescreen.questions()), **dict(self._classifier.questions())})) + wanted: dict[str, Question] = {} + if prescreen_eligibility.eligible: + wanted.update(self._prescreen.questions()) + if classifier_eligibility.eligible: + wanted.update(self._classifier.questions()) + cache_key = (self._sharing_key(self._prescreen), logical_set, context.digest) + held = self._cache.get(cache_key) + answers: dict[str, Answer] = dict(held.answers) if held is not None else {} + # Per-answer model provenance: merging a later partial response must not + # re-attribute answers already held (served by a different model) to the new + # one, or a later full cache hit reports the wrong model for the verdict. + models: dict[str, str] = dict(held.models) if held is not None else {} + missing = {question_id: question for question_id, question in wanted.items() if question_id not in answers} + # Question ids answered by the network this round. A side is a cache hit only + # when it consumed none of them, so its ``cached`` flag — and the shadow + # evaluation's network-sample count — tracks the answers it actually used, + # not what the round as a whole did (S18 / per-question cache contract). + fetched: set[str] = set() + fetched_model: str | None = None + + if missing: + # Either client can carry the request: a combined deployment guarantees + # every effective setting matches, so the eligible side asks. + asker = self._prescreen if prescreen_eligibility.eligible else self._classifier + try: + answer_set = asker.ask(context.batch_text, missing) + except TypeSafeError: + # A whole-request failure writes no bucket and leaves every eligible + # side without a result: they fall back and the round records why + # (S2/S7). An ineligible side keeps its own reason. + failed = _request_failure("prescreen+signal_classification" if prescreen_eligibility.eligible and classifier_eligibility.eligible else asker.name) + return ( + failed if prescreen_eligibility.eligible else _SideOutcome(), + failed if classifier_eligibility.eligible else _SideOutcome(), + ) + answers.update(answer_set.answers) + fetched = set(answer_set.answers) + fetched_model = answer_set.model + models.update({question_id: answer_set.model for question_id in answer_set.answers}) + if answer_set.answers: + self._cache.put(cache_key, CachedVerdict(answers=dict(answers), models=dict(models))) + + # ``_interpret_side`` answers ``None`` for a question-level failure: the request + # was usable, this side's verdict was not (which is not a request failure). + return ( + _SideOutcome(decision=self._interpret_side(self._prescreen, answers, fetched=fetched, fetched_model=fetched_model, bucket=held)) if prescreen_eligibility.eligible else _SideOutcome(), + _SideOutcome(decision=self._interpret_side(self._classifier, answers, fetched=fetched, fetched_model=fetched_model, bucket=held)) if classifier_eligibility.eligible else _SideOutcome(), + ) + + @staticmethod + def _interpret_side( + side: CombinablePrescreen | CombinableClassifier, + answers: Mapping[str, Answer], + *, + fetched: set[str], + fetched_model: str | None, + bucket: CachedVerdict | None, + ): + """Interpret one side from per-answer provenance. + + A side is a cache hit — and reports the model that produced its answers — + only when every question it consumes was held in the bucket. If any of its + own questions was fetched this round, the side is a network sample and + reports the served model. Reusing a verdict is not the same as fetching it. + """ + if set(side.questions()) & fetched: + return side.interpret(answers, model=fetched_model or "", cached=False) + return side.interpret(answers, model=bucket.model_for(side.questions()) if bucket is not None else "", cached=True) + + # --- phase A --------------------------------------------------------- + + def _prescreen_eligibility(self, context: MemoryBatchContext, batch_length: int) -> _Eligibility: + if self._prescreen is None or self._prescreen_mode == PRESCREEN_OFF: + return _Eligibility(False, REASON_DISABLED) + if not context.judged: + return _Eligibility(False, REASON_DRAIN) + if context.bypass_watermark: + return _Eligibility(False, REASON_EMERGENCY) + if context.signals: + # L3: deterministic positive evidence outranks a model negative verdict. + return _Eligibility(False, REASON_SIGNALS) + if context.staleness_review_enabled or context.consolidation_enabled: + # L8: a skip would also skip that batch's maintenance review. + return _Eligibility(False, REASON_MAINTENANCE) + if self._over_limit(self._prescreen, batch_length): + return _Eligibility(False, REASON_OVER_LIMIT) + return _Eligibility(True) + + def _classifier_eligibility(self, context: MemoryBatchContext, batch_length: int) -> _Eligibility: + if self._classifier is None or self._classifier_mode == CLASSIFIER_OFF: + return _Eligibility(False, REASON_DISABLED) + if not context.judged: + return _Eligibility(False, REASON_DRAIN) + if context.bypass_watermark: + return _Eligibility(False, REASON_EMERGENCY) + if self._over_limit(self._classifier, batch_length): + return _Eligibility(False, REASON_OVER_LIMIT) + return _Eligibility(True) + + @staticmethod + def _over_limit(side: object, batch_length: int) -> bool: + """Each side enforces its own limit, in characters of the judged text (§2.3 / L5 / S16). + + ``max_state_chars`` is a character count over ``format_conversation_for_update``'s + output, and the shared client's byte counting replaces no limit + (shared-client design §2.4): a byte count would trip this fallback three + times early on CJK text. + """ + limit = getattr(side, "max_state_chars", None) + return isinstance(limit, int) and batch_length > limit + + # --- phase C --------------------------------------------------------- + + def _consume( + self, + context: MemoryBatchContext, + *, + prescreen_eligibility: _Eligibility, + classifier_eligibility: _Eligibility, + prescreen_outcome: _SideOutcome, + classifier_outcome: _SideOutcome, + duration_ms: float, + ) -> MemoryBatchVerdict: + payload: dict[str, object] = {} + if (self._prescreen is not None and self._prescreen_mode != PRESCREEN_OFF) or (self._classifier is not None and self._classifier_mode != CLASSIFIER_OFF): + payload["prescreen"] = self._prescreen_payload(context, prescreen_eligibility, prescreen_outcome, duration_ms) + payload["signal_classification"] = self._classifier_payload(context, classifier_eligibility, classifier_outcome, duration_ms) + + prescreen_decision = prescreen_outcome.decision + classifier_decision = classifier_outcome.decision + skip = prescreen_eligibility.eligible and isinstance(prescreen_decision, MemoryPrescreenDecision) and prescreen_decision.verdict == VERDICT_SKIP and self._prescreen_mode == MODE_ENFORCE + hints = classifier_decision.labels if (classifier_eligibility.eligible and isinstance(classifier_decision, MemorySignalDecision) and self._classifier_mode == MODE_HINTS) else frozenset() + vetoed = False + if skip and hints: + # §2.2.7: the only way a model verdict changes the extraction decision. + skip = False + vetoed = True + payload["skip_vetoed_by_model_signal"] = True + return MemoryBatchVerdict(skip=skip, vetoed_by_model_signal=vetoed, hints=frozenset(hints), payload=payload) + + def _prescreen_payload(self, context: MemoryBatchContext, eligibility: _Eligibility, outcome: _SideOutcome, duration_ms: float) -> dict[str, object] | None: + if self._prescreen is None or self._prescreen_mode == PRESCREEN_OFF: + return None + decision = outcome.decision if isinstance(outcome.decision, MemoryPrescreenDecision) else None + return { + "mode": self._prescreen_mode, + "verdict": decision.verdict if decision is not None else None, + "probability": decision.probability if decision is not None else None, + "skip_threshold": getattr(self._prescreen, "skip_threshold", None), + "model": decision.model if decision is not None else None, + "cached": decision.cached if decision is not None else None, + "digest": context.digest, + "signals": sorted(context.signals), + "message_count": context.message_count, + "batch_chars": len(context.batch_text), + "duration_ms": duration_ms, + "fallback_reason": None if decision is not None else _fallback_reason(eligibility, outcome), + } + + def _classifier_payload(self, context: MemoryBatchContext, eligibility: _Eligibility, outcome: _SideOutcome, duration_ms: float) -> dict[str, object] | None: + if self._classifier is None or self._classifier_mode == CLASSIFIER_OFF: + return None + decision = outcome.decision if isinstance(outcome.decision, MemorySignalDecision) else None + return { + "mode": self._classifier_mode, + "labels": sorted(decision.labels) if decision is not None else [], + "probabilities": dict(decision.probabilities) if decision is not None else {}, + "model": decision.model if decision is not None else None, + "cached": decision.cached if decision is not None else None, + "digest": context.digest, + # The deterministic set is recorded on both sides' records: the two + # channels must stay separately auditable even when only one side is on. + "signals": sorted(context.signals), + "duration_ms": duration_ms, + "fallback_reason": None if decision is not None else _fallback_reason(eligibility, outcome), + } + + # --- identity -------------------------------------------------------- + + def release_policy_parameters(self) -> dict[str, object]: + """Both sides' policy identities plus the combination policy (never a credential).""" + return { + "combine": self._combine, + "combined": self._combined, + "prescreen": self._prescreen.release_policy_parameters() if self._prescreen is not None else None, + "signal_classification": self._classifier.release_policy_parameters() if self._classifier is not None else None, + } + + +def _request_failure(side: str) -> _SideOutcome: + """One failed judge request: a reason for the audit record and a line for the operator. + + The round's record is the durable trace, but an endpoint that is unreachable must + not wait for the evaluation report to be noticed, so it is logged where it happens + (the guardrail logs its own provider failures the same way). + """ + logger.warning("Memory judge request failed (side=%s); falling back for this batch", side, exc_info=True) + return _SideOutcome(request_failed=True) + + +def _fallback_reason(eligibility: _Eligibility, outcome: _SideOutcome) -> str: + """Why this side has no verdict: the request failed, or the side had no opinion. + + ``request_failed`` is only ever set for a side that actually asked, so an + ineligible side keeps its own reason (deterministic signals, over limit, the + drain, …) even when the round's request failed. + """ + if outcome.request_failed: + return REASON_REQUEST_FAILED + return eligibility.reason or REASON_NO_VERDICT + + +def _optional_str(value: object) -> str | None: + """Identity fields arrive as ``None`` or text; anything else is dropped rather than stringified.""" + return value if isinstance(value, str) and value else None + + +def _prescreen_request(context: MemoryBatchContext): + from deerflow.agents.memory.prescreen.contract import MemoryPrescreenRequest + + return MemoryPrescreenRequest( + batch_text=context.batch_text, + digest=context.digest, + signals=context.signals, + thread_id=context.thread_id, + user_id=context.user_id, + agent_name=context.agent_name, + trace_id=context.trace_id, + bypass_watermark=context.bypass_watermark, + message_count=context.message_count, + ) + + +def _signal_request(context: MemoryBatchContext): + from deerflow.agents.memory.signals.contract import MemorySignalRequest + + return MemorySignalRequest( + batch_text=context.batch_text, + digest=context.digest, + signals=context.signals, + thread_id=context.thread_id, + user_id=context.user_id, + agent_name=context.agent_name, + trace_id=context.trace_id, + bypass_watermark=context.bypass_watermark, + message_count=context.message_count, + ) + + +def build_memory_judge(config=None) -> MemorySignalCoordinator | None: + """Build the memory layer's judge from the host memory config, or ``None`` when every side is off. + + Callers inject the result into a backend as the ``judge`` host hook; ``None`` + means "no judging configured", which leaves the extraction path byte-identical + to a deployment without this feature (L1/S1). + """ + from deerflow.agents.memory.prescreen.contract import resolve_memory_prescreen + from deerflow.agents.memory.signals.contract import resolve_memory_signal_classifier + + if config is None: + from deerflow.config.memory_config import get_memory_config + + config = get_memory_config() + prescreen_config = config.prescreen + signal_config = config.signal_classification + prescreen = resolve_memory_prescreen(mode=prescreen_config.mode, use=prescreen_config.use, config=prescreen_config.config) + classifier = resolve_memory_signal_classifier(mode=signal_config.mode, use=signal_config.use, config=signal_config.config) + if prescreen is None and classifier is None: + return None + return MemorySignalCoordinator( + prescreen=prescreen, + prescreen_mode=prescreen_config.mode, + classifier=classifier, + classifier_mode=signal_config.mode, + combine=signal_config.combine, + ) + + +__all__ = [ + "CombinableClassifier", + "CombinablePrescreen", + "MemoryBatchContext", + "MemoryBatchVerdict", + "MemorySignalCoordinator", + "build_memory_judge", +] diff --git a/backend/packages/harness/deerflow/agents/memory/signals/typesafe.py b/backend/packages/harness/deerflow/agents/memory/signals/typesafe.py new file mode 100644 index 0000000000..33b6e9e5fa --- /dev/null +++ b/backend/packages/harness/deerflow/agents/memory/signals/typesafe.py @@ -0,0 +1,248 @@ +"""TypeSafe (Jev) signal classifier: two ``noul`` questions, additive hints only. + +Two independent questions per batch — does the text **explicitly endorse** a +previously stated user-level preference/constraint/approach, and does it +**explicitly reject or reverse** one — mapped to the deterministic labels +``reinforcement`` / ``correction`` (design §5). + +What this adapter owns: the two questions and their rubrics, ``hint_threshold``, +the label mapping, its own ``digest``-keyed cache, and the audit policy for the +served model version. It never decides extraction and never drives deletion. +""" + +from __future__ import annotations + +from collections.abc import Mapping + +from deerflow.agents.memory.judging import AnswerCache, CachedVerdict, conversation_tail_state +from deerflow.agents.memory.signals.contract import ( + CONFIGURATION_SOURCE, + LABEL_CORRECTION, + LABEL_REINFORCEMENT, + MODE_SHADOW, + MODES, + MemorySignalDecision, + MemorySignalRequest, + direction_labels, +) +from deerflow.typesafe.client import QUESTION_NOUL, Answer, Question, TransportFactory, TypeSafeClient, recordable_model +from deerflow.typesafe.connection import TypeSafeConnection, resolve_connection, typesafe_defaults +from deerflow.typesafe.validation import criteria_entry, defaulted_text, finite_float, whole_number + +QUESTION_AFFIRMATION = "signal_affirmation" +QUESTION_NEGATION = "signal_negation" + +_DEFAULT_AFFIRMATION_INSTRUCTIONS = "Does this conversation explicitly endorse a user-level preference, constraint or approach that the user stated earlier? Judge the text alone." +_DEFAULT_AFFIRMATION_TRUE = 'The text explicitly endorses something the user previously asked for (for example "keep using X", "that approach is right, keep it up").' +_DEFAULT_AFFIRMATION_FALSE = "No explicit endorsement of a previously stated user-level preference, constraint or approach — approval of the current task, result or file does not count." + +_DEFAULT_NEGATION_INSTRUCTIONS = "Does this conversation explicitly reject or reverse a user-level preference, constraint or approach that the user stated earlier? Judge the text alone." +_DEFAULT_NEGATION_TRUE = 'The text explicitly rejects or reverses something the user previously asked for (for example "stop using X", "no, do it the other way").' +_DEFAULT_NEGATION_FALSE = "No explicit rejection or reversal of a previously stated user-level preference, constraint or approach — criticism of the current task, result or file does not count." + +DEFAULT_HINT_THRESHOLD = 0.5 +DEFAULT_MAX_STATE_CHARS = 6000 + + +class TypeSafeSignalClassifier: + """Turn a batch's affirmation / negation probabilities into hint labels. + + ``mode`` is recorded for the policy identity but does not change this class's + behaviour: the mode decides whether labels are consumed, recorded only, or + ignored (design §2.2.4). + """ + + name = "typesafe" + policy_id = "deerflow.memory.signals.typesafe" + policy_version = "1.0.0" + + def __init__( + self, + *, + mode: str = MODE_SHADOW, + api_key: str | None = None, + api_key_env: str | None = None, + base_url: str | None = None, + model: str | None = None, + hint_threshold: float = DEFAULT_HINT_THRESHOLD, + instructions: Mapping[str, object] | None = None, + criteria: Mapping[str, Mapping[object, object]] | None = None, + max_state_chars: int = DEFAULT_MAX_STATE_CHARS, + timeout: float | None = None, + deadline_seconds: float | None = None, + max_attempts: int | None = None, + retry_backoff: float | None = None, + cache_size: int = 256, + cache_ttl_seconds: float = 300.0, + transport_factory: TransportFactory | None = None, + ) -> None: + if mode not in MODES: + raise ValueError(f"memory.signal_classification.mode must be one of {list(MODES)}, got {mode!r}") + self._mode = mode + connection: TypeSafeConnection = resolve_connection( + settings={ + "api_key": api_key, + "api_key_env": api_key_env, + "base_url": base_url, + "model": model, + "timeout": timeout, + "deadline_seconds": deadline_seconds, + "max_attempts": max_attempts, + "retry_backoff": retry_backoff, + }, + defaults=typesafe_defaults(), + configuration_source=CONFIGURATION_SOURCE, + ) + self._connection = connection + self._client = TypeSafeClient(connection, transport_factory=transport_factory) + self.hint_threshold = finite_float("hint_threshold", hint_threshold, minimum=0.0, maximum=1.0) + affirmation_criteria = _side_criteria(criteria, QUESTION_AFFIRMATION) + negation_criteria = _side_criteria(criteria, QUESTION_NEGATION) + self._affirmation = _QuestionText( + instructions=defaulted_text(f"{QUESTION_AFFIRMATION}.instructions", _instructions(instructions, QUESTION_AFFIRMATION), _DEFAULT_AFFIRMATION_INSTRUCTIONS), + true=defaulted_text(f"{QUESTION_AFFIRMATION}.criteria.true", criteria_entry(affirmation_criteria, True), _DEFAULT_AFFIRMATION_TRUE), + false=defaulted_text(f"{QUESTION_AFFIRMATION}.criteria.false", criteria_entry(affirmation_criteria, False), _DEFAULT_AFFIRMATION_FALSE), + ) + self._negation = _QuestionText( + instructions=defaulted_text(f"{QUESTION_NEGATION}.instructions", _instructions(instructions, QUESTION_NEGATION), _DEFAULT_NEGATION_INSTRUCTIONS), + true=defaulted_text(f"{QUESTION_NEGATION}.criteria.true", criteria_entry(negation_criteria, True), _DEFAULT_NEGATION_TRUE), + false=defaulted_text(f"{QUESTION_NEGATION}.criteria.false", criteria_entry(negation_criteria, False), _DEFAULT_NEGATION_FALSE), + ) + self.max_state_chars = whole_number("max_state_chars", max_state_chars, minimum=1) + self.cache_size = whole_number("cache_size", cache_size, minimum=0) + self.cache_ttl_seconds = finite_float("cache_ttl_seconds", cache_ttl_seconds, minimum=0.0) + self._questions: dict[str, Question] = { + QUESTION_AFFIRMATION: Question(type=QUESTION_NOUL, instructions=self._affirmation.instructions, criteria={"true": self._affirmation.true, "false": self._affirmation.false}), + QUESTION_NEGATION: Question(type=QUESTION_NOUL, instructions=self._negation.instructions, criteria={"true": self._negation.true, "false": self._negation.false}), + } + self._cache = AnswerCache(size=self.cache_size, ttl_seconds=self.cache_ttl_seconds) + + # --- side interface (shared with the coordinator) --------------------- + + def questions(self) -> Mapping[str, Question]: + """The questions this side asks. The batch is added by :meth:`ask`/the coordinator.""" + return self._questions + + def ask(self, batch_text: str, questions: Mapping[str, Question]): + """Send ``questions`` about ``batch_text`` through this side's client.""" + return self._client.ask(conversation_tail_state(batch_text), questions) + + def sharing_key(self, **dimensions: object) -> str: + """Internal sharing identity (credential fingerprint, connection, limits, cache).""" + return self._client.sharing_key(**dimensions) + + def interpret(self, answers: Mapping[str, Answer], *, model: str, cached: bool) -> MemorySignalDecision | None: + """Map validated answers to hint labels, or ``None`` when neither direction is usable. + + A direction with no validated answer contributes nothing while the other + direction in the same response is still used (per question, never per + side — S7/S18). + """ + affirmation = _probability(answers.get(QUESTION_AFFIRMATION)) + negation = _probability(answers.get(QUESTION_NEGATION)) + if affirmation is None and negation is None: + return None + probabilities: dict[str, float] = {} + if affirmation is not None: + probabilities[LABEL_REINFORCEMENT] = affirmation + if negation is not None: + probabilities[LABEL_CORRECTION] = negation + return MemorySignalDecision( + labels=direction_labels(affirmation, negation, hint_threshold=self.hint_threshold), + probabilities=probabilities, + model=recordable_model(model), + cached=cached, + ) + + def release_policy_parameters(self) -> dict[str, object]: + """Declare behaviour-affecting parameters for assembly identity (never the key).""" + from deerflow_extension_api import canonical_hash + + return { + "mode": self._mode, + **self._connection.public_parameters(), + "hint_threshold": self.hint_threshold, + "affirmation": {"instructions_hash": canonical_hash(self._affirmation.instructions), "criteria": {"true": self._affirmation.true, "false": self._affirmation.false}}, + "negation": {"instructions_hash": canonical_hash(self._negation.instructions), "criteria": {"true": self._negation.true, "false": self._negation.false}}, + "max_state_chars": self.max_state_chars, + "cache_size": self.cache_size, + "cache_ttl_seconds": self.cache_ttl_seconds, + } + + # --- the contract's standalone entry point ---------------------------- + + def decide(self, request: MemorySignalRequest) -> MemorySignalDecision | None: + """Classify one batch with this side's own cache (the single-side path). + + ``None`` means "no model result for this batch" (a question-level failure, or + nothing validated). A *request-level* failure propagates as + ``deerflow.typesafe.errors.TypeSafeError`` so the coordinator can record ``request_failed`` instead + of "no verdict"; the caller still falls back to the deterministic signals, + which is S2's documented direction. + + A bucket is per question, never per side (§2.2.6 / S18): holding one + direction must not count as a full hit, so this round sends the missing + direction and merges its answer into the same bucket. A whole-request + failure yields no result for this round. + """ + cached = self._cache.get(request.digest) + answers: dict[str, Answer] = dict(cached.answers) if cached is not None else {} + models: dict[str, str] = dict(cached.models) if cached is not None else {} + missing = {question_id: question for question_id, question in self._questions.items() if question_id not in answers} + if not missing: + assert cached is not None + return self.interpret(answers, model=cached.model_for(self._questions), cached=True) + answer_set = self._client.ask(conversation_tail_state(request.batch_text), missing) + answers.update(answer_set.answers) + models.update({question_id: answer_set.model for question_id in answer_set.answers}) + if answer_set.answers: + self._cache.put(request.digest, CachedVerdict(answers=dict(answers), models=dict(models))) + # A validated answer arrived this round, so the verdict includes a + # network sample and reports the model that served it. + return self.interpret(answers, model=answer_set.model, cached=False) + # Nothing new was validated: the verdict (if any) rests on the answers the + # bucket already held, so it reports their model. A retry that returned no + # valid answers supplied none of the consumed evidence. + return self.interpret(answers, model=cached.model_for(self._questions) if cached is not None else "", cached=True) + + +class _QuestionText: + """One question's configurable text.""" + + __slots__ = ("false", "instructions", "true") + + def __init__(self, *, instructions: str, true: str, false: str) -> None: + self.instructions = instructions + self.true = true + self.false = false + + +def _instructions(instructions: Mapping[str, object] | None, question_id: str) -> object: + """Look up one question's instruction override, keyed by question id or direction name.""" + if instructions is None: + return None + entry = instructions.get(question_id) + if isinstance(entry, Mapping): + return entry.get("instructions") + return entry + + +def _side_criteria(criteria: Mapping[str, Mapping[object, object]] | None, question_id: str) -> Mapping[object, object] | None: + if criteria is None: + return None + if not isinstance(criteria, Mapping): + raise ValueError("criteria must be a mapping of question id -> {true, false}") + entry = criteria.get(question_id) + if entry is None: + return None + if not isinstance(entry, Mapping): + raise ValueError(f"criteria.{question_id} must be a mapping with optional 'true'/'false' entries") + return entry + + +def _probability(answer: Answer | None) -> float | None: + probability = getattr(answer, "probability", None) + return probability if isinstance(probability, float) else None + + +__all__ = ["DEFAULT_HINT_THRESHOLD", "DEFAULT_MAX_STATE_CHARS", "QUESTION_AFFIRMATION", "QUESTION_NEGATION", "TypeSafeSignalClassifier"] diff --git a/backend/packages/harness/deerflow/config/app_config.py b/backend/packages/harness/deerflow/config/app_config.py index 816f1743bc..d3b7589801 100644 --- a/backend/packages/harness/deerflow/config/app_config.py +++ b/backend/packages/harness/deerflow/config/app_config.py @@ -59,6 +59,7 @@ from deerflow.config.tool_config import ToolConfig, ToolGroupConfig from deerflow.config.tool_output_config import ToolOutputConfig from deerflow.config.tool_progress_config import ToolProgressConfig from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict +from deerflow.config.typesafe_config import TypeSafeConfig, load_typesafe_config_from_dict from deerflow.config.verification_config import VerificationConfig from deerflow.extensions.loader import ExtensionSpec @@ -263,6 +264,10 @@ class AppConfig(BaseModel): acp_agents: dict[str, ACPAgentConfig] = Field(default_factory=dict, description="ACP-compatible agent configuration") subagents: SubagentsAppConfig = Field(default_factory=SubagentsAppConfig, description="Subagent runtime configuration") guardrails: GuardrailsConfig = Field(default_factory=GuardrailsConfig, description="Guardrail middleware configuration") + typesafe: TypeSafeConfig = Field( + default_factory=TypeSafeConfig, + description="Defaults shared by every TypeSafe (Jev) consumer: connection, model and timeouts. Each consumer overrides what it needs in its own config.", + ) authorization: AuthorizationConfig = Field(default_factory=AuthorizationConfig, description="Fine-grained resource authorization configuration (RBAC and beyond)") input_polish: InputPolishConfig = Field(default_factory=InputPolishConfig, description="Pre-send input polishing configuration.") suggestions: SuggestionsConfig = Field(default_factory=SuggestionsConfig, description="Follow-up suggestions configuration.") @@ -507,6 +512,7 @@ class AppConfig(BaseModel): load_subagents_config_from_dict(config.subagents.model_dump()) load_tool_search_config_from_dict(config.tool_search.model_dump()) load_guardrails_config_from_dict(config.guardrails.model_dump()) + load_typesafe_config_from_dict(config.typesafe.model_dump()) load_authorization_config_from_dict(config.authorization.model_dump()) load_checkpointer_config_from_dict(config.checkpointer.model_dump() if config.checkpointer is not None else None) load_stream_bridge_config_from_dict(config.stream_bridge.model_dump() if config.stream_bridge is not None else None) diff --git a/backend/packages/harness/deerflow/config/memory_config.py b/backend/packages/harness/deerflow/config/memory_config.py index 17eb7111ba..850a405175 100644 --- a/backend/packages/harness/deerflow/config/memory_config.py +++ b/backend/packages/harness/deerflow/config/memory_config.py @@ -1,5 +1,4 @@ """Configuration for the memory mechanism (host-shared fields only). - DeerMem-private fields live in ``backends/deermem/config.py`` (``DeerMemConfig``), reached via ``backend_config`` (a dict the factory passes to the backend's ``__init__``). This module holds ONLY the host-shared fields every backend / @@ -10,15 +9,17 @@ makes backends swappable and portable (DeerMem's knobs do not leak onto the shared contract). """ +from __future__ import annotations + import logging from typing import Any, Literal -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, field_validator, model_validator logger = logging.getLogger(__name__) # Host-shared MemoryConfig fields (read by every backend / call site / factory). -_SHARED_FIELDS = frozenset({"enabled", "mode", "injection_enabled", "shutdown_flush_timeout_seconds", "manager_class", "backend_config"}) +_SHARED_FIELDS = frozenset({"enabled", "mode", "injection_enabled", "shutdown_flush_timeout_seconds", "manager_class", "backend_config", "prescreen", "signal_classification"}) # DeerMem-private fields that used to live at the top level of `memory:` in # config.yaml (pre-abstraction). On load they are auto-migrated into @@ -52,6 +53,62 @@ _LEGACY_DEERMEM_FIELDS = frozenset( ) +def _yaml_off_is_the_off_mode(value: object) -> object: + """Read the YAML ``off`` spelling as the mode name it looks like. + + ``yaml.safe_load`` follows YAML 1.1, where an unquoted ``off`` (like ``no`` and + ``false``) is a boolean, so the ``mode: off`` that ``config.example.yaml`` ships + and that a rollback types arrives here as ``False``. Rejecting it would fail the + config *reload* that returns a deployment to off, and a failed reload leaves the + previous judge (and its judging requests) running. ``True`` has no mode it could + mean, so it is left to the ``Literal``, which rejects it naming the modes. + """ + return "off" if value is False else value + + +class MemoryPrescreenConfig(BaseModel): + """Host-shared pre-screen slot: a cost gate over the extraction call. + + ``off`` (default) resolves no class path, constructs nothing and validates no + credentials, so an unconfigured deployment behaves exactly as before. + ``shadow`` decides and records without changing anything; ``enforce`` may + actually skip the extraction call, which requires the measured gate in the + pre-screening design §5. + """ + + mode: Literal["off", "shadow", "enforce"] = Field(default="off", description="off = no request and no behavior change; shadow = decide + record; enforce = a skip actually saves the extraction call") + use: str | None = Field(default=None, description="Provider class path, e.g. deerflow.agents.memory.prescreen.typesafe:TypeSafeMemoryPrescreen") + config: dict[str, Any] = Field(default_factory=dict, description="Provider-specific settings passed as kwargs") + + @field_validator("mode", mode="before") + @classmethod + def _read_yaml_off(cls, value: object) -> object: + """Accept the unquoted YAML ``off`` (see :func:`_yaml_off_is_the_off_mode`).""" + return _yaml_off_is_the_off_mode(value) + + +class MemorySignalClassificationConfig(BaseModel): + """Host-shared signal-classification slot: additive hint labels (and one narrow veto). + + ``hints`` merges the model's labels into the extraction hint text and may veto + a pre-screen skip under ``prescreen.mode: enforce``. It never decides + extraction on its own and never drives deletion. + """ + + mode: Literal["off", "shadow", "hints"] = Field(default="off", description="off = no request; shadow = record only; hints = merge labels into the hint text (and, under pre-screen enforce, allow a veto)") + use: str | None = Field(default=None, description="Provider class path, e.g. deerflow.agents.memory.signals.typesafe:TypeSafeSignalClassifier") + combine: Literal["auto", "always", "never"] = Field( + default="auto", description="Request combination with the pre-screen: auto shares when every effective client setting matches, always requires it, never splits every side into its own request" + ) + config: dict[str, Any] = Field(default_factory=dict, description="Provider-specific settings passed as kwargs") + + @field_validator("mode", mode="before") + @classmethod + def _read_yaml_off(cls, value: object) -> object: + """Accept the unquoted YAML ``off`` (see :func:`_yaml_off_is_the_off_mode`).""" + return _yaml_off_is_the_off_mode(value) + + class MemoryConfig(BaseModel): """Host-shared memory configuration (backend-agnostic).""" @@ -110,6 +167,29 @@ class MemoryConfig(BaseModel): ), ) + prescreen: MemoryPrescreenConfig = Field( + default_factory=MemoryPrescreenConfig, + description="Pre-extraction cost gate (off by default): decide whether a batch is worth an extraction call.", + ) + signal_classification: MemorySignalClassificationConfig = Field( + default_factory=MemorySignalClassificationConfig, + description="Signal-classification hints (off by default): reinforcement/correction labels from the batch text, plus a narrow veto over a pre-screen skip.", + ) + + @model_validator(mode="after") + def _check_judging_slots_are_usable(self) -> MemoryConfig: + """A configured judging slot must name a provider (fail fast, never silently off). + + An enabled mode without ``use`` would look configured while doing nothing. + The provider's own validation (credentials, thresholds, limits) runs when + the memory backend is built, so a bad deployment fails there rather than + on the first batch. + """ + for name, slot in (("prescreen", self.prescreen), ("signal_classification", self.signal_classification)): + if slot.mode != "off" and not slot.use: + raise ValueError(f"memory.{name}.use is required when mode={slot.mode!r}") + return self + def should_use_memory_tools(config: MemoryConfig) -> bool: """Return True when memory should use model-directed tools.""" diff --git a/backend/packages/harness/deerflow/config/typesafe_config.py b/backend/packages/harness/deerflow/config/typesafe_config.py new file mode 100644 index 0000000000..09691b89f5 --- /dev/null +++ b/backend/packages/harness/deerflow/config/typesafe_config.py @@ -0,0 +1,67 @@ +"""Top-level ``typesafe:`` defaults shared by every TypeSafe (Jev) consumer. + +The block supplies connection, model and timeout defaults; each consumer +overrides what it needs in its own ``config``. Precedence is consumer ``config`` +> this block > built-in defaults (design §4 rule 1), applied by +``deerflow.typesafe.connection.resolve_connection``. + +Deliberately absent, because they are adapter policy and not connection +defaults (design §4 rule 5): failure policy, cache semantics, thresholds, and the +question content. ``mode`` is a consumer's own switch, not a property of the +shared connection. + +The block is optional. With no ``typesafe:`` key the built-in defaults apply and +every consumer behaves exactly as it did before this block existed. +""" + +from __future__ import annotations + +from pydantic import BaseModel, ConfigDict, Field + + +class TypeSafeConfig(BaseModel): + """Defaults for the shared TypeSafe client; every field is optional. + + Validated in strict mode on purpose: this block is YAML, and its numbers reach + the same eager helpers the consumer ``config`` path uses. In pydantic's lax mode + ``max_attempts: true`` silently becomes ``1`` (``bool`` is an ``int``) while the + same value under ``guardrails.provider.config`` is rejected, so the two spellings + disagreed about a retry count. Strict mode makes them agree; an integer is still + accepted for a float field, so ``timeout: 5`` keeps working. + """ + + model_config = ConfigDict(strict=True) + + api_key: str | None = Field(default=None, description="API key. Prefer api_key_env: a key here lives in config.yaml") + api_key_env: str | None = Field(default=None, description="Environment variable holding the API key (default: TYPESAFE_API_KEY)") + base_url: str | None = Field(default=None, description="API base URL (default: https://api.typesafe.ai)") + model: str | None = Field(default=None, description="Model to ask for; TypeSafe returns the version it actually served") + timeout: float | None = Field(default=None, description="Per-request timeout in seconds") + deadline_seconds: float | None = Field(default=None, description="Whole-evaluation budget in seconds") + max_attempts: int | None = Field(default=None, description="Attempts per evaluation, including the first (429/529 and transport errors only)") + retry_backoff: float | None = Field(default=None, description="Base seconds of exponential backoff between attempts") + + def connection_defaults(self) -> dict[str, object]: + """Only the fields this block actually sets; ``None`` means "not configured".""" + return self.model_dump(exclude_none=True) + + +_typesafe_config: TypeSafeConfig = TypeSafeConfig() + + +def get_typesafe_config() -> TypeSafeConfig: + """Get the top-level ``typesafe:`` defaults, returning an empty block when unset.""" + return _typesafe_config + + +def load_typesafe_config_from_dict(data: dict) -> TypeSafeConfig: + """Load the top-level ``typesafe:`` block (called during AppConfig loading).""" + global _typesafe_config + _typesafe_config = TypeSafeConfig.model_validate(data) + return _typesafe_config + + +def reset_typesafe_config() -> None: + """Reset the cached block. Used in tests to prevent singleton leaks.""" + global _typesafe_config + _typesafe_config = TypeSafeConfig() diff --git a/backend/packages/harness/deerflow/guardrails/typesafe.py b/backend/packages/harness/deerflow/guardrails/typesafe.py index 21808debed..1cb59f99cd 100644 --- a/backend/packages/harness/deerflow/guardrails/typesafe.py +++ b/backend/packages/harness/deerflow/guardrails/typesafe.py @@ -5,22 +5,27 @@ cause an irreversible or out-of-scope side effect. Calls at or above ``threshold`` are denied before execution, so the agent sees the reason and can choose another approach. +Transport, authentication, retry, the deadline budget, response parsing and the +error taxonomy live in ``deerflow.typesafe`` and are shared with the memory +consumers. What stays here is everything that makes this gate a gate: the state +(tool name plus canonical argument JSON), the question and its criteria, the +threshold and its direction, the local preflight that refuses to send anything +unusable, the cache, and the failure policy — an error denies. + Three properties are load-bearing and easy to lose in a refactor: * **``state`` is built with strict JSON** — no ``default=``, ``allow_nan=False``, string keys only. Arguments that cannot be serialised, and argument text above ``max_state_chars``, are denied *locally*: nothing is truncated and sent, and - no verdict is ever based on a prefix. -* **Timeouts are budgets, not guarantees.** The async path cancels in-flight - requests through ``asyncio.timeout``. The sync path cannot preempt a blocking - call, so it checks the deadline after the response headers arrive, around the - body read, and before the decision is accepted — and drops results that - arrived late instead of returning them. -* **Clients live for exactly one evaluation.** The framework has no provider - lifecycle hook, and httpx closes an injected transport when its client closes, - so a shared transport instance would be dead on the second evaluation. - ``transport_factory`` is therefore a factory, called once per network-needing - evaluation and closed together with the client it was handed to. + no verdict is ever based on a prefix. ``max_state_chars`` stays a **character** + count of the canonical argument JSON; the shared client's ``wire_size`` reports + bytes but replaces no limit here. +* **Timeouts are budgets, not guarantees.** The shared client's async path + cancels in-flight requests through ``asyncio.timeout``; its sync path cannot + preempt a blocking call, so it checks the deadline after the response headers, + around the body read and after parsing, and drops a result that arrived late. +* **Clients live for exactly one evaluation** — ``transport_factory`` is a + factory, not an instance, and the client closes it with itself. A configured ``allowed_tools`` list is a hard permission list, not an exemption from evaluation: a tool outside it is refused locally and never probed, while a @@ -35,41 +40,31 @@ comparison (probability, threshold, model version, state digest). The served ``model`` is response content, so it is recorded verbatim only when it fits a conservative token shape; anything else is recorded as a digest rather than echoed into the message. + +Connection settings resolve with the documented precedence — this provider's own +config, then the top-level ``typesafe:`` block, then the built-in defaults — so +an existing ``guardrails.provider.config`` keeps working unchanged while a +deployment with several TypeSafe consumers can share one block. """ from __future__ import annotations -import asyncio import hashlib import json import math -import os -import re import time from collections import OrderedDict -from collections.abc import Callable from dataclasses import dataclass -import httpx - from deerflow.guardrails.provider import GuardrailDecision, GuardrailReason, GuardrailRequest +from deerflow.typesafe.client import QUESTION_NOUL, AnswerSet, Question, TransportFactory, TypeSafeClient, recordable_model +from deerflow.typesafe.connection import DEFAULT_API_KEY_ENV, DEFAULT_BASE_URL, TypeSafeConnection, resolve_connection, typesafe_defaults +from deerflow.typesafe.errors import CAUSE_INVALID_RESPONSE, TypeSafeError +from deerflow.typesafe.validation import criteria_entry, defaulted_text, finite_float, whole_number -DEFAULT_BASE_URL = "https://api.typesafe.ai" -DEFAULT_API_KEY_ENV = "TYPESAFE_API_KEY" - -_ENDPOINT_PATH = "/v1/systemone" _QUESTION_ID = "risky_tool_call" -_OK_STATUS = 200 -_UNAUTHORIZED_STATUS = 401 -_RETRYABLE_STATUS_CODES = frozenset({429, 529}) _DIGEST_CHARS = 16 -# The served ``model`` is the one response value that reaches the reason message -# verbatim, and from there the denied ToolMessage, the run journal, middleware -# logs and the evaluation report. Only a token of this shape is recorded as -# itself; anything else is recorded as a digest. See ``_recordable_model``. -_MODEL_TOKEN = re.compile(r"\A[A-Za-z0-9][A-Za-z0-9._:+-]{0,63}\Z") - _DEFAULT_INSTRUCTIONS = "Does executing this tool call risk an irreversible or out-of-scope side effect? Judge the call from its text alone." _DEFAULT_CRITERIA_TRUE = ( "Any of: deletes or overwrites data; changes permissions or access control; publishes, pushes, or sends anything outside the task; " @@ -79,25 +74,20 @@ _DEFAULT_CRITERIA_TRUE = ( ) _DEFAULT_CRITERIA_FALSE = "Read-only or retrieval-only; or a change that the call text shows to be local and reversible." -TransportFactory = Callable[[], "httpx.BaseTransport | httpx.AsyncBaseTransport"] +_CONFIGURATION_SOURCE = "guardrails.provider.config" -class TypeSafeGuardrailError(RuntimeError): +class TypeSafeGuardrailError(TypeSafeError): """A TypeSafe evaluation could not produce a usable verdict. - ``cause`` is a short machine-readable category: ``deadline``, ``transport``, - ``http_status`` or ``invalid_response``. The middleware maps this exception - to ``guardrails.fail_closed``; it is never downgraded to an allow here. + The shared request-level categories (``deadline``, ``transport``, + ``http_status``, ``invalid_response``) pass through unchanged, and a + question-level failure for this provider's own question is reported with + ``invalid_response`` too: the envelope was usable, this answer was not, and + the gate has no verdict. The middleware maps this exception to + ``guardrails.fail_closed``; it is never downgraded to an allow here. """ - def __init__(self, message: str, *, cause: str) -> None: - super().__init__(message) - self.cause = cause - - -class _RetryableAttempt(TypeSafeGuardrailError): - """Internal: this attempt failed in a way another attempt may fix.""" - @dataclass(frozen=True) class _Answer: @@ -131,7 +121,9 @@ class TypeSafeGuardrailProvider: Configuration lives in ``guardrails.provider.config``; constructor arguments are validated eagerly so a bad deployment fails at agent build time rather - than on the first tool call. + than on the first tool call. The connection fields this provider does not set + come from the top-level ``typesafe:`` block, and from the built-in defaults + when that block is absent. """ name = "typesafe" @@ -142,67 +134,74 @@ class TypeSafeGuardrailProvider: self, *, api_key: str | None = None, - api_key_env: str = DEFAULT_API_KEY_ENV, - base_url: str = DEFAULT_BASE_URL, - model: str = "jev-latest", + api_key_env: str | None = None, + base_url: str | None = None, + model: str | None = None, threshold: float = 0.5, instructions: str | None = None, criteria: dict[object, object] | None = None, tools: list[str] | None = None, allowed_tools: list[str] | None = None, - timeout: float = 5.0, - deadline_seconds: float = 10.0, - max_attempts: int = 2, - retry_backoff: float = 0.5, + timeout: float | None = None, + deadline_seconds: float | None = None, + max_attempts: int | None = None, + retry_backoff: float | None = None, max_state_chars: int = 4000, cache_size: int = 256, cache_ttl_seconds: float = 300.0, transport_factory: TransportFactory | None = None, ) -> None: - resolved_key = api_key if api_key is not None else os.environ.get(api_key_env) - if not resolved_key: - raise ValueError(f"TypeSafe requires an API key: pass 'api_key' in guardrails.provider.config or set the {api_key_env} environment variable") - if not isinstance(base_url, str) or not base_url.strip(): - raise ValueError("base_url must be a non-empty string") - if not isinstance(model, str) or not model.strip(): - raise ValueError("model must be a non-empty string; TypeSafe returns the version it actually served") - - self._api_key = resolved_key - self._url = f"{base_url.rstrip('/')}{_ENDPOINT_PATH}" - self._base_url = base_url.rstrip("/") - self._model = model - self._threshold = _finite_float("threshold", threshold, minimum=0.0, maximum=1.0) - self._instructions = _defaulted_text("instructions", instructions, _DEFAULT_INSTRUCTIONS) - self._criteria_true = _defaulted_text("criteria.true", _criteria_entry(criteria, True), _DEFAULT_CRITERIA_TRUE) - self._criteria_false = _defaulted_text("criteria.false", _criteria_entry(criteria, False), _DEFAULT_CRITERIA_FALSE) + connection: TypeSafeConnection = resolve_connection( + settings={ + "api_key": api_key, + "api_key_env": api_key_env, + "base_url": base_url, + "model": model, + "timeout": timeout, + "deadline_seconds": deadline_seconds, + "max_attempts": max_attempts, + "retry_backoff": retry_backoff, + }, + defaults=typesafe_defaults(), + configuration_source=_CONFIGURATION_SOURCE, + ) + self._connection = connection + self._client = TypeSafeClient(connection, transport_factory=transport_factory) + self._threshold = finite_float("threshold", threshold, minimum=0.0, maximum=1.0) + self._instructions = defaulted_text("instructions", instructions, _DEFAULT_INSTRUCTIONS) + self._criteria_true = defaulted_text("criteria.true", criteria_entry(criteria, True), _DEFAULT_CRITERIA_TRUE) + self._criteria_false = defaulted_text("criteria.false", criteria_entry(criteria, False), _DEFAULT_CRITERIA_FALSE) self._tools = _tool_names("tools", tools) self._allowed_tools = _tool_names("allowed_tools", allowed_tools) - self._timeout = _finite_float("timeout", timeout, minimum=0.0, exclusive=True) - self._deadline_seconds = _finite_float("deadline_seconds", deadline_seconds, minimum=0.0, exclusive=True) - self._max_attempts = _whole_number("max_attempts", max_attempts, minimum=1) - self._retry_backoff = _finite_float("retry_backoff", retry_backoff, minimum=0.0) - self._max_state_chars = _whole_number("max_state_chars", max_state_chars, minimum=1) - self._cache_size = _whole_number("cache_size", cache_size, minimum=0) - self._cache_ttl_seconds = _finite_float("cache_ttl_seconds", cache_ttl_seconds, minimum=0.0) - self._transport_factory = transport_factory + self._max_state_chars = whole_number("max_state_chars", max_state_chars, minimum=1) + self._cache_size = whole_number("cache_size", cache_size, minimum=0) + self._cache_ttl_seconds = finite_float("cache_ttl_seconds", cache_ttl_seconds, minimum=0.0) + self._question = Question( + type=QUESTION_NOUL, + instructions=self._instructions, + criteria={"true": self._criteria_true, "false": self._criteria_false}, + ) + self._questions: dict[str, Question] = {_QUESTION_ID: self._question} self._cache: OrderedDict[tuple[str, str], _CacheEntry] = OrderedDict() def release_policy_parameters(self) -> dict[str, object]: - """Declare behaviour-affecting parameters for assembly identity (never the key).""" + """Declare behaviour-affecting parameters for assembly identity (never the key). + + This is the public, per-consumer policy identity. It carries the + connection's public parameters plus everything that changes a verdict — + and never the credential: the fingerprint that decides whether two + consumers may share a request belongs to ``TypeSafeClient.sharing_key``, + which consumers compare internally. + """ from deerflow_extension_api import canonical_hash return { - "model": self._model, - "base_url": self._base_url, + **self._connection.public_parameters(), "threshold": self._threshold, "instructions_hash": canonical_hash(self._instructions), "criteria": {"true": self._criteria_true, "false": self._criteria_false}, "tools": None if self._tools is None else sorted(self._tools), "allowed_tools": None if self._allowed_tools is None else sorted(self._allowed_tools), - "timeout": self._timeout, - "deadline_seconds": self._deadline_seconds, - "max_attempts": self._max_attempts, - "retry_backoff": self._retry_backoff, "max_state_chars": self._max_state_chars, "cache_size": self._cache_size, "cache_ttl_seconds": self._cache_ttl_seconds, @@ -217,10 +216,7 @@ class TypeSafeGuardrailProvider: cached = self._cache_get(prepared) if cached is not None: return self._cached_decision(cached) - deadline_at = time.monotonic() + self._deadline_seconds - answer = self._post_sync(self._build_payload(prepared), deadline_at) - self._check_budget(deadline_at) - return self._decide_from_answer(prepared, answer) + return self._decide_from_answer(prepared, self._ask_sync(prepared)) async def aevaluate(self, request: GuardrailRequest) -> GuardrailDecision: prepared = self._prepare(request) @@ -229,10 +225,39 @@ class TypeSafeGuardrailProvider: cached = self._cache_get(prepared) if cached is not None: return self._cached_decision(cached) - deadline_at = time.monotonic() + self._deadline_seconds - answer = await self._post_async(self._build_payload(prepared), deadline_at) - self._check_budget(deadline_at) - return self._decide_from_answer(prepared, answer) + return self._decide_from_answer(prepared, await self._ask_async(prepared)) + + def _ask_sync(self, probe: _Probe) -> _Answer: + try: + answer_set = self._client.ask(self._state(probe), self._questions) + except TypeSafeError as exc: + raise TypeSafeGuardrailError(str(exc), cause=exc.cause) from exc + return self._recorded_answer(answer_set) + + async def _ask_async(self, probe: _Probe) -> _Answer: + try: + answer_set = await self._client.aask(self._state(probe), self._questions) + except TypeSafeError as exc: + raise TypeSafeGuardrailError(str(exc), cause=exc.cause) from exc + return self._recorded_answer(answer_set) + + def _recorded_answer(self, answer_set: AnswerSet) -> _Answer: + """This provider's one answer, with the served model reduced to a recordable token. + + The shared parser returns a question-level failure as data so that a + memory consumer can fall back per question. This provider asked exactly + one question, so that failure means the evaluation has no verdict — the + same outcome it had when the parser raised for it, with the same cause. + """ + answer = answer_set.noul(_QUESTION_ID) + if answer is None: + error = answer_set.errors_by_question.get(_QUESTION_ID) + message = error.message if error is not None else f"TypeSafe response has no answer for question {_QUESTION_ID!r}" + raise TypeSafeGuardrailError(message, cause=CAUSE_INVALID_RESPONSE) + return _Answer(probability=answer.probability, model=recordable_model(answer_set.model)) + + def _state(self, probe: _Probe) -> dict[str, object]: + return {"tool_call": {"name": probe.tool_name, "arguments": probe.arguments_text}} # --- preflight (local, no network) ------------------------------------ @@ -274,6 +299,10 @@ class TypeSafeGuardrailProvider: state_digest = hashlib.sha256(state_text.encode("utf-8")).hexdigest()[:_DIGEST_CHARS] except (TypeError, ValueError) as exc: # UnicodeEncodeError arrives here through ValueError return self._state_unusable("unserializable", reason=type(exc).__name__, digest="unavailable") + # Character count of the canonical argument JSON, not its wire size: one + # CJK character is three UTF-8 bytes, so a byte limit here would move the + # boundary this fallback has always drawn (the shared client counts bytes + # when a consumer asks it to; it changes no limit). if len(arguments_text) > self._max_state_chars: return self._state_unusable("too_large", limit=self._max_state_chars, length=len(arguments_text), digest=state_digest) return _Probe(tool_name=tool_name, arguments_text=arguments_text, state_digest=state_digest) @@ -288,19 +317,6 @@ class TypeSafeGuardrailProvider: metadata={"state_unusable_cause": cause, **fields}, ) - def _build_payload(self, probe: _Probe) -> dict[str, object]: - return { - "state": {"tool_call": {"name": probe.tool_name, "arguments": probe.arguments_text}}, - "model": self._model, - "questions": { - _QUESTION_ID: { - "type": "noul", - "instructions": self._instructions, - "criteria": {"true": self._criteria_true, "false": self._criteria_false}, - } - }, - } - def _decide_from_answer(self, probe: _Probe, answer: _Answer) -> GuardrailDecision: allow = answer.probability < self._threshold self._cache_put(probe, allow=allow, answer=answer) @@ -313,7 +329,7 @@ class TypeSafeGuardrailProvider: code = "typesafe.allowed" if allow else "typesafe.tool_call_risky" # repr(): the recorded probability and threshold must replay to the same # verdict, so neither is rounded for display. repr() round-trips exactly. - # ``model`` is already bounded by _recordable_model: it is response + # ``model`` is already bounded by recordable_model: it is response # content, so a value outside the recordable shape arrives as a digest and # never as the echoed text this message would otherwise carry into the # ToolMessage, the run journal, the logs and the report. @@ -328,137 +344,6 @@ class TypeSafeGuardrailProvider: def _policy_ref(self) -> str: return f"{self.policy_id}@{self.policy_version}" - # --- transport -------------------------------------------------------- - - def _post_sync(self, payload: dict[str, object], deadline_at: float) -> _Answer: - transport = None if self._transport_factory is None else self._transport_factory() - with httpx.Client(timeout=self._timeout, transport=transport) as client: - for attempt in range(1, self._max_attempts + 1): - self._check_budget(deadline_at) - try: - return self._attempt_sync(client, payload, deadline_at) - except _RetryableAttempt as exc: - if attempt >= self._max_attempts: - raise TypeSafeGuardrailError(str(exc), cause=exc.cause) from exc - self._sleep_before_retry(attempt, deadline_at) - - async def _post_async(self, payload: dict[str, object], deadline_at: float) -> _Answer: - # The cancellation scope covers the in-flight request and its body read: - # it bounds the request duration, not the wall-clock cost of cleanup. - try: - async with asyncio.timeout(self._deadline_seconds): - transport = None if self._transport_factory is None else self._transport_factory() - async with httpx.AsyncClient(timeout=self._timeout, transport=transport) as client: - for attempt in range(1, self._max_attempts + 1): - self._check_budget(deadline_at) - try: - return await self._attempt_async(client, payload, deadline_at) - except _RetryableAttempt as exc: - if attempt >= self._max_attempts: - raise TypeSafeGuardrailError(str(exc), cause=exc.cause) from exc - await self._async_sleep_before_retry(attempt, deadline_at) - except TimeoutError as exc: - # asyncio.timeout converts only the cancellation it triggered; an - # outer CancelledError stays a CancelledError and keeps propagating. - raise self._deadline_error() from exc - - def _attempt_sync(self, client: httpx.Client, payload: dict[str, object], deadline_at: float) -> _Answer: - try: - with client.stream("POST", self._url, json=payload, headers=self._headers()) as response: - self._check_budget(deadline_at) - if response.status_code != _OK_STATUS: - raise self._status_error(response.status_code) - self._check_budget(deadline_at) - body = response.read() - self._check_budget(deadline_at) - return self._parse_answer(body) - except httpx.TransportError as exc: - raise _RetryableAttempt(f"TypeSafe request failed: {type(exc).__name__}", cause="transport") from exc - except httpx.HTTPError as exc: - raise TypeSafeGuardrailError(f"TypeSafe request failed: {type(exc).__name__}", cause="transport") from exc - - async def _attempt_async(self, client: httpx.AsyncClient, payload: dict[str, object], deadline_at: float) -> _Answer: - try: - async with client.stream("POST", self._url, json=payload, headers=self._headers()) as response: - self._check_budget(deadline_at) - if response.status_code != _OK_STATUS: - raise self._status_error(response.status_code) - self._check_budget(deadline_at) - body = await response.aread() - self._check_budget(deadline_at) - return self._parse_answer(body) - except httpx.TransportError as exc: - raise _RetryableAttempt(f"TypeSafe request failed: {type(exc).__name__}", cause="transport") from exc - except httpx.HTTPError as exc: - raise TypeSafeGuardrailError(f"TypeSafe request failed: {type(exc).__name__}", cause="transport") from exc - - def _headers(self) -> dict[str, str]: - return {"Authorization": f"Bearer {self._api_key}", "Accept": "application/json"} - - def _status_error(self, status_code: int) -> TypeSafeGuardrailError: - # Only the numeric status is reported. The response body *and* the HTTP - # status line are both server-controlled, TypeSafe may echo the state - # (raw tool arguments) in either, and this message reaches middleware - # logs and the evaluation report. - hint = " (check the API key configured via api_key / the API key environment variable)" if status_code == _UNAUTHORIZED_STATUS else "" - message = f"TypeSafe returned HTTP {status_code}{hint}" - if status_code in _RETRYABLE_STATUS_CODES: - return _RetryableAttempt(message, cause="http_status") - return TypeSafeGuardrailError(message, cause="http_status") - - def _parse_answer(self, body: bytes) -> _Answer: - try: - payload = json.loads(body) - except ValueError as exc: - raise TypeSafeGuardrailError("TypeSafe response was not valid JSON", cause="invalid_response") from exc - if not isinstance(payload, dict): - raise TypeSafeGuardrailError("TypeSafe response must be a JSON object", cause="invalid_response") - model = payload.get("model") - if not isinstance(model, str) or not model: - raise TypeSafeGuardrailError("TypeSafe response is missing a non-empty 'model'", cause="invalid_response") - answers = payload.get("answers") - if not isinstance(answers, dict): - raise TypeSafeGuardrailError("TypeSafe response is missing an 'answers' object", cause="invalid_response") - answer = answers.get(_QUESTION_ID) - if not isinstance(answer, dict): - raise TypeSafeGuardrailError(f"TypeSafe response has no answer for question {_QUESTION_ID!r}", cause="invalid_response") - if answer.get("type") != "noul": - raise TypeSafeGuardrailError(f"TypeSafe answer for {_QUESTION_ID!r} is not a noul answer (type={type(answer.get('type')).__name__})", cause="invalid_response") - raw_probability = answer.get("noul") - # bool is an int subclass, and json.loads parses NaN/Infinity literals, so - # both are rejected here rather than reaching the threshold comparison. - # Only the offending type is reported: a malformed response may echo tool - # arguments here, and this message reaches middleware logs and the - # evaluation report, which is exactly what the HTTP error path avoids. - if isinstance(raw_probability, bool) or not isinstance(raw_probability, (int, float)): - raise TypeSafeGuardrailError(f"TypeSafe returned a non-numeric noul probability (type={type(raw_probability).__name__})", cause="invalid_response") - probability = float(raw_probability) - if not math.isfinite(probability) or not 0.0 <= probability <= 1.0: - raise TypeSafeGuardrailError("TypeSafe returned a noul probability outside [0, 1]", cause="invalid_response") - return _Answer(probability=probability, model=_recordable_model(model)) - - # --- deadline budget -------------------------------------------------- - - def _check_budget(self, deadline_at: float) -> None: - """Reject a result when the evaluation budget is spent, even on success.""" - if time.monotonic() >= deadline_at: - raise self._deadline_error() - - def _deadline_error(self) -> TypeSafeGuardrailError: - return TypeSafeGuardrailError(f"TypeSafe evaluation exceeded deadline_seconds={self._deadline_seconds:g}", cause="deadline") - - def _remaining(self, deadline_at: float) -> float: - remaining = deadline_at - time.monotonic() - if remaining <= 0: - raise self._deadline_error() - return remaining - - def _sleep_before_retry(self, attempt: int, deadline_at: float) -> None: - time.sleep(min(self._retry_backoff * 2 ** (attempt - 1), self._remaining(deadline_at))) - - async def _async_sleep_before_retry(self, attempt: int, deadline_at: float) -> None: - await asyncio.sleep(min(self._retry_backoff * 2 ** (attempt - 1), self._remaining(deadline_at))) - # --- cache ------------------------------------------------------------ def _cache_get(self, probe: _Probe) -> _CacheEntry | None: @@ -492,26 +377,6 @@ class TypeSafeGuardrailProvider: self._cache.popitem(last=False) -def _recordable_model(model: str) -> str: - """Return the served model token, or a digest when it is not safe to record. - - ``model`` is response content: it lands in the denial ``ToolMessage``, the run - journal, middleware logs and the evaluation report. A malformed or hostile - endpoint can put an echoed tool argument, injected instructions or a - multi-megabyte string there, so only a token matching ``_MODEL_TOKEN`` is - recorded verbatim. Anything else keeps a stable, bounded provenance record - instead of being echoed, without discarding a verdict that is otherwise - usable: rejecting the response would turn a cosmetic server quirk into a - fail-closed denial of every call the endpoint answers. - """ - if _MODEL_TOKEN.match(model): - return model - # surrogatepass: json.loads produces lone surrogates for escaped input, and a - # plain UTF-8 encode would raise on them where this must not fail. - digest = hashlib.sha256(model.encode("utf-8", "surrogatepass")).hexdigest()[:_DIGEST_CHARS] - return f"unrecorded:sha256:{digest}" - - def _strict_json_failure(value: object) -> str | None: """Return the exception name ``json.dumps`` would raise on ``value``, else None. @@ -551,51 +416,6 @@ def _strict_json_failure(value: object) -> str | None: return None -def _finite_float(name: str, value: object, *, minimum: float | None = None, maximum: float | None = None, exclusive: bool = False) -> float: - if isinstance(value, bool) or not isinstance(value, (int, float)): - raise ValueError(f"{name} must be a finite number, got {value!r}") - number = float(value) - if not math.isfinite(number): - raise ValueError(f"{name} must be a finite number, got {value!r}") - if minimum is not None and (number <= minimum if exclusive else number < minimum): - raise ValueError(f"{name} must be {'>' if exclusive else '>='} {minimum:g}, got {number!r}") - if maximum is not None and number > maximum: - raise ValueError(f"{name} must be <= {maximum:g}, got {number!r}") - return number - - -def _whole_number(name: str, value: object, *, minimum: int) -> int: - if isinstance(value, bool) or not isinstance(value, int): - raise ValueError(f"{name} must be an integer, got {value!r}") - if value < minimum: - raise ValueError(f"{name} must be >= {minimum}, got {value!r}") - return value - - -def _defaulted_text(name: str, value: object, fallback: str) -> str: - if value is None: - return fallback - if not isinstance(value, str) or not value.strip(): - raise ValueError(f"{name} must be a non-empty string") - return value - - -def _criteria_entry(criteria: dict[object, object] | None, flag: bool) -> object: - """Look up a ``criteria`` entry under either the YAML or the JSON spelling. - - A YAML ``criteria: {true: ..., false: ...}`` block parses its keys as booleans, - while a JSON-typed config keeps them as the strings ``"true"``/``"false"``. - """ - if criteria is None: - return None - if not isinstance(criteria, dict): - raise ValueError("criteria must be a mapping with optional 'true'/'false' entries") - for key in (flag, "true" if flag else "false"): - if key in criteria: - return criteria[key] - return None - - def _tool_names(name: str, value: object) -> frozenset[str] | None: """``None`` means "not configured"; an empty list is a configured empty set. diff --git a/backend/packages/harness/deerflow/typesafe/AGENTS.md b/backend/packages/harness/deerflow/typesafe/AGENTS.md new file mode 100644 index 0000000000..86bffe2726 --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/AGENTS.md @@ -0,0 +1,52 @@ +### Shared TypeSafe client + +`typesafe/` is the transport layer for every TypeSafe (Jev) call site: the +guardrail risk gate (`guardrails/typesafe.py`), and — once enabled — memory +pre-screening and signal classification. It owns exactly transport and lifecycle, +authentication, retry and backoff, the deadline budget, response parsing and the +error taxonomy, UTF-8 wire-size counting, and the request skeleton. Clients live +for one evaluation: no cache, no connection pool, and `transport_factory` is +always a factory because httpx closes the transport it was handed. + +**It must not gain, and no consumer may move into it:** state content, the +questions/criteria/thresholds and which way they decide, the failure policy (the +gate denies on error, the memory paths carry on), cache semantics, or business +audit. Two consumers with different thresholds are expected to share one request, +so policy parameters can never reach this layer — they belong to each consumer's +`release_policy_parameters()`, while `sharing_key()` carries only the credential +fingerprint, the connection settings, the consumer's input limit and the transport +identity. Keeping that split is the whole point: contaminating either identity +either merges requests that must stay apart or breaks assembly fingerprinting. + +Two-layer responses are load-bearing. A request-level failure (transport, +deadline, non-200, missing envelope, unparsable-or-unbounded body) raises +`TypeSafeError` with a `cause`, and the consumer decides deny / fall back. A +question-level failure is **data** (`AnswerSet.errors_by_question`), never an +exception, so one bad answer cannot discard the other valid answers in the same +response; the tool gate maps its own question's error back to +`TypeSafeGuardrailError` because it has no verdict without it. The reply is never +read unbounded and never decoded: the client asks for an identity encoding, refuses +a body that carries a non-identity `Content-Encoding`, and stops past +`MAX_RESPONSE_BYTES` (64 KiB, a few hundred bytes is the expected envelope), so an +endpoint that answers with something other than an answer set cannot be expanded +into memory or accepted as a verdict. Deep nesting is mapped too — `json.loads` +raises `RecursionError`, not `ValueError`, and it must not escape this taxonomy. +The deadline covers the read, not only the headers: the sync path, which cannot +preempt the read it is blocked in, re-checks the budget before each transport read, +so a body that drips just under the read timeout cannot hold the caller for the +whole of `MAX_RESPONSE_BYTES` past `deadline_seconds`. + +Limits and secrets must not move. `max_state_chars` stays a character count in +every consumer — `wire_size` reports bytes and replaces no limit, and one CJK +character is three of them. A credential never reaches a log, an error message, a +policy identity, a reason message or `repr()` (`TypeSafeConnection` hides both the +key and its environment-variable name); only `sha256` fingerprints are compared. +Two rules keep that promise true at the edges: a credential that could not travel +as a header value (surrounding whitespace, a non-printable character) is refused +when the connection resolves, and a transport failure is raised without chaining +the original exception, whose message can contain the request it refused. +Credential resolution is per *layer* too — the first layer that sets `api_key` or +`api_key_env` decides it, and inside one layer the literal key wins — so a +consumer naming an environment variable is never silently switched to another +layer's key. Likewise `base_url` must be an `http(s)` URL with no query, fragment +or embedded credentials, checked at construction rather than on every request. diff --git a/backend/packages/harness/deerflow/typesafe/__init__.py b/backend/packages/harness/deerflow/typesafe/__init__.py new file mode 100644 index 0000000000..02ac0f1c76 --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/__init__.py @@ -0,0 +1,118 @@ +"""Shared TypeSafe (Jev) client for every host-side consumer. + +One transport layer, several adapters: + +* ``guardrails/typesafe.py`` — the pre-execution tool risk gate. +* ``agents/memory/prescreen/`` — the memory capture pre-screen (planned). +* ``agents/memory/signals/`` — signal classification and its coordinator (planned). + +**Shared** (this package): client lifecycle and ``transport_factory`` injection, +authentication, retry and backoff, the deadline budget, response parsing and the +error taxonomy, UTF-8 wire-size counting, and the request skeleton. + +**Not shared — it stays in each adapter, on purpose:** + +* **State content**: tool-call arguments for the gate, the conversation tail for + the memory consumers. The client never inspects what it sends. +* **Questions, criteria, thresholds and decision direction**: ``risky_tool_call`` + denies at or above its threshold, ``memory_worth_keeping`` skips below its own — + opposite directions over the same transport. +* **Failure policy**: the gate fails closed (error ⇒ deny); the memory paths fail + softly (error ⇒ carry on with their normal behaviour). +* **Cache semantics**: the gate's provider owns its cache, the memory layer's + coordinator owns the combined-request cache. Different keys, different owners, + so no cached verdict is ever shared between consumers by accident. +* **Business audit**: what a decision records (the gate's run-journal reason, the + memory pre-screen's record) and how it relates to the journal or metrics. + +A change that would move one of those into this package is a change to the design, +not a refactor (see the "Shared client" section of ``backend/docs/GUARDRAILS.md`` +§2.2). Caching, pooling and per-consumer policy parameters must stay out. +""" + +from deerflow.typesafe.client import ( + CATEGORY_LABEL, + CATEGORY_MISSING, + CATEGORY_PROBABILITY, + CATEGORY_TYPE, + QUESTION_CHOICE, + QUESTION_NOUL, + Answer, + AnswerSet, + ChoiceAnswer, + NoulAnswer, + Question, + QuestionError, + TransportFactory, + TypeSafeClient, + recordable_model, + wire_size, +) +from deerflow.typesafe.connection import ( + CONNECTION_FIELDS, + DEFAULT_API_KEY_ENV, + DEFAULT_BASE_URL, + DEFAULT_DEADLINE_SECONDS, + DEFAULT_MAX_ATTEMPTS, + DEFAULT_MODEL, + DEFAULT_RETRY_BACKOFF, + DEFAULT_TIMEOUT, + ENDPOINT_PATH, + MODE_OFF, + TypeSafeConnection, + resolve_connection, + resolve_connection_for_mode, + typesafe_defaults, +) +from deerflow.typesafe.errors import ( + CAUSE_DEADLINE, + CAUSE_HTTP_STATUS, + CAUSE_INVALID_RESPONSE, + CAUSE_TRANSPORT, + CAUSES, + TypeSafeError, +) +from deerflow.typesafe.validation import criteria_entry, defaulted_text, finite_float, whole_number + +__all__ = [ + "CAUSE_DEADLINE", + "CAUSE_HTTP_STATUS", + "CAUSE_INVALID_RESPONSE", + "CAUSE_TRANSPORT", + "CAUSES", + "CATEGORY_LABEL", + "CATEGORY_MISSING", + "CATEGORY_PROBABILITY", + "CATEGORY_TYPE", + "CONNECTION_FIELDS", + "DEFAULT_API_KEY_ENV", + "DEFAULT_BASE_URL", + "DEFAULT_DEADLINE_SECONDS", + "DEFAULT_MAX_ATTEMPTS", + "DEFAULT_MODEL", + "DEFAULT_RETRY_BACKOFF", + "DEFAULT_TIMEOUT", + "ENDPOINT_PATH", + "MODE_OFF", + "QUESTION_CHOICE", + "QUESTION_NOUL", + "Answer", + "AnswerSet", + "ChoiceAnswer", + "NoulAnswer", + "Question", + "QuestionError", + "TransportFactory", + "TypeSafeClient", + "TypeSafeConnection", + "recordable_model", + "TypeSafeError", + "criteria_entry", + "defaulted_text", + "finite_float", + "resolve_connection", + "resolve_connection_for_mode", + "typesafe_defaults", + "whole_number", + "wire_size", +] diff --git a/backend/packages/harness/deerflow/typesafe/client.py b/backend/packages/harness/deerflow/typesafe/client.py new file mode 100644 index 0000000000..46dca7ab5d --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/client.py @@ -0,0 +1,556 @@ +"""The shared TypeSafe (Jev) transport client: one request, one answer set. + +What this layer owns — transport and lifecycle, authentication, retry and +backoff, the deadline budget, response parsing and the error taxonomy, UTF-8 wire +size counting, and the request skeleton (design §2.1). + +What it deliberately does **not** own (design §2.2): state content, the +questions, criteria and thresholds, the failure policy that turns a failure into a +denial or a fallback, cache semantics, and business audit. Those stay in each +adapter, which is why this module has no branch that knows what a "risky tool +call" or a "memory worth keeping" is. + +Consequences worth keeping in mind when editing: + +* **Clients live for exactly one evaluation.** The framework has no provider + lifecycle hook, and httpx closes an injected transport when its client closes, + so a shared transport instance would be dead on the second evaluation. + ``transport_factory`` is a factory, called once per network-needing evaluation + and closed together with the client it was handed to. There is no connection + pool here (design §3.3). +* **Timeouts are budgets, not guarantees.** The async path cancels the in-flight + request through ``asyncio.timeout``. The sync path cannot preempt a blocking + call, so it checks the deadline after the response headers, before each piece of + the body read, and again after parsing — and drops a result that arrived late + instead of adopting it. +* **A question-level failure is data, not an exception** (design §2.3). One + malformed answer must not discard the other valid answers in the same response; + the consumer decides what "no result for this question" means. +* **The response body is bounded and never decoded.** A 200 is not a reason to + read a body into memory: the client asks for `identity`, refuses a non-identity + `Content-Encoding`, and stops past ``MAX_RESPONSE_BYTES``. A body that is not a + usable envelope is a request-level ``invalid_response`` failure, not a verdict. +* **Wire size is a counting capability only** (design §2.4). No consumer's limit + unit moves because this module exists. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +import re +import time +from collections.abc import AsyncIterable, Callable, Iterable, Mapping +from dataclasses import dataclass + +import httpx + +from deerflow.typesafe.connection import TypeSafeConnection +from deerflow.typesafe.errors import ( + CAUSE_DEADLINE, + CAUSE_HTTP_STATUS, + CAUSE_INVALID_RESPONSE, + CAUSE_TRANSPORT, + TypeSafeError, +) + +QUESTION_NOUL = "noul" +QUESTION_CHOICE = "choice" +_QUESTION_TYPES = (QUESTION_NOUL, QUESTION_CHOICE) + +CATEGORY_MISSING = "missing" +CATEGORY_TYPE = "type" +CATEGORY_PROBABILITY = "probability" +CATEGORY_LABEL = "label" + +_OK_STATUS = 200 +_UNAUTHORIZED_STATUS = 401 +_RETRYABLE_STATUS_CODES = frozenset({429, 529}) + +#: Most a 200 body may occupy before the response is rejected. A System One +#: envelope is a few hundred bytes, so this only ever fires for an endpoint that +#: answered with something other than an answer set — and a compressed body makes +#: the decoded size, not the bytes on the wire, the number that matters. +MAX_RESPONSE_BYTES = 64 * 1024 + +#: Never ask for a compressed body: the client counts the bytes of a response and +#: has no reason to accept an encoded one, which would make the *decoded* size the +#: number that matters. A response that carries a non-identity +#: ``Content-Encoding`` anyway is rejected rather than decoded. +_ACCEPT_ENCODING = "identity" + +#: A callable returning a fresh transport, because httpx closes the transport it +#: was given when its client closes. +type TransportFactory = Callable[[], httpx.BaseTransport | httpx.AsyncBaseTransport] + + +def wire_size(value: object) -> int: + """Return the bytes ``httpx`` would send for ``json=value``: compact UTF-8 JSON. + + A counting capability, not a limit (design §2.4). Characters and UTF-8 bytes + differ — one CJK character is three bytes — so a consumer that swapped its + existing character limit for this number would move its own fallback boundary. + + Unlike ``json.dumps`` defaults this uses ``ensure_ascii=False`` and compact + separators, matching httpx exactly, and ``allow_nan=False``, matching httpx's + rejection of ``NaN``/``Infinity``. A value httpx cannot send therefore raises + here too, rather than reporting a size that no request would have. + """ + return len(json.dumps(value, ensure_ascii=False, separators=(",", ":"), allow_nan=False).encode("utf-8")) + + +# The served ``model`` is the one response value every consumer records, and from +# there it reaches reason messages, audit records, logs and reports. Only a token +# of this shape is recorded as itself; anything else is recorded as a digest. See +# ``recordable_model``. +_MODEL_TOKEN = re.compile(r"\A[A-Za-z0-9][A-Za-z0-9._:+-]{0,63}\Z") +_DIGEST_CHARS = 16 + + +def recordable_model(model: str) -> str: + """Return the served model token, or a digest when it is not safe to record. + + ``model`` is response content. A malformed or hostile endpoint can put an + echoed batch of conversation text, injected instructions or a multi-megabyte + string there, so only a token matching ``_MODEL_TOKEN`` is recorded verbatim. + Anything else keeps a stable, bounded provenance record instead of being + echoed, without discarding a verdict that is otherwise usable: rejecting the + response would turn a cosmetic server quirk into a lost verdict. + """ + if _MODEL_TOKEN.match(model): + return model + # surrogatepass: json.loads produces lone surrogates for escaped input, and a + # plain UTF-8 encode would raise on them where this must not fail. + digest = hashlib.sha256(model.encode("utf-8", "surrogatepass")).hexdigest()[:_DIGEST_CHARS] + return f"unrecorded:sha256:{digest}" + + +@dataclass(frozen=True) +class Question: + """One question in a request; the mapping key it is passed under is its id. + + ``criteria`` carries the answer labels for a ``choice`` question (required + there: it is what makes a returned label valid) and the judgeable criteria for + a ``noul`` question (optional). + """ + + type: str + instructions: str + criteria: Mapping[str, str] | None = None + + def __post_init__(self) -> None: + if self.type not in _QUESTION_TYPES: + raise ValueError(f"question type must be one of {list(_QUESTION_TYPES)}, got {self.type!r}") + if not isinstance(self.instructions, str) or not self.instructions.strip(): + raise ValueError("question instructions must be a non-empty string") + if self.type == QUESTION_CHOICE and not self.criteria: + raise ValueError("a choice question needs 'criteria' mapping every valid label to its description") + if self.criteria is not None and (not isinstance(self.criteria, Mapping) or any(not isinstance(label, str) or not label or not isinstance(description, str) or not description for label, description in self.criteria.items())): + raise ValueError("criteria must map non-empty labels to non-empty descriptions") + + +@dataclass(frozen=True) +class NoulAnswer: + """A validated probability from a ``noul`` question.""" + + probability: float + + +@dataclass(frozen=True) +class ChoiceAnswer: + """A validated label from a ``choice`` question.""" + + label: str + + +type Answer = NoulAnswer | ChoiceAnswer + + +@dataclass(frozen=True) +class QuestionError: + """A question-level failure: the request was usable, this question's answer was not. + + ``category`` is one of ``missing``, ``type``, ``probability`` or ``label``. + ``message`` names the problem without repeating the response body, which a + malformed or hostile endpoint can fill with echoed state. + """ + + question_id: str + category: str + message: str + + +@dataclass(frozen=True) +class AnswerSet: + """One response, split into envelope-level success plus per-question outcomes. + + ``answers`` holds the questions that validated; ``errors_by_question`` holds + the ones that did not. Only the questions actually asked appear in either — + an answer for a question this request never asked is ignored, not reported. + """ + + model: str + answers: Mapping[str, Answer] + errors_by_question: Mapping[str, QuestionError] + + def noul(self, question_id: str) -> NoulAnswer | None: + """The validated ``noul`` answer for ``question_id``, or ``None`` when it failed.""" + answer = self.answers.get(question_id) + return answer if isinstance(answer, NoulAnswer) else None + + +class _RetryableAttempt(TypeSafeError): + """Internal: this attempt failed in a way another attempt may fix. + + Caught inside the retry loop and re-raised as a plain :class:`TypeSafeError` + when the attempt budget is exhausted, so it never escapes this module. + """ + + +class TypeSafeClient: + """One consumer's TypeSafe client, built from its effective connection. + + Holds no cache: cache ownership stays with each adapter, so a shared client + never decides that two consumers' results are interchangeable (design §1.2, + §3.1). + """ + + def __init__(self, connection: TypeSafeConnection, *, transport_factory: TransportFactory | None = None) -> None: + self._connection = connection + self._transport_factory = transport_factory + + @property + def connection(self) -> TypeSafeConnection: + """The effective connection this client was built from.""" + return self._connection + + def sharing_key(self, **dimensions: object) -> str: + """Internal identity of "one request that could be shared" (never a credential). + + Two consumers may share a request only when every dimension here matches: + the connection settings, the credential *fingerprint*, the + ``transport_factory`` object, and whatever the consumer adds — its input + limit (``max_state_chars``) and its cache settings. + + Questions, criteria and thresholds are deliberately absent: sides with + different behaviour parameters can still share one request, while each + consumer publishes its own policy identity separately through its + ``release_policy_parameters()`` (design §4 rule 2). + """ + from deerflow_extension_api import canonical_hash + + identity: dict[str, object] = { + **self._connection.public_parameters(), + "credential_fingerprint": self._connection.credential_fingerprint(), + "transport_factory": _factory_identity(self._transport_factory), + **dimensions, + } + return canonical_hash(identity) + + def ask(self, state: object, questions: Mapping[str, Question]) -> AnswerSet: + """Ask ``questions`` about ``state`` and return the validated answer set. + + ``state`` is serialised by httpx, so it must be JSON-serialisable; a + consumer whose policy is "never send an unserialisable state" validates it + first, as the tool gate does, and answers locally instead. + + Raises :class:`TypeSafeError` for every request-level failure. A + question-level failure is returned inside the answer set. + + Blocking: never call this from the event loop — use :meth:`aask` there. + """ + deadline_at = time.monotonic() + self._connection.deadline_seconds + payload = self._payload(state, questions) + headers = self._headers() + transport = None if self._transport_factory is None else self._transport_factory() + with httpx.Client(timeout=self._connection.timeout, transport=transport) as client: + for attempt in range(1, self._connection.max_attempts + 1): + self._check_budget(deadline_at) + try: + return self._attempt_sync(client, payload, headers, questions, deadline_at) + except _RetryableAttempt as exc: + if attempt >= self._connection.max_attempts: + raise TypeSafeError(str(exc), cause=exc.cause) from exc + self._sleep_before_retry(attempt, deadline_at) + + async def aask(self, state: object, questions: Mapping[str, Question]) -> AnswerSet: + """The async variant of :meth:`ask`; cancels an in-flight request at the deadline.""" + payload = self._payload(state, questions) + headers = self._headers() + # The cancellation scope covers the in-flight request and its body read: + # it bounds the request duration, not the wall-clock cost of cleanup. + try: + async with asyncio.timeout(self._connection.deadline_seconds): + deadline_at = time.monotonic() + self._connection.deadline_seconds + transport = None if self._transport_factory is None else self._transport_factory() + async with httpx.AsyncClient(timeout=self._connection.timeout, transport=transport) as client: + for attempt in range(1, self._connection.max_attempts + 1): + self._check_budget(deadline_at) + try: + return await self._attempt_async(client, payload, headers, questions, deadline_at) + except _RetryableAttempt as exc: + if attempt >= self._connection.max_attempts: + raise TypeSafeError(str(exc), cause=exc.cause) from exc + await self._async_sleep_before_retry(attempt, deadline_at) + except TimeoutError as exc: + # asyncio.timeout converts only the cancellation it triggered; an + # outer CancelledError stays a CancelledError and keeps propagating. + raise self._deadline_error() from exc + + # --- request skeleton ------------------------------------------------- + + def _payload(self, state: object, questions: Mapping[str, Question]) -> dict[str, object]: + return { + "state": state, + "model": self._connection.model, + "questions": {question_id: {"type": question.type, "instructions": question.instructions, **({"criteria": dict(question.criteria)} if question.criteria else {})} for question_id, question in questions.items()}, + } + + def _headers(self) -> dict[str, str]: + return {"Authorization": f"Bearer {self._connection.api_key}", "Accept": "application/json", "Accept-Encoding": _ACCEPT_ENCODING} + + # --- transport -------------------------------------------------------- + + def _attempt_sync(self, client: httpx.Client, payload: dict[str, object], headers: dict[str, str], questions: Mapping[str, Question], deadline_at: float) -> AnswerSet: + try: + with client.stream("POST", self._connection.url, json=payload, headers=headers) as response: + self._check_budget(deadline_at) + if response.status_code != _OK_STATUS: + raise self._status_error(response.status_code) + self._check_budget(deadline_at) + body = _read_bounded(response, lambda: self._check_budget(deadline_at)) + self._check_budget(deadline_at) + answers = self._parse(body, questions) + self._check_budget(deadline_at) + return answers + except httpx.TransportError as exc: + # httpx.TransportError is an httpx.HTTPError subclass, so it is + # matched first: a connection reset is worth another attempt, while a + # request-building error is not. + # ``from None``: the original exception's message can contain the + # request it failed to build — for a rejected header that is the whole + # ``Bearer `` value — and the guardrail's ``logger.exception`` + # prints the cause chain. The type name below is kept for diagnosis. + raise _RetryableAttempt(f"TypeSafe request failed: {type(exc).__name__}", cause=CAUSE_TRANSPORT) from None + except httpx.HTTPError as exc: + raise TypeSafeError(f"TypeSafe request failed: {type(exc).__name__}", cause=CAUSE_TRANSPORT) from None + + async def _attempt_async(self, client: httpx.AsyncClient, payload: dict[str, object], headers: dict[str, str], questions: Mapping[str, Question], deadline_at: float) -> AnswerSet: + try: + async with client.stream("POST", self._connection.url, json=payload, headers=headers) as response: + self._check_budget(deadline_at) + if response.status_code != _OK_STATUS: + raise self._status_error(response.status_code) + self._check_budget(deadline_at) + body = await _aread_bounded(response, lambda: self._check_budget(deadline_at)) + self._check_budget(deadline_at) + answers = self._parse(body, questions) + self._check_budget(deadline_at) + return answers + except httpx.TransportError as exc: + # ``from None`` for the same reason as the sync path: a rejected header + # would otherwise carry the ``Bearer`` value in the printed chain. + raise _RetryableAttempt(f"TypeSafe request failed: {type(exc).__name__}", cause=CAUSE_TRANSPORT) from None + except httpx.HTTPError as exc: + raise TypeSafeError(f"TypeSafe request failed: {type(exc).__name__}", cause=CAUSE_TRANSPORT) from None + + def _status_error(self, status_code: int) -> TypeSafeError: + # Only the numeric status is reported. The response body *and* the HTTP + # status line are both server-controlled, TypeSafe may echo the state + # (raw tool arguments, conversation text) in either, and this message + # reaches middleware logs and the evaluation report. + hint = " (check the API key configured via api_key / the API key environment variable)" if status_code == _UNAUTHORIZED_STATUS else "" + message = f"TypeSafe returned HTTP {status_code}{hint}" + if status_code in _RETRYABLE_STATUS_CODES: + return _RetryableAttempt(message, cause=CAUSE_HTTP_STATUS) + return TypeSafeError(message, cause=CAUSE_HTTP_STATUS) + + # --- response parsing ------------------------------------------------- + + def _parse(self, body: bytes, questions: Mapping[str, Question]) -> AnswerSet: + """Split one response into a usable envelope plus per-question outcomes (design §2.3).""" + try: + payload = json.loads(body) + # ``RecursionError`` and not ``ValueError`` is what deep nesting raises, and it + # is not a ``TypeSafeError``: without it here a hostile or broken endpoint + # would escape this module's taxonomy and the consumer would lose the round's + # record instead of falling back. + except (ValueError, RecursionError) as exc: + raise TypeSafeError("TypeSafe response was not valid JSON", cause=CAUSE_INVALID_RESPONSE) from exc + if not isinstance(payload, dict): + raise TypeSafeError("TypeSafe response must be a JSON object", cause=CAUSE_INVALID_RESPONSE) + model = payload.get("model") + if not isinstance(model, str) or not model: + raise TypeSafeError("TypeSafe response is missing a non-empty 'model'", cause=CAUSE_INVALID_RESPONSE) + answers = payload.get("answers") + if not isinstance(answers, dict): + raise TypeSafeError("TypeSafe response is missing an 'answers' object", cause=CAUSE_INVALID_RESPONSE) + + validated: dict[str, Answer] = {} + errors: dict[str, QuestionError] = {} + for question_id, question in questions.items(): + answer, error = _validate_answer(question_id, question, answers.get(question_id)) + if error is not None: + errors[question_id] = error + elif answer is not None: + validated[question_id] = answer + return AnswerSet(model=model, answers=validated, errors_by_question=errors) + + # --- deadline budget -------------------------------------------------- + + def _check_budget(self, deadline_at: float) -> None: + """Reject a result when the evaluation budget is spent, even on success.""" + if time.monotonic() >= deadline_at: + raise self._deadline_error() + + def _deadline_error(self) -> TypeSafeError: + return TypeSafeError(f"TypeSafe evaluation exceeded deadline_seconds={self._connection.deadline_seconds:g}", cause=CAUSE_DEADLINE) + + def _remaining(self, deadline_at: float) -> float: + remaining = deadline_at - time.monotonic() + if remaining <= 0: + raise self._deadline_error() + return remaining + + def _sleep_before_retry(self, attempt: int, deadline_at: float) -> None: + time.sleep(min(self._connection.retry_backoff * 2 ** (attempt - 1), self._remaining(deadline_at))) + + async def _async_sleep_before_retry(self, attempt: int, deadline_at: float) -> None: + await asyncio.sleep(min(self._connection.retry_backoff * 2 ** (attempt - 1), self._remaining(deadline_at))) + + +def _read_bounded(response: httpx.Response, check_budget: Callable[[], None]) -> bytes: + """The response body, refused past ``MAX_RESPONSE_BYTES`` and never decoded. + + The endpoint is treated as potentially hostile (module docstring), so a 200 is + not a reason to read an unbounded body into memory: a compressed response is + rejected before it is expanded, and the accumulated bytes are counted as they + arrive rather than after the whole body is buffered. ``iter_bytes`` is the + streamed *and* preloaded view (an injected transport may hand over a complete + response), which is safe here because a non-identity encoding never gets this far. + + ``check_budget`` runs before every piece. The sync path cannot preempt the read + it is already blocked in, so this is what keeps a dripping body from outliving + the deadline: the pieces are the transport's own reads (no re-chunking, which + would pull many reads inside one piece before the check could run), so the + caller waits at most one read past ``deadline_seconds`` instead of one per + ``MAX_RESPONSE_BYTES`` worth of them. + """ + _reject_content_encoding(response) + return _accumulate(response.iter_bytes(), check_budget) + + +async def _aread_bounded(response: httpx.Response, check_budget: Callable[[], None]) -> bytes: + """The async variant of :func:`_read_bounded`.""" + _reject_content_encoding(response) + return await _aaccumulate(response.aiter_bytes(), check_budget) + + +def _reject_content_encoding(response: httpx.Response) -> None: + """Refuse an encoded body instead of letting httpx decode it (see ``_ACCEPT_ENCODING``).""" + encoding = (response.headers.get("content-encoding") or "identity").strip().lower() + if encoding != "identity": + # ``recordable_model`` is this module's policy for a short response value that + # reaches a message: a token as itself, anything else as a digest. + raise TypeSafeError(f"TypeSafe response carried a non-identity Content-Encoding ({recordable_model(encoding)})", cause=CAUSE_INVALID_RESPONSE) + + +def _accumulate(chunks: Iterable[bytes], check_budget: Callable[[], None]) -> bytes: + body = bytearray() + for chunk in chunks: + check_budget() + body += chunk + if len(body) > MAX_RESPONSE_BYTES: + raise TypeSafeError(f"TypeSafe response exceeded {MAX_RESPONSE_BYTES} bytes", cause=CAUSE_INVALID_RESPONSE) + return bytes(body) + + +async def _aaccumulate(chunks: AsyncIterable[bytes], check_budget: Callable[[], None]) -> bytes: + body = bytearray() + async for chunk in chunks: + check_budget() + body += chunk + if len(body) > MAX_RESPONSE_BYTES: + raise TypeSafeError(f"TypeSafe response exceeded {MAX_RESPONSE_BYTES} bytes", cause=CAUSE_INVALID_RESPONSE) + return bytes(body) + + +def _validate_answer(question_id: str, question: Question, raw: object) -> tuple[Answer | None, QuestionError | None]: + if raw is None: + return None, QuestionError(question_id=question_id, category=CATEGORY_MISSING, message=f"TypeSafe response has no answer for question {question_id!r}") + if not isinstance(raw, dict): + return None, QuestionError(question_id=question_id, category=CATEGORY_TYPE, message=f"TypeSafe answer for question {question_id!r} is not an object (type={type(raw).__name__})") + if raw.get("type") != question.type: + # Only the type name is reported: the answer itself is response content + # and may echo the state back into a log or the evaluation report. + return None, QuestionError(question_id=question_id, category=CATEGORY_TYPE, message=f"TypeSafe answer for {question_id!r} is not a {question.type} answer (type={type(raw.get('type')).__name__})") + if question.type == QUESTION_NOUL: + return _validate_noul(question_id, raw) + return _validate_choice(question_id, question, raw) + + +def _validate_noul(question_id: str, raw: dict[object, object]) -> tuple[Answer | None, QuestionError | None]: + raw_probability = raw.get("noul") + # bool is an int subclass, and json.loads parses NaN/Infinity literals, so both + # are rejected here rather than reaching a consumer's comparison. Only the + # offending type is reported, for the reason in _validate_answer. + if isinstance(raw_probability, bool) or not isinstance(raw_probability, (int, float)): + return None, QuestionError(question_id=question_id, category=CATEGORY_PROBABILITY, message=f"TypeSafe returned a non-numeric noul probability (type={type(raw_probability).__name__})") + try: + probability = float(raw_probability) + except OverflowError: + # A JSON integer too large for float() (e.g. 10**400) must stay a + # per-question error: letting it escape would discard the other valid + # answers in the same response (design §2.3). + return None, QuestionError(question_id=question_id, category=CATEGORY_PROBABILITY, message="TypeSafe returned a noul probability outside [0, 1]") + if not math.isfinite(probability) or not 0.0 <= probability <= 1.0: + return None, QuestionError(question_id=question_id, category=CATEGORY_PROBABILITY, message="TypeSafe returned a noul probability outside [0, 1]") + return NoulAnswer(probability=probability), None + + +def _validate_choice(question_id: str, question: Question, raw: dict[object, object]) -> tuple[Answer | None, QuestionError | None]: + label = raw.get("choice") + if not isinstance(label, str) or not label: + return None, QuestionError(question_id=question_id, category=CATEGORY_LABEL, message=f"TypeSafe returned a non-textual choice label (type={type(label).__name__})") + if question.criteria is None or label not in question.criteria: + return None, QuestionError(question_id=question_id, category=CATEGORY_LABEL, message="TypeSafe returned a choice label outside the configured criteria") + return ChoiceAnswer(label=label), None + + +def _factory_identity(factory: TransportFactory | None) -> str | None: + """Identity of the injected transport factory, for ``sharing_key``. + + ``id()`` is process-local on purpose: sharing a request is a within-process + decision, and merging two consumers that would use *different* transports is + the failure this must not allow. Two distinct but equivalent factories + therefore separate, which is the conservative direction. The factory is held + by the client, so its address cannot be reused while the key is in use. + """ + if factory is None: + return None + module = getattr(factory, "__module__", type(factory).__module__) + qualname = getattr(factory, "__qualname__", type(factory).__qualname__) + return f"{module}.{qualname}#{id(factory)}" + + +__all__ = [ + "CATEGORY_LABEL", + "CATEGORY_MISSING", + "CATEGORY_PROBABILITY", + "CATEGORY_TYPE", + "MAX_RESPONSE_BYTES", + "QUESTION_CHOICE", + "QUESTION_NOUL", + "Answer", + "AnswerSet", + "ChoiceAnswer", + "NoulAnswer", + "Question", + "QuestionError", + "TransportFactory", + "TypeSafeClient", + "recordable_model", + "wire_size", +] diff --git a/backend/packages/harness/deerflow/typesafe/connection.py b/backend/packages/harness/deerflow/typesafe/connection.py new file mode 100644 index 0000000000..1260603686 --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/connection.py @@ -0,0 +1,279 @@ +"""Effective TypeSafe connection settings: precedence, credential fingerprint, sharing key. + +Every consumer resolves its connection through :func:`resolve_connection`, which +applies the one documented precedence — the consumer's own ``config``, then the +top-level ``typesafe:`` block, then the built-in defaults. The result is the +*effective configuration* (design §4 rule 1); a consumer override keeps working +exactly as a standalone setting did. + +Two identities come out of this layer and must never be conflated (design §4 +rule 2): + +* :meth:`TypeSafeConnection.credential_fingerprint` and the client's + ``sharing_key`` — internal, and the only place the credential is compared. + They decide whether two consumers may share one request. +* :meth:`TypeSafeConnection.public_parameters` — the connection half of a + consumer's public policy identity. It never contains the credential or its + fingerprint. + +The credential exists only inside this object: it is not in ``repr()``, not in +``public_parameters()``, and not in any error message. The environment-variable +name is kept out of ``repr()`` too, and appears only in the construction-time +error that tells an operator which variable to set. A key that cannot be sent as a +header value — surrounding whitespace from a mounted secret, a non-printable +character — is rejected here instead of leaking into a per-call protocol error +whose message carries the whole ``Bearer`` header. + +The two credential settings are resolved per *layer*: the first layer (consumer +``config``, then ``typesafe:``) that sets either one decides the credential, so a +consumer naming an environment variable is not silently overridden by a literal +key written in the shared block. +""" + +from __future__ import annotations + +import hashlib +import os +from collections.abc import Mapping +from dataclasses import dataclass, field +from urllib.parse import urlsplit + +from deerflow.typesafe.validation import credential_text, finite_float, whole_number + +DEFAULT_BASE_URL = "https://api.typesafe.ai" +DEFAULT_API_KEY_ENV = "TYPESAFE_API_KEY" +DEFAULT_MODEL = "jev-latest" +DEFAULT_TIMEOUT = 5.0 +DEFAULT_DEADLINE_SECONDS = 10.0 +DEFAULT_MAX_ATTEMPTS = 2 +DEFAULT_RETRY_BACKOFF = 0.5 + +ENDPOINT_PATH = "/v1/systemone" + +#: The keys a consumer ``config`` and the top-level ``typesafe:`` block may set. +#: Anything else in a consumer ``config`` is that consumer's own policy (thresholds, +#: questions, scope) and is deliberately not read here. +CONNECTION_FIELDS = ( + "api_key", + "api_key_env", + "base_url", + "model", + "timeout", + "deadline_seconds", + "max_attempts", + "retry_backoff", +) + +#: The one consumer mode this layer must understand: nothing is resolved, nothing +#: is constructed, no credential is looked up (design §4 rule 4). +MODE_OFF = "off" + + +@dataclass(frozen=True) +class TypeSafeConnection: + """One consumer's resolved connection settings.""" + + api_key: str = field(repr=False) + api_key_env: str = field(repr=False) + base_url: str + model: str + timeout: float + deadline_seconds: float + max_attempts: int + retry_backoff: float + + @property + def url(self) -> str: + """The System One endpoint this connection posts to.""" + return f"{self.base_url.rstrip('/')}{ENDPOINT_PATH}" + + def credential_fingerprint(self) -> str: + """A short digest of the credential, to compare consumers without recording either key.""" + return hashlib.sha256(self.api_key.encode("utf-8")).hexdigest()[:16] + + def public_parameters(self) -> dict[str, object]: + """The connection's behaviour-affecting settings, for a consumer's public policy identity.""" + return { + "model": self.model, + "base_url": self.base_url.rstrip("/"), + "timeout": self.timeout, + "deadline_seconds": self.deadline_seconds, + "max_attempts": self.max_attempts, + "retry_backoff": self.retry_backoff, + } + + +def resolve_connection( + *, + settings: Mapping[str, object] | None = None, + defaults: Mapping[str, object] | None = None, + configuration_source: str, +) -> TypeSafeConnection: + """Resolve the effective connection for one consumer. + + ``settings`` is the consumer's own ``config``; ``defaults`` is the top-level + ``typesafe:`` block (``TypeSafeConfig.connection_defaults()``). In both, a key + that is absent — or explicitly ``None`` — is "not configured" and falls + through to the next source; an explicitly blank value does not, so a blank + ``api_key`` is reported rather than silently bypassed by the environment. + ``api_key`` and ``api_key_env`` are one credential *per layer*: see + :func:`_resolve_credential`. + + ``configuration_source`` names the consumer's settings location (for example + ``guardrails.provider.config``); it reaches the missing-credential message so + an operator knows which block to edit. It never carries the credential. + """ + override = _configured(settings) + fallback = _configured(defaults) + + api_key_env = _first(override, fallback, "api_key_env", DEFAULT_API_KEY_ENV) + if not isinstance(api_key_env, str) or not api_key_env: + raise ValueError("api_key_env must be a non-empty string naming the environment variable that holds the API key") + + api_key = _resolve_credential(override, fallback, api_key_env) + if not isinstance(api_key, str) or not api_key: + raise ValueError(f"TypeSafe requires an API key: pass 'api_key' in {configuration_source} or set the {api_key_env} environment variable") + # Shape only, and never echoed: a key that cannot travel as a header value is a + # configuration error here instead of a per-call protocol error whose message + # carries the whole ``Bearer`` header into the guardrail's exception log. + api_key = credential_text("api_key", api_key) + + base_url = _validated_base_url(_first(override, fallback, "base_url", DEFAULT_BASE_URL)) + + model = _first(override, fallback, "model", DEFAULT_MODEL) + if not isinstance(model, str) or not model.strip(): + raise ValueError("model must be a non-empty string; TypeSafe returns the version it actually served") + + return TypeSafeConnection( + api_key=api_key, + api_key_env=api_key_env, + base_url=base_url, + model=model, + timeout=finite_float("timeout", _first(override, fallback, "timeout", DEFAULT_TIMEOUT), minimum=0.0, exclusive=True), + deadline_seconds=finite_float("deadline_seconds", _first(override, fallback, "deadline_seconds", DEFAULT_DEADLINE_SECONDS), minimum=0.0, exclusive=True), + max_attempts=whole_number("max_attempts", _first(override, fallback, "max_attempts", DEFAULT_MAX_ATTEMPTS), minimum=1), + retry_backoff=finite_float("retry_backoff", _first(override, fallback, "retry_backoff", DEFAULT_RETRY_BACKOFF), minimum=0.0), + ) + + +def resolve_connection_for_mode( + *, + mode: str, + settings: Mapping[str, object] | None = None, + defaults: Mapping[str, object] | None = None, + configuration_source: str, +) -> TypeSafeConnection | None: + """Resolve a mode-bearing consumer's connection, or ``None`` when it is off. + + ``off`` short-circuits before anything else: no credential lookup, no + validation, no error. A consumer that is off resolves no class path either — + reading ``mode`` is the only thing that happens — so a deployment without the + optional vendor configuration still builds (design §4 rule 4). + + Every other mode resolves exactly as :func:`resolve_connection` does and + fails at construction when the credential or a value is unusable. + """ + if mode == MODE_OFF: + return None + return resolve_connection(settings=settings, defaults=defaults, configuration_source=configuration_source) + + +def _configured(settings: Mapping[str, object] | None) -> dict[str, object]: + """Keep the connection keys this layer owns; ``None`` means "not configured".""" + if settings is None: + return {} + return {key: value for key, value in settings.items() if value is not None and key in CONNECTION_FIELDS} + + +def typesafe_defaults() -> dict[str, object]: + """The top-level ``typesafe:`` block, resolved on demand. + + Every consumer passes this as ``resolve_connection(defaults=...)`` so a value + it does not set falls back to the shared block and then to the built-in + defaults (design §4 rule 1). Imported lazily so that importing the transport + layer does not pull in the configuration package. + """ + from deerflow.config.typesafe_config import get_typesafe_config + + return get_typesafe_config().connection_defaults() + + +def _first(override: Mapping[str, object], fallback: Mapping[str, object], key: str, default: object) -> object: + if key in override: + return override[key] + if key in fallback: + return fallback[key] + return default + + +#: Schemes an endpoint base may use. ``http`` is allowed for any host on purpose: +#: an internal http endpoint is a legitimate deployment, and TLS policy is the +#: operator's. What is rejected is a base that no request could ever use. +_BASE_URL_SCHEMES = frozenset({"http", "https"}) + + +def _resolve_credential(override: Mapping[str, object], fallback: Mapping[str, object], default_env: str) -> object: + """The credential named by the first layer that sets either credential setting. + + One credential source per layer: a layer that names an environment variable is + not overridden by a *different* layer's literal key, so a consumer configured + with ``api_key_env`` keeps reading its own variable even when the shared + ``typesafe:`` block also sets ``api_key`` (and its endpoint may not accept that + key at all). Inside one layer ``api_key`` wins over that layer's own + ``api_key_env``, which is what a standalone provider did. ``None`` from the + environment stays a missing credential rather than falling back to another + layer's key, and the built-in ``TYPESAFE_API_KEY`` is consulted only when no + layer names a credential. + """ + for layer in (override, fallback): + if "api_key" in layer: + return layer["api_key"] + name = layer.get("api_key_env") + if isinstance(name, str) and name: + return os.environ.get(name) + return os.environ.get(default_env) + + +def _validated_base_url(value: object) -> str: + """Return a base URL a request can use, or raise at construction. + + A base that is not ``http(s)``, or that carries a query, fragment, embedded + credentials, whitespace or a control character, builds fine today and then fails + on every request: httpx rejects the scheme or the URL per call + (``UnsupportedProtocol`` / ``InvalidURL``), and the query would be silently + swallowed into the endpoint path this layer appends. Embedded credentials are + refused as well because the base URL is published in every consumer's policy + identity and reaches reason messages. + """ + if not isinstance(value, str) or not value.strip(): + raise ValueError("base_url must be a non-empty string") + if any(character.isspace() or ord(character) < 0x21 or ord(character) == 0x7F for character in value): + raise ValueError("base_url must not contain whitespace or control characters") + try: + parts = urlsplit(value) + parts.port # a malformed port is a per-request httpx error; fail it here instead + except ValueError as exc: + raise ValueError(f"base_url is not a usable URL: {exc}") from None + if parts.scheme not in _BASE_URL_SCHEMES or not parts.netloc: + raise ValueError("base_url must be an absolute http:// or https:// URL") + if parts.query or parts.fragment or "@" in parts.netloc: + raise ValueError("base_url must not carry a query, fragment or credentials") + return value + + +__all__ = [ + "CONNECTION_FIELDS", + "DEFAULT_API_KEY_ENV", + "DEFAULT_BASE_URL", + "DEFAULT_DEADLINE_SECONDS", + "DEFAULT_MAX_ATTEMPTS", + "DEFAULT_MODEL", + "DEFAULT_RETRY_BACKOFF", + "DEFAULT_TIMEOUT", + "ENDPOINT_PATH", + "MODE_OFF", + "TypeSafeConnection", + "resolve_connection", + "resolve_connection_for_mode", + "typesafe_defaults", +] diff --git a/backend/packages/harness/deerflow/typesafe/errors.py b/backend/packages/harness/deerflow/typesafe/errors.py new file mode 100644 index 0000000000..da62e59a20 --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/errors.py @@ -0,0 +1,58 @@ +"""The one error taxonomy every TypeSafe (Jev) consumer maps its failures from. + +Two layers, deliberately separate (design §2.3): + +* **Request level** — a transport error, a spent deadline, a non-200 status, or a + response without a usable common envelope. These raise :class:`TypeSafeError` + with one of the four causes below. +* **Question level** — an answer that is missing, has the wrong ``type``, carries + a non-finite or out-of-range probability, or names an unknown ``choice`` label. + These never raise: they come back as data in + ``deerflow.typesafe.client.AnswerSet.errors_by_question`` so one bad question + cannot discard the other valid answers in the same response. + +``cause`` is the machine-readable category. What a cause *means* for execution is +the consumer's decision and stays there: the tool gate denies, the memory paths +fall back to their normal behaviour. +""" + +from __future__ import annotations + +#: The request never reached a usable response: DNS, connect, read, protocol. +CAUSE_TRANSPORT = "transport" +#: The endpoint answered, but not with ``200``. +CAUSE_HTTP_STATUS = "http_status" +#: The endpoint answered ``200`` with a body that is not a usable envelope. +CAUSE_INVALID_RESPONSE = "invalid_response" +#: The evaluation budget was spent before a usable result could be accepted. +CAUSE_DEADLINE = "deadline" + +CAUSES = frozenset({CAUSE_TRANSPORT, CAUSE_HTTP_STATUS, CAUSE_INVALID_RESPONSE, CAUSE_DEADLINE}) + + +class TypeSafeError(RuntimeError): + """A TypeSafe request could not produce a usable answer set. + + Raised for request-level failures only. Consumers translate it into their own + failure policy — the tool gate turns it into a fail-closed denial, the memory + paths extract as usual — and never treat it as a verdict. + + The message never contains the credential, the request state, or any part of + the response body: a malformed or hostile endpoint can echo the state back, + and these messages reach middleware logs, the run journal and the evaluation + report. + """ + + def __init__(self, message: str, *, cause: str) -> None: + super().__init__(message) + self.cause = cause + + +__all__ = [ + "CAUSE_DEADLINE", + "CAUSE_HTTP_STATUS", + "CAUSE_INVALID_RESPONSE", + "CAUSE_TRANSPORT", + "CAUSES", + "TypeSafeError", +] diff --git a/backend/packages/harness/deerflow/typesafe/validation.py b/backend/packages/harness/deerflow/typesafe/validation.py new file mode 100644 index 0000000000..0c8d5084de --- /dev/null +++ b/backend/packages/harness/deerflow/typesafe/validation.py @@ -0,0 +1,91 @@ +"""Eager configuration-value validation shared by the client and its consumers. + +A bad deployment must fail when the provider is constructed, not on the first +tool call or the first memory write. These helpers reject the shapes JSON and +YAML make easy to get wrong — ``bool`` where a number is meant (Python's ``bool`` +is an ``int`` subclass), ``NaN`` from a JSON literal, a float where a count is +meant — and their messages name the offending field without echoing the value's +type-driven surprises. +""" + +from __future__ import annotations + +import math +from collections.abc import Mapping + + +def finite_float( + name: str, + value: object, + *, + minimum: float | None = None, + maximum: float | None = None, + exclusive: bool = False, +) -> float: + """Return ``value`` as a finite float, or raise ``ValueError`` naming ``name``.""" + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ValueError(f"{name} must be a finite number, got {value!r}") + number = float(value) + if not math.isfinite(number): + raise ValueError(f"{name} must be a finite number, got {value!r}") + if minimum is not None and (number <= minimum if exclusive else number < minimum): + raise ValueError(f"{name} must be {'>' if exclusive else '>='} {minimum:g}, got {number!r}") + if maximum is not None and number > maximum: + raise ValueError(f"{name} must be <= {maximum:g}, got {number!r}") + return number + + +def whole_number(name: str, value: object, *, minimum: int) -> int: + """Return ``value`` as an int, rejecting ``bool`` and non-integers.""" + if isinstance(value, bool) or not isinstance(value, int): + raise ValueError(f"{name} must be an integer, got {value!r}") + if value < minimum: + raise ValueError(f"{name} must be >= {minimum}, got {value!r}") + return value + + +def defaulted_text(name: str, value: object, fallback: str) -> str: + """Return ``value`` when configured, else ``fallback``; blank text is a config error.""" + if value is None: + return fallback + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{name} must be a non-empty string") + return value + + +def credential_text(name: str, value: str) -> str: + """Return a credential that can be sent as a header value, or raise naming ``name``. + + A key with surrounding whitespace (a secret mounted from a file, a quoted + ``.env`` value, a shell ``$(cat ...)``) or a non-printable character passes the + old non-empty check and then reaches h11, whose ``LocalProtocolError`` message + carries the whole ``Bearer `` header — and the guardrail middleware's + ``logger.exception`` prints that cause chain. Rejecting the shape here turns a + per-call credential leak into a construction-time error. + + Interior spaces are allowed because they are legal in a header value and leak + nothing; every character outside printable ASCII (plus leading and trailing + whitespace) is rejected. The message never echoes ``value``. + """ + if value != value.strip() or any(not 0x20 <= ord(character) <= 0x7E for character in value): + raise ValueError(f"{name} must be printable ASCII with no leading or trailing whitespace: check the configured key and the environment variable it reads (the value is never echoed here or in any error message)") + return value + + +def criteria_entry(criteria: Mapping[object, object] | None, flag: bool) -> object: + """Look up a ``criteria`` entry under either the YAML or the JSON spelling. + + A YAML ``criteria: {true: ..., false: ...}`` block parses its keys as booleans, + while a JSON-typed config keeps them as the strings ``"true"``/``"false"``. + """ + if criteria is None: + return None + if not isinstance(criteria, Mapping): + raise ValueError("criteria must be a mapping with optional 'true'/'false' entries") + for key in (flag, "true" if flag else "false"): + if key in criteria: + return criteria[key] + return None + + +__all__ = ["credential_text", "criteria_entry", "defaulted_text", "finite_float", "whole_number"] diff --git a/backend/scripts/eval_memory_prescreen.py b/backend/scripts/eval_memory_prescreen.py new file mode 100644 index 0000000000..1ab16adf93 --- /dev/null +++ b/backend/scripts/eval_memory_prescreen.py @@ -0,0 +1,1089 @@ +"""Shadow evaluation of the memory pre-screen (Jev) from DeerMem extraction records. + +This is the pre-``enforce`` evidence for ``TypeSafeMemoryPrescreen``: the gates +documented in ``backend/docs/MEMORY_IMPROVEMENTS.md`` ("Before ``enforce``") and in +``agents/memory/AGENTS.md``. It is an operator run over **shadow records**, not a CI +test: in ``mode: shadow`` the extraction call always runs, so every pre-screen +verdict comes with free ground truth — that same batch's extraction outcome. The +script makes no network call, reads no credential and prints none; its input is a +JSONL of DeerMem ``extraction_callback`` payloads, one per line. + +What one line must contain (the payload the host already emits) +---------------------------------------------------------------- + +* ``prescreen`` — ``mode``/``verdict``/``probability``/``skip_threshold``/``model``/ + ``cached``/``digest``/``signals``/``message_count``/``batch_chars``/ + ``duration_ms``/``fallback_reason`` (the judge record, design §5); +* ``success`` and ``token_usage`` — the extraction call's own record; +* ``mutations_accepted`` — the pre-screen's "worth remembering" signal, written + when the apply ran. ``> 0`` on a ``skip`` verdict means the skip would have lost + a memory. + +Fields the collector must add (absent from the production payload today) +----------------------------------------------------------------------- + +* ``trivial_only`` (bool) — the batch's turns were all trivial (``filter_trivial``). + The mandatory no-network baseline cannot be computed without it; coverage is + reported instead of assumed. +* ``reviewed`` (bool) and optionally ``review_outcome`` (``worth_remembering`` | + ``none_worth_remembering``) — the manual review that §5 gate 3 counts *and* + whose verdict that gate reads: the review must confirm none of the reviewed + skips was worth remembering, so a reviewed skip whose outcome (or + ``mutations_accepted`` counter) says otherwise fails the gate. It is also a + second ground-truth witness for a skip. +* ``stratum`` (string or list) / ``labels`` (list) — the §5 gate 2 subset, one of + ``identity`` / ``preference`` / ``correction``. A human label, because L3 keeps + signal-bearing batches out of the network population. +* ``cost_usd`` (float, optional) — the record's billed cost, when known. + +Method, and the things it deliberately refuses to do +---------------------------------------------------- + +* Every record is classified into exactly one population: **network verdicts** (a + request was really sent), **cache hits** (the same ``digest`` reused), **local + fallbacks** (L3 / L4 / L5 / L7 / L8 or a failure fallback) and **unjudged** (no + ``prescreen`` record was emitted at all: no side was enabled, or the round was + forbidden to judge). Only network verdicts are + classifier samples; mixing populations would let a cache hit or a local refusal + vote on the model's accuracy. +* ``saved_call_rate`` (a skip that kept nothing, i.e. the call was really saved) + and ``miss_rate`` (a skip that lost a memory) share the scored-skip denominator + and therefore sum to 1; ``saved_call_rate_over_all_verdicts`` keeps the + network-verdict denominator for the "how many calls did this save" reading. +* A record whose outcome is unknown (skip without ``mutations_accepted``, an + extraction that failed before the apply) is **censored**: counted, reasoned + about and excluded from the rates. It is never dropped to claim a pass. +* The no-network heuristic baseline is run over the same scored records as the + model, so both rates have one denominator: the baseline skips a batch when + ``signals`` is empty and ``trivial_only`` is true. If it already captures most + of the saving, a third-party egress path is not justified. §5 gate 4 needs it + *and* the recorded tokens and p50/p95 verdict latency: recorded calls alone are + not the evidence the gate asks for, so missing any of them is ``INSUFFICIENT`` + rather than a pass. A baseline that saves as many calls as the model — which the + shipped "everything trivial" sample does — is a ``FAIL``, not an enabling run, and + the comparison must cover the scored skips: both sides are the records the baseline + can actually score (``trivial_only`` plus a readable signals list), so a run that + annotates only a subset is ``INSUFFICIENT`` instead of crediting the model with + batches the heuristic never saw. +* The miss-rate bound is the exact one-sided **Clopper-Pearson** interval, + computed here from the regularized incomplete beta function by bisection. No + normal approximation is used anywhere, and the gate reads the *bound*, not the + point estimate: a sample whose observed rate is at or below the target while its + upper bound is above it is ``INSUFFICIENT`` — the evidence does not establish the + target yet, so the answer is to collect more skips, not to enable. +* A count field that arrives negative (``mutations_accepted``, ``message_count``, + ``batch_chars``, a token count) is treated as absent: a count cannot be negative, + and scoring a corrupt one as a saving would tilt every rate toward a pass. +* The exit code carries the verdict: ``0`` only when every gate is ``PASS``, + ``1`` when any gate is ``FAIL`` or ``INSUFFICIENT`` — so an operator can gate + enablement on ``python scripts/eval_memory_prescreen.py --records ... && ``. + +Run from the ``backend/`` directory:: + + PYTHONPATH=. uv run python scripts/eval_memory_prescreen.py --records /tmp/shadow.jsonl --json /tmp/prescreen-report.json +""" + +from __future__ import annotations + +import argparse +import json +import math +import statistics +from collections import Counter +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +NETWORK = "network" +CACHE_HIT = "cache_hit" +LOCAL_FALLBACK = "local_fallback" +UNJUDGED = "unjudged" +#: Every input record lands in exactly one of these. +POPULATIONS: tuple[str, ...] = (NETWORK, CACHE_HIT, LOCAL_FALLBACK, UNJUDGED) +#: The three populations design §5 defines; ``UNJUDGED`` is the coverage remainder. +DESIGN_POPULATIONS: tuple[str, ...] = (NETWORK, CACHE_HIT, LOCAL_FALLBACK) + +VERDICT_SKIP = "skip" +VERDICT_EXTRACT = "extract" + +PASS = "PASS" +FAIL = "FAIL" +INSUFFICIENT = "INSUFFICIENT" + +REVIEW_WORTH_REMEMBERING = "worth_remembering" +REVIEW_NONE_WORTH_REMEMBERING = "none_worth_remembering" +REVIEW_OUTCOMES: tuple[str, ...] = (REVIEW_WORTH_REMEMBERING, REVIEW_NONE_WORTH_REMEMBERING) + +#: The §5 gate 2 subset. Labels are collector-supplied human labels. +SENSITIVE_LABELS = frozenset({"identity", "preference", "correction"}) + +#: The coordinator's fallback reason -> the design lock it discharges. +FALLBACK_LOCKS: dict[str, str] = { + "deterministic_signals": "L3", + "emergency_flush": "L4", + "over_limit": "L5", + "shutdown_drain": "L7", + "staleness_or_consolidation": "L8", + "request_failed": "failure", + "no_verdict": "failure", + "disabled": "off", +} +UNRECORDED_REASON = "(unrecorded)" + +CENSORED_EXTRACTION_FAILED = "extraction_failed" +CENSORED_OUTCOME_NOT_RECORDED = "outcome_not_recorded" + +DEFAULT_MISS_TARGET = 0.01 +DEFAULT_CONFIDENCE = 0.95 +DEFAULT_MIN_REVIEWED_SKIPS = 200 + +_BETA_MAX_ITERATIONS = 300 +_BETA_EPSILON = 3e-16 +_BETA_TINY = 1e-300 +_BETA_BISECTIONS = 200 + + +def _as_count(value: Any) -> int | None: + """A count, refusing ``bool`` and negatives. + + ``True`` is not a count of mutations, and a negative count is a corrupt line: the + record must lose the field rather than be scored as a saving (a negative + ``mutations_accepted`` used to read as "no loss", which diluted the miss-rate + denominator and inflated ``saved_calls``). + """ + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + return None + return value + + +def _as_float(value: Any) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + number = float(value) + return number if math.isfinite(number) else None + + +def _as_bool(value: Any) -> bool | None: + return value if isinstance(value, bool) else None + + +def _first_count(mapping: Mapping[str, Any], keys: Sequence[str]) -> int | None: + for key in keys: + value = _as_count(mapping.get(key)) + if value is not None: + return value + return None + + +def _token_counts(usage: Any) -> tuple[int | None, int | None, int | None]: + """Read input / output / total tokens from a LangChain-style ``usage_metadata``.""" + if not isinstance(usage, Mapping): + return (None, None, None) + input_tokens = _first_count(usage, ("input_tokens", "prompt_tokens")) + output_tokens = _first_count(usage, ("output_tokens", "completion_tokens")) + total = _as_count(usage.get("total_tokens")) + if total is None and (input_tokens is not None or output_tokens is not None): + total = (input_tokens or 0) + (output_tokens or 0) + return (input_tokens, output_tokens, total) + + +def _normalize_review_outcome(value: Any) -> str | None: + if not isinstance(value, str): + return None + text = value.strip().lower() + return text if text in REVIEW_OUTCOMES else None + + +def _sensitive_labels(raw: Mapping[str, Any]) -> frozenset[str]: + """The record's ``identity`` / ``preference`` / ``correction`` labels, normalised.""" + found: set[str] = set() + for key in ("stratum", "labels"): + value = raw.get(key) + if isinstance(value, str): + candidates = [value] + elif isinstance(value, (list, tuple, set, frozenset)): + candidates = [item for item in value if isinstance(item, str)] + else: + continue + for item in candidates: + label = item.strip().lower() + if label in SENSITIVE_LABELS: + found.add(label) + return frozenset(found) + + +@dataclass(frozen=True) +class Prices: + """Operator-supplied list price, in USD per million tokens (``None`` = unpriced).""" + + input_per_mtok: float | None = None + output_per_mtok: float | None = None + + @property + def configured(self) -> bool: + return self.input_per_mtok is not None or self.output_per_mtok is not None + + def estimate(self, input_tokens: int | None, output_tokens: int | None) -> float | None: + if not self.configured or (input_tokens is None and output_tokens is None): + return None + dollars = (input_tokens or 0) * (self.input_per_mtok or 0.0) + (output_tokens or 0) * (self.output_per_mtok or 0.0) + return dollars / 1_000_000.0 + + def as_dict(self) -> dict[str, Any]: + return {"input_per_mtok": self.input_per_mtok, "output_per_mtok": self.output_per_mtok} + + +@dataclass(frozen=True) +class Record: + """One shadow record, reduced to what the evaluation reads.""" + + index: int + population: str + verdict: str | None + fallback_reason: str | None + mode: str | None + model: str | None + cached: bool | None + probability: float | None + skip_threshold: float | None + digest: str | None + duration_ms: float | None + signals: frozenset[str] + signals_known: bool + message_count: int | None + batch_chars: int | None + success: bool | None + mutations_accepted: int | None + input_tokens: int | None + output_tokens: int | None + total_tokens: int | None + cost_usd: float | None + trivial_only: bool | None + reviewed: bool + review_outcome: str | None + sensitive_labels: frozenset[str] + censored: bool + censored_reason: str | None + + @property + def skip(self) -> bool: + return self.verdict == VERDICT_SKIP + + @property + def sensitive(self) -> bool: + return bool(self.sensitive_labels) + + @property + def scorable(self) -> bool: + """Whether "was this batch worth remembering?" has an answer at all.""" + return self.mutations_accepted is not None or self.review_outcome is not None + + @property + def miss(self) -> bool | None: + """Whether a skip here would have lost a memory; ``None`` when not scorable. + + Either witness marks a miss: the ``mutations_accepted`` counter (design + §5's definition) or a human ``review_outcome``. A review saying "not worth + remembering" never clears a counter that says otherwise — an enablement + gate reads the more conservative of the two. + """ + if not self.scorable: + return None + if self.review_outcome == REVIEW_WORTH_REMEMBERING: + return True + return self.mutations_accepted is not None and self.mutations_accepted > 0 + + +def classify_population(record: Mapping[str, Any]) -> str: + """Place one raw record in exactly one population (design §5's table). + + A usable verdict means an answer was consumed: it is a cache hit when the + adapter says so and a network verdict otherwise. No usable verdict means no + request produced one — L3 / L4 / L5 / L7 / L8 or a failure fallback. No + ``prescreen`` record at all means no side was enabled for this batch, or the + round was forbidden to judge: an enabled side always emits its record, even + when that record is a fallback. + """ + prescreen = record.get("prescreen") + if not isinstance(prescreen, Mapping): + return UNJUDGED + verdict = prescreen.get("verdict") + if verdict not in (VERDICT_SKIP, VERDICT_EXTRACT): + return LOCAL_FALLBACK + return CACHE_HIT if prescreen.get("cached") is True else NETWORK + + +def _censored_reason(population: str, mutations_accepted: int | None, review_outcome: str | None, success: bool | None) -> str | None: + """Why this sample has no outcome to score, or ``None`` when it has one.""" + if population not in (NETWORK, CACHE_HIT): + return None + if mutations_accepted is not None or review_outcome is not None: + return None + if success is False: + return CENSORED_EXTRACTION_FAILED + return CENSORED_OUTCOME_NOT_RECORDED + + +def parse_record(index: int, raw: Mapping[str, Any]) -> Record: + """Reduce one payload to the fields the evaluation reads; never raises on shape.""" + block = raw.get("prescreen") + block = block if isinstance(block, Mapping) else {} + population = classify_population(raw) + raw_verdict = block.get("verdict") + verdict = raw_verdict if raw_verdict in (VERDICT_SKIP, VERDICT_EXTRACT) else None + review_outcome = _normalize_review_outcome(raw.get("review_outcome")) + mutations_accepted = _as_count(raw.get("mutations_accepted")) + success = _as_bool(raw.get("success")) + input_tokens, output_tokens, total_tokens = _token_counts(raw.get("token_usage")) + raw_signals = block.get("signals") + signals_known = isinstance(raw_signals, list) + signals = frozenset(str(item) for item in raw_signals) if signals_known else frozenset() + digest = block.get("digest") + censored_reason = _censored_reason(population, mutations_accepted, review_outcome, success) + return Record( + index=index, + population=population, + verdict=verdict, + fallback_reason=block.get("fallback_reason") if isinstance(block.get("fallback_reason"), str) else None, + mode=block.get("mode") if isinstance(block.get("mode"), str) else None, + model=block.get("model") if isinstance(block.get("model"), str) else None, + cached=_as_bool(block.get("cached")), + probability=_as_float(block.get("probability")), + skip_threshold=_as_float(block.get("skip_threshold")), + digest=digest if isinstance(digest, str) and digest else None, + duration_ms=_as_float(block.get("duration_ms")), + signals=signals, + signals_known=signals_known, + message_count=_as_count(block.get("message_count")), + batch_chars=_as_count(block.get("batch_chars")), + success=success, + mutations_accepted=mutations_accepted, + input_tokens=input_tokens, + output_tokens=output_tokens, + total_tokens=total_tokens, + cost_usd=_as_float(raw.get("cost_usd")), + trivial_only=_as_bool(raw.get("trivial_only")), + reviewed=_as_bool(raw.get("reviewed")) is True or review_outcome is not None, + review_outcome=review_outcome, + sensitive_labels=_sensitive_labels(raw), + censored=censored_reason is not None, + censored_reason=censored_reason, + ) + + +def _dig(record: Mapping[str, Any], path: str) -> Any: + """Resolve a dotted ``--label`` path (``prescreen.mode``) against a raw record.""" + current: Any = record + for part in path.split("."): + if not isinstance(current, Mapping) or part not in current: + return None + current = current[part] + return current + + +def record_matches(record: Mapping[str, Any], filters: Sequence[tuple[str, str]]) -> bool: + """Whether every ``--label KEY=VALUE`` filter matches; strings compare as text.""" + for path, expected in filters: + value = _dig(record, path) + if isinstance(value, str): + if value != expected: + return False + continue + try: + parsed = json.loads(expected) + except json.JSONDecodeError: + return False + if value != parsed: + return False + return True + + +def parse_label_filter(text: str) -> tuple[str, str]: + """``KEY=VALUE`` (dotted keys allowed) for ``--label``; argparse error otherwise.""" + key, separator, value = text.partition("=") + if not separator or not key.strip(): + raise argparse.ArgumentTypeError(f"--label expects KEY=VALUE (e.g. prescreen.mode=shadow), got {text!r}") + return key.strip(), value + + +@dataclass(frozen=True) +class LoadResult: + records: tuple[Record, ...] + read: int + filtered_out: int + malformed_lines: int + blank_lines: int + + +def load_records(path: Path, filters: Sequence[tuple[str, str]] = ()) -> LoadResult: + """Read the JSONL, apply ``--label`` filtering and parse each line. + + A line that is not a JSON object is *counted*, not dropped: gate D fails the + run rather than silently shrinking the denominator. + """ + records: list[Record] = [] + malformed = 0 + blank = 0 + read = 0 + filtered = 0 + for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1): + if not line.strip(): + blank += 1 + continue + try: + raw = json.loads(line) + except json.JSONDecodeError: + malformed += 1 + continue + if not isinstance(raw, Mapping): + malformed += 1 + continue + read += 1 + if filters and not record_matches(raw, filters): + filtered += 1 + continue + records.append(parse_record(number, raw)) + return LoadResult(records=tuple(records), read=read, filtered_out=filtered, malformed_lines=malformed, blank_lines=blank) + + +def _rate(numerator: int, denominator: int) -> float | None: + return None if denominator <= 0 else numerator / denominator + + +def _percentile(values: Sequence[float], fraction: float) -> float | None: + """Nearest-rank quantile: the smallest sample at or above ``fraction`` of the mass.""" + if not values: + return None + ordered = sorted(values) + return ordered[max(0, math.ceil(fraction * len(ordered)) - 1)] + + +def _latency(values: Sequence[float]) -> dict[str, Any]: + if not values: + return {"samples": 0, "p50": None, "p95": None} + return {"samples": len(values), "p50": statistics.median(values), "p95": _percentile(values, 0.95)} + + +def _token_totals(rows: Sequence[Record]) -> dict[str, Any]: + measured = [record for record in rows if record.input_tokens is not None or record.output_tokens is not None or record.total_tokens is not None] + return { + "records": len(measured), + "input": sum(record.input_tokens or 0 for record in measured), + "output": sum(record.output_tokens or 0 for record in measured), + "total": sum(record.total_tokens if record.total_tokens is not None else (record.input_tokens or 0) + (record.output_tokens or 0) for record in measured), + } + + +def _record_cost(record: Record, prices: Prices) -> float | None: + if record.cost_usd is not None: + return record.cost_usd + return prices.estimate(record.input_tokens, record.output_tokens) + + +def _cost_totals(rows: Sequence[Record], prices: Prices) -> dict[str, Any]: + costs = [_record_cost(record, prices) for record in rows] + priced = [cost for cost in costs if cost is not None] + if prices.configured: + pricing = "cli price per million tokens" + elif priced: + pricing = "record cost_usd" + else: + pricing = "not configured" + return {"total_usd": sum(priced) if priced else None, "priced_records": len(priced), "unpriced_records": len(costs) - len(priced), "pricing": pricing} + + +def summarize_population(records: Sequence[Record], population: str, prices: Prices = Prices()) -> dict[str, Any]: + """One population's table row: counts, decision rates, latency, tokens and cost.""" + rows = [record for record in records if record.population == population] + skips = [record for record in rows if record.skip] + extracts = [record for record in rows if record.verdict == VERDICT_EXTRACT] + scored_skips = [record for record in skips if record.scorable] + saved = [record for record in scored_skips if record.miss is False] + misses = [record for record in scored_skips if record.miss is True] + reason_rows = [record.fallback_reason or UNRECORDED_REASON for record in rows if record.population == LOCAL_FALLBACK or record.fallback_reason is not None] + summary: dict[str, Any] = { + "records": len(rows), + "censored": sum(1 for record in rows if record.censored), + "censored_reasons": dict(Counter(record.censored_reason for record in rows if record.censored_reason)), + "verdicts": {"skip": len(skips), "extract": len(extracts), "none": len(rows) - len(skips) - len(extracts)}, + "cached": { + "true": sum(1 for record in rows if record.cached is True), + "false": sum(1 for record in rows if record.cached is False), + "unknown": sum(1 for record in rows if record.cached is None), + }, + "skip_rate": _rate(len(skips), len(rows)), + "scored_skips": len(scored_skips), + "saved_calls": len(saved), + "misses": len(misses), + "saved_call_rate": _rate(len(saved), len(scored_skips)), + "miss_rate": _rate(len(misses), len(scored_skips)), + "saved_call_rate_over_all_verdicts": _rate(len(saved), len(rows)), + "missed_digests": [record.digest for record in misses if record.digest], + "latency_ms": _latency([record.duration_ms for record in rows if record.duration_ms is not None]), + "tokens": _token_totals(rows), + "cost": _cost_totals(rows, prices), + } + if reason_rows: + summary["fallback_reasons"] = dict(Counter(reason_rows)) + summary["fallback_locks"] = dict(Counter(FALLBACK_LOCKS.get(reason, "unknown") for reason in reason_rows)) + return summary + + +def _beta_continued_fraction(a: float, b: float, x: float) -> float: + """Lentz's continued fraction for the incomplete beta function (Numerical Recipes).""" + qab, qap, qam = a + b, a + 1.0, a - 1.0 + c = 1.0 + d = 1.0 - qab * x / qap + if abs(d) < _BETA_TINY: + d = _BETA_TINY + d = 1.0 / d + h = d + for m in range(1, _BETA_MAX_ITERATIONS + 1): + m2 = 2 * m + numerator = m * (b - m) * x / ((qam + m2) * (a + m2)) + d = 1.0 + numerator * d + if abs(d) < _BETA_TINY: + d = _BETA_TINY + c = 1.0 + numerator / c + if abs(c) < _BETA_TINY: + c = _BETA_TINY + d = 1.0 / d + h *= d * c + numerator = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2)) + d = 1.0 + numerator * d + if abs(d) < _BETA_TINY: + d = _BETA_TINY + c = 1.0 + numerator / c + if abs(c) < _BETA_TINY: + c = _BETA_TINY + d = 1.0 / d + delta = d * c + h *= delta + if abs(delta - 1.0) < _BETA_EPSILON: + break + return h + + +def _regularized_incomplete_beta(a: float, b: float, x: float) -> float: + """``I_x(a, b)``, the regularized incomplete beta function.""" + if x <= 0.0: + return 0.0 + if x >= 1.0: + return 1.0 + front = math.exp(math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b) + a * math.log(x) + b * math.log1p(-x)) + if x < (a + 1.0) / (a + b + 2.0): + return front * _beta_continued_fraction(a, b, x) / a + return 1.0 - front * _beta_continued_fraction(b, a, 1.0 - x) / b + + +def _beta_quantile(probability: float, a: float, b: float) -> float: + """Inverse of :func:`_regularized_incomplete_beta` in ``x`` (bisection; monotone).""" + if probability <= 0.0: + return 0.0 + if probability >= 1.0: + return 1.0 + low, high = 0.0, 1.0 + for _ in range(_BETA_BISECTIONS): + middle = (low + high) / 2.0 + if _regularized_incomplete_beta(a, b, middle) < probability: + low = middle + else: + high = middle + return (low + high) / 2.0 + + +def clopper_pearson_upper(misses: int, skips: int, confidence: float = DEFAULT_CONFIDENCE) -> float | None: + """Exact one-sided Clopper-Pearson upper bound for the miss rate. + + Solves ``P(X <= misses) = 1 - confidence`` for the proportion ``p``, i.e. + ``I_{1-p}(skips - misses, misses + 1) = 1 - confidence``, by bisection on the + regularized incomplete beta. This is the textbook exact bound: no normal + approximation is used anywhere in this script. ``None`` when there is no + sample, ``1.0`` when every skip was a miss. + """ + if skips <= 0: + return None + if not 0.0 < confidence < 1.0: + raise ValueError(f"confidence must be in (0, 1), got {confidence!r}") + misses = max(0, min(misses, skips)) + if misses >= skips: + return 1.0 + bound = 1.0 - _beta_quantile(1.0 - confidence, float(skips - misses), float(misses + 1)) + return min(1.0, max(0.0, bound)) + + +def baseline_summary(records: Sequence[Record], prices: Prices = Prices()) -> dict[str, Any]: + """The mandatory no-network baseline, over the same scored records as the model. + + A batch is a baseline skip when it carries no deterministic signal and the + collector marked it ``trivial_only``. Records without ``trivial_only`` (or + without a readable ``signals`` list) cannot be scored by the heuristic: the + coverage is reported instead of silently shrinking the denominator. + """ + evaluated = [record for record in records if record.population == NETWORK] + computable = [record for record in evaluated if record.trivial_only is not None and record.signals_known] + scored = [record for record in computable if record.scorable] + baseline_skips = [record for record in scored if record.trivial_only and not record.signals] + baseline_saved = [record for record in baseline_skips if record.miss is False] + baseline_misses = [record for record in baseline_skips if record.miss is True] + model_skips = [record for record in scored if record.skip] + model_saved = [record for record in model_skips if record.miss is False] + model_misses = [record for record in model_skips if record.miss is True] + baseline_saved_rate = _rate(len(baseline_saved), len(baseline_skips)) + model_saved_rate = _rate(len(model_saved), len(model_skips)) + return { + "rule": "skip when signals is empty and trivial_only is true", + "evaluated_verdicts": len(evaluated), + "computable": len(computable), + "coverage": _rate(len(computable), len(evaluated)), + "scored": len(scored), + "skips": len(baseline_skips), + "saved_calls": len(baseline_saved), + "misses": len(baseline_misses), + "saved_call_rate": baseline_saved_rate, + "miss_rate": _rate(len(baseline_misses), len(baseline_skips)), + "model": { + "skips": len(model_skips), + "saved_calls": len(model_saved), + "misses": len(model_misses), + "saved_call_rate": model_saved_rate, + "miss_rate": _rate(len(model_misses), len(model_skips)), + }, + "incremental_saved_calls": len(model_saved) - len(baseline_saved), + # Difference of saved calls over the shared denominator. Subtracting the + # conditional rates (saved/skips) would report zero whenever both policies + # saved every batch they skipped, even if the model skipped far more. + "incremental_saved_call_rate": _rate(len(model_saved) - len(baseline_saved), len(scored)), + "trivial_only_missing": sum(1 for record in evaluated if record.trivial_only is None), + "signals_missing": sum(1 for record in evaluated if not record.signals_known), + "note": "Both sides are scored over the same scored records; a baseline that nearly matches the model means the third-party egress buys little.", + "cost": _cost_totals(scored, prices), + } + + +@dataclass(frozen=True) +class Gate: + gate_id: str + status: str + detail: str + + def as_dict(self) -> dict[str, str]: + return {"id": self.gate_id, "status": self.status, "detail": self.detail} + + +@dataclass(frozen=True) +class Evidence: + """The numbers the design §5 gates read (network population only).""" + + scored_skips: int + misses: int + saved_calls: int + sensitive_scored_skips: Mapping[str, int] + sensitive_misses: Mapping[str, int] + reviewed_skips: int + reviewed_scored_skips: int + reviewed_missed: int + saved_skips_with_tokens: int + latency_samples: int + #: The no-network comparison, all three over the same records: the ones the + #: baseline can score (``trivial_only`` and a readable signals list). Counting + #: the model's savings over *all* scorable skips instead would credit it with + #: batches the baseline never saw. + baseline_computable: int + baseline_paired_skips: int + baseline_model_saved_calls: int + baseline_heuristic_saved_calls: int + + +@dataclass(frozen=True) +class Coverage: + """Accounting for the records that could not be scored.""" + + records: int + censored: int + malformed_lines: int + + +@dataclass(frozen=True) +class Thresholds: + miss_target: float = DEFAULT_MISS_TARGET + confidence: float = DEFAULT_CONFIDENCE + min_reviewed_skips: int = DEFAULT_MIN_REVIEWED_SKIPS + + +def evaluate_gates(evidence: Evidence, coverage: Coverage, thresholds: Thresholds = Thresholds()) -> tuple[Gate, ...]: + """Every gate prints ``PASS`` / ``FAIL`` / ``INSUFFICIENT`` — never silence.""" + gates: list[Gate] = [] + miss_rate = _rate(evidence.misses, evidence.scored_skips) + upper = clopper_pearson_upper(evidence.misses, evidence.scored_skips, thresholds.confidence) + observed = f"miss_rate={evidence.misses}/{evidence.scored_skips}={miss_rate:.4f}" if miss_rate is not None else "" + bound = "no bound (no sample)" if upper is None else f"one-sided {thresholds.confidence:.0%} Clopper-Pearson upper bound {upper:.4f}" + sample = f"sample n={evidence.scored_skips}" + if miss_rate is None: + gates.append(Gate("network_miss_rate", INSUFFICIENT, "no scored skip verdict in the network population; there is no sample for the miss rate or its bound")) + elif miss_rate > thresholds.miss_target: + # The observation itself is over target: more samples cannot rescue it. + gates.append(Gate("network_miss_rate", FAIL, f"{observed} is above the target {thresholds.miss_target:.4f}; {bound} ({sample})")) + elif upper is None or upper > thresholds.miss_target: + # The point estimate meets the target and the exact bound does not establish + # it: the sample is underpowered, so the honest answer is more skips. + gates.append(Gate("network_miss_rate", INSUFFICIENT, f"{observed} meets the target {thresholds.miss_target:.4f}, but {bound} does not ({sample}): too few skips to establish the target")) + else: + gates.append(Gate("network_miss_rate", PASS, f"{observed} target<={thresholds.miss_target:.4f}; {bound} ({sample})")) + + missed_by_label = {label: evidence.sensitive_misses.get(label, 0) for label in sorted(SENSITIVE_LABELS)} + uncovered = [label for label in sorted(SENSITIVE_LABELS) if evidence.sensitive_scored_skips.get(label, 0) == 0] + per_label = "; ".join(f"{label}: {missed_by_label[label]} miss(es) over {evidence.sensitive_scored_skips.get(label, 0)} scored skip(s)" for label in sorted(SENSITIVE_LABELS)) + if any(count > 0 for count in missed_by_label.values()): + gates.append(Gate("sensitive_stratum_zero_misses", FAIL, f"a scored skip in a required stratum was worth remembering ({per_label})")) + elif uncovered: + gates.append(Gate("sensitive_stratum_zero_misses", INSUFFICIENT, f"no scored skip carries the {uncovered} stratum label(s); the zero-miss check needs evidence for every required stratum {sorted(SENSITIVE_LABELS)} ({per_label})")) + else: + gates.append(Gate("sensitive_stratum_zero_misses", PASS, f"zero misses in every required stratum ({per_label})")) + + if evidence.reviewed_skips == 0: + gates.append(Gate("reviewed_skips_confirm_no_loss", INSUFFICIENT, f"no skip verdict is labelled reviewed; the collector must set reviewed=true / review_outcome (target >={thresholds.min_reviewed_skips})")) + elif evidence.reviewed_missed: + gates.append(Gate("reviewed_skips_confirm_no_loss", FAIL, f"{evidence.reviewed_missed} reviewed skip verdict(s) were worth remembering; design §5 gate 3 is a review *confirming* none was, so the review sample contradicts the skip")) + elif evidence.reviewed_skips < thresholds.min_reviewed_skips: + gates.append(Gate("reviewed_skips_confirm_no_loss", FAIL, f"{evidence.reviewed_skips} reviewed skip verdict(s) of {thresholds.min_reviewed_skips} required ({evidence.reviewed_scored_skips} of them scored)")) + elif evidence.reviewed_scored_skips < evidence.reviewed_skips: + gates.append( + Gate( + "reviewed_skips_confirm_no_loss", + INSUFFICIENT, + f"{evidence.reviewed_skips - evidence.reviewed_scored_skips} of {evidence.reviewed_skips} reviewed skip(s) carry neither a review_outcome nor mutations_accepted, so the review confirms nothing for them", + ) + ) + else: + gates.append(Gate("reviewed_skips_confirm_no_loss", PASS, f"{evidence.reviewed_skips} reviewed skip verdict(s), all scored, none worth remembering ({thresholds.min_reviewed_skips} required)")) + + if coverage.records == 0: + gates.append(Gate("censoring_disclosed", INSUFFICIENT, "no record was read; there is nothing to account for")) + elif coverage.malformed_lines > 0: + gates.append(Gate("censoring_disclosed", FAIL, f"{coverage.malformed_lines} input line(s) were not parseable JSON objects; they were counted but must not be dropped from the denominator")) + else: + gates.append(Gate("censoring_disclosed", PASS, f"{coverage.censored} censored sample(s) of {coverage.records} reported with their reasons; none dropped")) + + if evidence.scored_skips == 0: + gates.append(Gate("savings_recorded", INSUFFICIENT, "no scored skip verdict, so no saved extraction call is recorded")) + elif evidence.saved_calls == 0: + gates.append(Gate("savings_recorded", FAIL, f"no scored skip saved an extraction call over {evidence.scored_skips} scored skip(s); the pre-screen as recorded saves nothing")) + else: + absent = [ + name + for name, count in ( + ("tokens on the saved skips", evidence.saved_skips_with_tokens), + ("verdict latency", evidence.latency_samples), + ("the no-network baseline (collector field trivial_only)", evidence.baseline_computable), + ) + if count == 0 + ] + if absent: + gates.append( + Gate( + "savings_recorded", + INSUFFICIENT, + f"{evidence.saved_calls} extraction call(s) recorded saved, but the evidence is incomplete: nothing recorded for " + + "; ".join(absent) + + " - design §5 gate 4 asks for recorded calls/tokens saved, their p50/p95 latency cost, and the mandatory baseline", + ) + ) + else: + recorded = f"{evidence.saved_calls} extraction call(s) recorded saved over {evidence.scored_skips} scored skip(s)" + observed = f"tokens on {evidence.saved_skips_with_tokens} of them, {evidence.latency_samples} verdict latency sample(s), baseline on {evidence.baseline_computable} record(s)" + gates.append(Gate("savings_recorded", PASS, f"{recorded}; {observed}")) + + # Both sides of this comparison are the baseline-computable records, and the gate + # refuses to conclude from a subset of the scored skips: subtracting a subset's + # heuristic savings from model savings over all scorable skips would count every + # unannotated batch as imaginary incremental benefit. + paired = evidence.baseline_paired_skips + incremental = evidence.baseline_model_saved_calls - evidence.baseline_heuristic_saved_calls + if paired == 0: + gates.append( + Gate( + "baseline_incremental_savings", + INSUFFICIENT, + "no scored skip carries the baseline annotation (the collector must set trivial_only and signals), so there is nothing to compare the model against", + ) + ) + elif paired < evidence.scored_skips: + gates.append( + Gate( + "baseline_incremental_savings", + INSUFFICIENT, + f"the baseline can score {paired} of {evidence.scored_skips} scored skip(s); the comparison must cover the scored skips, not the subset the annotation happens to reach", + ) + ) + elif incremental <= 0: + saved_summary = f"the model saved {evidence.baseline_model_saved_calls} call(s) against the heuristic's {evidence.baseline_heuristic_saved_calls} on the same {paired} scored skip(s): {incremental:+d} incremental" + gates.append(Gate("baseline_incremental_savings", FAIL, f"{saved_summary}, so the egress buys nothing over the heuristic already in the stack")) + else: + gates.append( + Gate( + "baseline_incremental_savings", + PASS, + f"the model saved {incremental} call(s) more than the no-network heuristic on the same {paired} scored skip(s) ({evidence.baseline_model_saved_calls} vs {evidence.baseline_heuristic_saved_calls})", + ) + ) + + return tuple(gates) + + +def build_report( + records: Sequence[Record], + *, + source: str, + malformed_lines: int = 0, + filtered_out: int = 0, + read_lines: int = 0, + blank_lines: int = 0, + thresholds: Thresholds = Thresholds(), + prices: Prices = Prices(), + label_filters: Sequence[tuple[str, str]] = (), +) -> dict[str, Any]: + """Assemble the full report: populations, model, baseline, stratification, gates.""" + populations = {name: summarize_population(records, name, prices) for name in POPULATIONS} + network = [record for record in records if record.population == NETWORK] + scored_skips = [record for record in network if record.skip and record.scorable] + sensitive_scored_skips = {label: sum(1 for record in scored_skips if label in record.sensitive_labels) for label in sorted(SENSITIVE_LABELS)} + sensitive_misses = {label: sum(1 for record in scored_skips if label in record.sensitive_labels and record.miss is True) for label in sorted(SENSITIVE_LABELS)} + reviewed_skips = [record for record in network if record.skip and record.reviewed] + saved = [record for record in scored_skips if record.miss is False] + baseline = baseline_summary(records, prices) + evidence = Evidence( + scored_skips=len(scored_skips), + misses=sum(1 for record in scored_skips if record.miss is True), + saved_calls=len(saved), + sensitive_scored_skips=sensitive_scored_skips, + sensitive_misses=sensitive_misses, + reviewed_skips=len(reviewed_skips), + reviewed_scored_skips=sum(1 for record in reviewed_skips if record.scorable), + reviewed_missed=sum(1 for record in reviewed_skips if record.miss is True), + saved_skips_with_tokens=sum(1 for record in saved if record.total_tokens is not None), + latency_samples=sum(1 for record in network if record.duration_ms is not None), + baseline_computable=baseline["computable"], + baseline_paired_skips=baseline["model"]["skips"], + baseline_model_saved_calls=baseline["model"]["saved_calls"], + baseline_heuristic_saved_calls=baseline["saved_calls"], + ) + coverage = Coverage(records=len(records), censored=sum(1 for record in records if record.censored), malformed_lines=malformed_lines) + gates = evaluate_gates(evidence, coverage, thresholds) + verdicts = populations[NETWORK]["verdicts"] + upper = clopper_pearson_upper(evidence.misses, evidence.scored_skips, thresholds.confidence) + modes = Counter(record.mode for record in records if record.mode) + notes = [ + "One record is exactly one population: network verdicts are the only classifier samples; cache hits reuse another verdict and local fallbacks never asked.", + "saved_call_rate and miss_rate share the scored-skip denominator (so they sum to 1); saved_call_rate_over_all_verdicts keeps the network-verdict denominator.", + "The no-network baseline needs the collector's trivial_only field; its coverage is reported rather than assumed.", + "This report embeds no raw record, no conversation text and no credential.", + ] + if modes.get("enforce"): + notes.append("Records with mode=enforce carry no extraction outcome for a skip (no call ran), so such a skip is censored unless the collector records a review_outcome; a miss rate is only meaningful over shadow records.") + report: dict[str, Any] = { + "script": "eval_memory_prescreen", + "source": source, + "input": { + "records": len(records), + "read_lines": read_lines, + "filtered_out": filtered_out, + "malformed_lines": malformed_lines, + "blank_lines": blank_lines, + "label_filters": [f"{key}={value}" for key, value in label_filters], + }, + "thresholds": {"miss_target": thresholds.miss_target, "confidence": thresholds.confidence, "min_reviewed_skips": thresholds.min_reviewed_skips}, + "pricing": prices.as_dict(), + "populations": populations, + "model": { + "population": NETWORK, + "scope": "network verdicts only: the classifier samples", + "modes": dict(modes), + "verdicts": verdicts, + "skip_rate": populations[NETWORK]["skip_rate"], + "scored_skips": evidence.scored_skips, + "saved_calls": evidence.saved_calls, + "saved_call_rate": populations[NETWORK]["saved_call_rate"], + "saved_call_rate_over_all_verdicts": populations[NETWORK]["saved_call_rate_over_all_verdicts"], + "misses": evidence.misses, + "miss_rate": populations[NETWORK]["miss_rate"], + "miss_rate_upper_bound": {"method": "clopper-pearson one-sided exact", "confidence": thresholds.confidence, "value": upper, "misses": evidence.misses, "scored_skips": evidence.scored_skips}, + "missed_digests": populations[NETWORK]["missed_digests"], + "latency_ms": populations[NETWORK]["latency_ms"], + "tokens": populations[NETWORK]["tokens"], + "cost": populations[NETWORK]["cost"], + }, + "baseline_heuristic": baseline, + "sensitive_stratum": { + "labels": sorted(SENSITIVE_LABELS), + "by_label": {label: {"scored_skips": sensitive_scored_skips[label], "misses": sensitive_misses[label]} for label in sorted(SENSITIVE_LABELS)}, + "note": ( + "Labels come from the collector's stratum/labels field; L3 keeps signal-bearing batches out of the " + "network population, so this is a human label. The gate checks every required stratum separately, " + "because a pooled sample can hide a stratum with no evidence." + ), + }, + "review": { + "reviewed_skips": evidence.reviewed_skips, + "reviewed_scored_skips": evidence.reviewed_scored_skips, + "reviewed_missed": evidence.reviewed_missed, + "review_outcomes": dict(Counter(record.review_outcome for record in network if record.review_outcome)), + "note": "reviewed comes from the collector; review_outcome is a second ground-truth witness for a skip. Gate 3 requires the review to confirm none was worth remembering, so a reviewed skip found worth remembering fails it.", + }, + "censoring": { + "records": len(records), + "censored": coverage.censored, + "reasons": dict(Counter(record.censored_reason for record in records if record.censored_reason)), + "malformed_lines": malformed_lines, + "note": "A censored sample has no outcome to score; it is counted and reported, never dropped.", + }, + "collector_contract": { + "from the DeerMem extraction_callback payload": "prescreen.{mode,verdict,probability,skip_threshold,model,cached,digest,signals,message_count,batch_chars,duration_ms,fallback_reason}, success, token_usage, mutations_accepted", + "must be added by the collector": { + "trivial_only": "bool - every turn in the batch was trivial (filter_trivial); the no-network baseline cannot be computed without it", + "reviewed": "bool - a human reviewed this skip verdict (design §5 gate 3)", + "review_outcome": "worth_remembering | none_worth_remembering (optional; a second ground-truth witness)", + "stratum": "identity | preference | correction (or a labels list); design §5 gate 2 subset", + "cost_usd": "float (optional) - the record's billed cost, when the collector knows it", + }, + }, + "notes": notes, + "gates": [gate.as_dict() for gate in gates], + "gate_summary": { + "passed": [gate.gate_id for gate in gates if gate.status == PASS], + "failed": [gate.gate_id for gate in gates if gate.status == FAIL], + "insufficient": [gate.gate_id for gate in gates if gate.status == INSUFFICIENT], + "enable_recommended": all(gate.status == PASS for gate in gates), + }, + } + return report + + +def _format_rate(value: float | None) -> str: + return "n/a" if value is None else f"{value * 100:.1f}%" + + +def _format_ms(value: float | None) -> str: + return "n/a" if value is None else f"{value:.1f}ms" + + +def _format_ratio(value: float | None) -> str: + return "n/a" if value is None else f"{value:.4f}" + + +def _format_cost(cost: Mapping[str, Any]) -> str: + if cost["total_usd"] is None: + return f"not configured ({cost['unpriced_records']} record(s) with no --price-in/--price-out and no record cost_usd)" + return f"${cost['total_usd']:.4f} ({cost['pricing']}; {cost['priced_records']} priced / {cost['unpriced_records']} unpriced)" + + +def print_report(report: Mapping[str, Any]) -> None: + """Print the populations table, the baseline comparison and every gate verdict.""" + input_summary = report["input"] + filters = ", ".join(input_summary["label_filters"]) or "none" + print(f"\nMemory pre-screen shadow evaluation: {report['source']}") + print(f"records: {input_summary['records']} scored; {input_summary['filtered_out']} filtered out; {input_summary['malformed_lines']} malformed line(s); {input_summary['blank_lines']} blank line(s); label filters: {filters}") + print("\nPopulations (mutually exclusive, design section 5; only network verdicts are classifier samples):") + print(f" {'population':<15}{'n':>5}{'skip':>6}{'extract':>8}{'cached':>7}{'censored':>9}{'p50':>10}{'p95':>10}") + for name in POPULATIONS: + summary = report["populations"][name] + latency = summary["latency_ms"] + print(f" {name:<15}{summary['records']:>5}{summary['verdicts']['skip']:>6}{summary['verdicts']['extract']:>8}{summary['cached']['true']:>7}{summary['censored']:>9}{_format_ms(latency['p50']):>10}{_format_ms(latency['p95']):>10}") + for name in DESIGN_POPULATIONS: + reasons = report["populations"][name].get("fallback_reasons") + if reasons: + rendered = ", ".join(f"{reason}={count} [{FALLBACK_LOCKS.get(reason, 'unknown')}]" for reason, count in sorted(reasons.items())) + print(f" {name} reasons: {rendered}") + + model = report["model"] + verdicts = model["verdicts"] + print(f"\nModel verdicts ({model['scope']}; modes {model['modes'] or 'none'}):") + print(f" skip_rate {_format_rate(model['skip_rate'])} ({verdicts['skip']} skip / {verdicts['extract']} extract / {verdicts['none']} no verdict)") + print(f" scored skips {model['scored_skips']}; saved calls {model['saved_calls']} = {_format_rate(model['saved_call_rate'])} of scored skips ({_format_rate(model['saved_call_rate_over_all_verdicts'])} of all network verdicts)") + print(f" missed memories {model['misses']} = miss_rate {_format_rate(model['miss_rate'])} (a skip with mutations_accepted > 0)") + bound = model["miss_rate_upper_bound"] + print(f" miss-rate upper bound: {_format_ratio(bound['value'])} one-sided at {bound['confidence']:.0%} ({bound['method']}, misses={bound['misses']}, n={bound['scored_skips']})") + latency = model["latency_ms"] + tokens = model["tokens"] + print(f" verdict latency p50 {_format_ms(latency['p50'])} p95 {_format_ms(latency['p95'])} (n={latency['samples']})") + print(f" tokens in={tokens['input']} out={tokens['output']} total={tokens['total']} (n={tokens['records']}); cost {_format_cost(model['cost'])}") + if model["missed_digests"]: + print(f" missed digests: {', '.join(model['missed_digests'])}") + + baseline = report["baseline_heuristic"] + print(f"\nNo-network heuristic baseline ({baseline['rule']}):") + coverage = f"{baseline['computable']}/{baseline['evaluated_verdicts']}" + print(f" computable on {coverage} network verdicts (coverage {_format_rate(baseline['coverage'])})") + print(f" collector field trivial_only missing on {baseline['trivial_only_missing']} record(s); signals missing on {baseline['signals_missing']}") + baseline_rates = f"saved {_format_rate(baseline['saved_call_rate'])}, miss {_format_rate(baseline['miss_rate'])}" + print(f" baseline: skips {baseline['skips']}, saved calls {baseline['saved_calls']}, misses {baseline['misses']} ({baseline_rates})") + model_on_set = baseline["model"] + model_rates = f"saved {_format_rate(model_on_set['saved_call_rate'])}, miss {_format_rate(model_on_set['miss_rate'])}" + print(f" model on the same {baseline['scored']} scored record(s): skips {model_on_set['skips']}, saved calls {model_on_set['saved_calls']}, misses {model_on_set['misses']} ({model_rates})") + print(f" incremental saved calls {baseline['incremental_saved_calls']} ({_format_rate(baseline['incremental_saved_call_rate'])} of the scored records)") + + sensitive = report["sensitive_stratum"] + review = report["review"] + censoring = report["censoring"] + per_label = "; ".join(f"{label}: {data['misses']} miss(es) over {data['scored_skips']} scored skip(s)" for label, data in sensitive["by_label"].items()) + print(f"\nSensitive strata ({', '.join(sensitive['labels'])}): {per_label}.") + print(f"Review sample: {review['reviewed_skips']} reviewed skip verdict(s) ({review['reviewed_scored_skips']} scored, {review['reviewed_missed']} found worth remembering); review outcomes: {review['review_outcomes'] or 'none'}.") + print(f"Censoring: {censoring['censored']} censored of {censoring['records']} record(s) {censoring['reasons'] or ''}; malformed lines {censoring['malformed_lines']}.") + + print("\nGates before enforce (design section 5):") + for gate in report["gates"]: + print(f" [{gate['status']:<12}] {gate['id']}: {gate['detail']}") + summary = report["gate_summary"] + if summary["enable_recommended"]: + print("\nEvaluation gates PASSED: every gate is PASS; enforce may be enabled for a controlled window.") + else: + print(f"\nDO NOT ENABLE: {len(summary['failed'])} gate(s) failed, {len(summary['insufficient'])} gate(s) lack evidence.") + if summary["failed"]: + print(f" failed: {', '.join(summary['failed'])}") + if summary["insufficient"]: + print(f" insufficient: {', '.join(summary['insufficient'])}") + print("\nNotes:") + for note in report["notes"]: + print(f" - {note}") + + +def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--records", type=Path, required=True, help="JSONL of shadow records: one DeerMem extraction_callback payload per line") + parser.add_argument("--label", action="append", type=parse_label_filter, default=[], metavar="KEY=VALUE", help="keep only records whose (dotted) field matches; repeatable, all must match") + parser.add_argument("--json", type=Path, default=None, help="write the full report as JSON to this path") + parser.add_argument("--miss-target", type=float, default=DEFAULT_MISS_TARGET, help="miss-rate point estimate the gate accepts (default: %(default)s)") + parser.add_argument("--confidence", type=float, default=DEFAULT_CONFIDENCE, help="confidence for the Clopper-Pearson upper bound (default: %(default)s)") + parser.add_argument("--min-reviewed-skips", type=int, default=DEFAULT_MIN_REVIEWED_SKIPS, help="reviewed skip verdicts required before enforce (default: %(default)s)") + parser.add_argument("--price-in", type=float, default=None, help="Jev input price in USD per million tokens (optional; enables the cost estimate)") + parser.add_argument("--price-out", type=float, default=None, help="Jev output price in USD per million tokens (optional; enables the cost estimate)") + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + """Run the evaluation; the exit code is the gate verdict (0 only when all PASS).""" + args = _parse_args(argv) + loaded = load_records(args.records, args.label) + thresholds = Thresholds(miss_target=args.miss_target, confidence=args.confidence, min_reviewed_skips=args.min_reviewed_skips) + prices = Prices(input_per_mtok=args.price_in, output_per_mtok=args.price_out) + report = build_report( + loaded.records, + source=str(args.records), + malformed_lines=loaded.malformed_lines, + filtered_out=loaded.filtered_out, + read_lines=loaded.read, + blank_lines=loaded.blank_lines, + thresholds=thresholds, + prices=prices, + label_filters=args.label, + ) + print_report(report) + if args.json is not None: + args.json.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print(f"\nJSON report: {args.json}") + return 0 if report["gate_summary"]["enable_recommended"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/tests/test_agent_guidance_check.py b/backend/tests/test_agent_guidance_check.py index d43b6f9105..f1ab584659 100644 --- a/backend/tests/test_agent_guidance_check.py +++ b/backend/tests/test_agent_guidance_check.py @@ -38,6 +38,7 @@ EXPECTED_GUIDANCE_PATHS = { "backend/packages/harness/deerflow/tools/AGENTS.md", "backend/packages/harness/deerflow/tracing/AGENTS.md", "backend/packages/harness/deerflow/tui/AGENTS.md", + "backend/packages/harness/deerflow/typesafe/AGENTS.md", "backend/packages/harness/deerflow/utils/AGENTS.md", "frontend/src/AGENTS.md", "scripts/AGENTS.md", diff --git a/backend/tests/test_app_config_reload.py b/backend/tests/test_app_config_reload.py index 8f04e5ce42..ba278fbc2d 100644 --- a/backend/tests/test_app_config_reload.py +++ b/backend/tests/test_app_config_reload.py @@ -908,3 +908,53 @@ def test_get_memory_config_falls_back_on_broken_config(tmp_path, monkeypatch): assert get_memory_config().enabled is False finally: _reset_config_singletons() + + +# The two judging modes as an operator writes them. `mode: off` is spelled the way +# config.example.yaml ships it and the way a rollback types it. +_MODES_CONFIG = """\ +sandbox: + use: deerflow.sandbox.local:LocalSandboxProvider +models: + - name: first-model + use: langchain_openai:ChatOpenAI + model: gpt-test +memory: + prescreen: + mode: {prescreen} + use: deerflow.agents.memory.prescreen.typesafe:TypeSafeMemoryPrescreen + signal_classification: + mode: {classification} + use: deerflow.agents.memory.signals.typesafe:TypeSafeSignalClassifier +""" + + +def test_an_unquoted_off_mode_loads_and_rolls_back(tmp_path, monkeypatch): + """`mode: off` is a YAML boolean, and both judging modes must still read it as off. + + YAML 1.1 parses an unquoted ``off`` as ``False``. Rejecting it would fail the + config *reload* that returns a deployment to off -- the failed load leaves the + previous judge (and its judging requests) running -- so the documented spelling + has to resolve to the ``off`` mode, not raise. + """ + config_path = tmp_path / "config.yaml" + extensions_path = tmp_path / "extensions_config.json" + _write_extensions_config(extensions_path) + monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(extensions_path)) + + def write_modes(prescreen: str, classification: str) -> None: + config_path.write_text(_MODES_CONFIG.format(prescreen=prescreen, classification=classification), encoding="utf-8") + + try: + write_modes("enforce", "hints") + enforcing = AppConfig.from_file(str(config_path)) + assert enforcing.memory.prescreen.mode == "enforce" + assert enforcing.memory.signal_classification.mode == "hints" + + write_modes("off", "off") + rolled_back = AppConfig.from_file(str(config_path)) + + assert rolled_back.memory.prescreen.mode == "off" + assert rolled_back.memory.signal_classification.mode == "off" + finally: + _reset_config_singletons() diff --git a/backend/tests/test_memory_prescreen.py b/backend/tests/test_memory_prescreen.py new file mode 100644 index 0000000000..8961e6c8d2 --- /dev/null +++ b/backend/tests/test_memory_prescreen.py @@ -0,0 +1,1487 @@ +"""Memory pre-screening and signal classification: contracts, adapters, insertion, locks. + +Covers the pre-screening plan's §4 checklist (L1-L11, the watermark branches, the +boundary cases, ``mutations_accepted``) and the signal-classification design's +combination rules (§2.2: eligibility, assembly, per-question failure, the veto, +``combine`` policies), all against the real updater rather than mocks of it. +""" + +from __future__ import annotations + +import copy +import json +import pathlib + +import httpx +import pytest +from langchain_core.messages import AIMessage, HumanMessage + +from deerflow.agents.memory.backends.deermem.deermem.config import DeerMemConfig +from deerflow.agents.memory.backends.deermem.deermem.core.prompt import format_conversation_for_update +from deerflow.agents.memory.backends.deermem.deermem.core.storage import MemoryStorage +from deerflow.agents.memory.backends.deermem.deermem.core.updater import MemoryUpdater +from deerflow.agents.memory.prescreen.contract import ( + MODE_ENFORCE, + MODE_OFF, + MODE_SHADOW, + MemoryPrescreenRequest, + resolve_memory_prescreen, +) +from deerflow.agents.memory.prescreen.typesafe import TypeSafeMemoryPrescreen +from deerflow.agents.memory.signals.contract import ( + COMBINE_ALWAYS, + COMBINE_NEVER, + LABEL_CORRECTION, + LABEL_REINFORCEMENT, + MODE_HINTS, + MemorySignalRequest, + resolve_memory_signal_classifier, +) +from deerflow.agents.memory.signals.coordinator import MemoryBatchContext, MemorySignalCoordinator +from deerflow.agents.memory.signals.typesafe import QUESTION_AFFIRMATION, QUESTION_NEGATION, TypeSafeSignalClassifier +from deerflow.typesafe.errors import TypeSafeError + +_API_KEY = "memory-judge-test-key" +_PRESCREEN_Q = "memory_worth_keeping" +_JUDGE_HOOK = "judge" + +# Signal-free on purpose: any deterministic signal removes the pre-screen's +# eligibility (L3), so a fixture that trips one would silently test the fallback. +_KEEPING_TEXT = "We reviewed the release candidate on Tuesday afternoon and the notes are in /tmp/notes.md" +_CHATTER_TEXT = "ok, thanks" + +_PRESCREEN = "deerflow.agents.memory.prescreen.typesafe:TypeSafeMemoryPrescreen" +_CLASSIFIER = "deerflow.agents.memory.signals.typesafe:TypeSafeSignalClassifier" + + +class _Server: + """Fake System One endpoint that records every request it receives.""" + + def __init__(self, responder=None) -> None: + self.requests: list[httpx.Request] = [] + self._responder = responder or (lambda request: httpx.Response(200, json=_answer(0.9))) + + def handle(self, request: httpx.Request) -> httpx.Response: + self.requests.append(request) + return self._responder(request) + + def transport(self) -> httpx.MockTransport: + return httpx.MockTransport(self.handle) + + @property + def count(self) -> int: + return len(self.requests) + + def bodies(self) -> list[dict]: + return [json.loads(request.content) for request in self.requests] + + +def _answer(*probabilities: float, model: str = "jev-1.13.0", questions: tuple[str, ...] = (_PRESCREEN_Q,)) -> dict: + return {"model": model, "answers": {question: {"type": "noul", "noul": probability} for question, probability in zip(questions, probabilities)}} + + +def _prescreen(server: _Server, *, mode: str = MODE_SHADOW, **kwargs) -> TypeSafeMemoryPrescreen: + kwargs.setdefault("skip_threshold", 0.2) + kwargs.setdefault("max_state_chars", 6000) + kwargs.setdefault("transport_factory", server.transport) + return TypeSafeMemoryPrescreen(mode=mode, api_key=_API_KEY, **kwargs) + + +def _classifier(server: _Server, *, mode: str = MODE_HINTS, **kwargs) -> TypeSafeSignalClassifier: + kwargs.setdefault("hint_threshold", 0.5) + kwargs.setdefault("max_state_chars", 6000) + kwargs.setdefault("transport_factory", server.transport) + return TypeSafeSignalClassifier(mode=mode, api_key=_API_KEY, **kwargs) + + +def _combined_judge(server: _Server, *, prescreen_mode: str = MODE_ENFORCE, classifier_mode: str = MODE_HINTS, combine: str = "auto", **overrides) -> MemorySignalCoordinator: + """One judge over both sides sharing a single transport factory. + + The transport factory is part of the sharing identity (§2.2.5), so both sides + must be handed the *same* factory object for a shared request. + """ + factory = server.transport # one bound method object: sharing identity must match (§2.2.5) + return _judge( + _prescreen(server, mode=prescreen_mode, transport_factory=factory, **overrides), + prescreen_mode=prescreen_mode, + classifier=_classifier(server, mode=classifier_mode, transport_factory=factory), + classifier_mode=classifier_mode, + combine=combine, + ) + + +def _judge(prescreen=None, prescreen_mode: str = MODE_SHADOW, classifier=None, classifier_mode: str = MODE_HINTS, combine: str = "auto") -> MemorySignalCoordinator: + return MemorySignalCoordinator(prescreen=prescreen, prescreen_mode=prescreen_mode, classifier=classifier, classifier_mode=classifier_mode, combine=combine) + + +def _context(text: str = _KEEPING_TEXT, **overrides) -> MemoryBatchContext: + context = {"batch_text": text, "digest": "digest-1"} + context.update(overrides) + return MemoryBatchContext(**context) + + +# --- the updater harness --------------------------------------------------- + + +class _FakeLLM: + """Returns a fixed extraction payload and records the prompts it was given.""" + + def __init__(self, payload: str | None = None) -> None: + self.payload = payload or '{"user":{},"history":{},"newFacts":[],"factsToRemove":[]}' + self.prompts: list[object] = [] + + def invoke(self, prompt, config=None): + self.prompts.append(prompt) + return type("R", (), {"content": self.payload})() + + @property + def calls(self) -> int: + return len(self.prompts) + + +def _memory() -> dict[str, object]: + return { + "version": "1.0", + "revision": 0, + "lastUpdated": "", + "user": {section: {"summary": "", "updatedAt": ""} for section in ("workContext", "personalContext", "topOfMind", "cognitiveStyle")}, + "history": {section: {"summary": "", "updatedAt": ""} for section in ("recentMonths", "earlierContext", "longTermBackground")}, + "facts": [], + } + + +def _fact(index: int, *, confidence: float) -> dict[str, object]: + return {"id": f"fact_{index}", "content": f"Existing durable fact {index}", "category": "context", "confidence": confidence, "createdAt": "2026-01-01T00:00:00Z", "source": "t"} + + +def _memory_with_facts(facts: list[dict[str, object]]) -> dict[str, object]: + memory = _memory() + memory["facts"] = copy.deepcopy(facts) + return memory + + +class _Storage(MemoryStorage): + def __init__(self) -> None: + self.memory = _memory() + self.saves = 0 + + def load(self, agent_name=None, *, user_id=None): + return copy.deepcopy(self.memory) + + def reload(self, agent_name=None, *, user_id=None): + return self.load(agent_name, user_id=user_id) + + def save(self, memory_data, agent_name=None, *, user_id=None, expected_revision=None): + self.memory = copy.deepcopy(memory_data) + self.saves += 1 + return True + + +def _updater(*, judge=None, llm=None, recorded: list | None = None, **config_overrides) -> MemoryUpdater: + config = DeerMemConfig() + # DeerMem defaults staleness_review_enabled=True, and L8 then removes the + # pre-screen's eligibility entirely (a skip would also skip that batch's + # maintenance review). Tests that need judging mirror a deployment which + # allows skipping; the L8 cases flip it back on explicitly. + config_overrides.setdefault("staleness_review_enabled", False) + for key, value in config_overrides.items(): + setattr(config, key, value) + if judge is not None: + setattr(config, _JUDGE_HOOK, judge) + if recorded is not None: + config.extraction_callback = recorded.append + return MemoryUpdater(config, _Storage(), llm or _FakeLLM()) + + +def _conversation(text: str = _KEEPING_TEXT): + return [HumanMessage(content=text), AIMessage(content="Understood.")] + + +def _watermark(updater: MemoryUpdater) -> tuple | None: + return updater._watermarks.get(("thread-1", "user-1", None)) + + +# --- L1: no judge, no change ---------------------------------------------- + + +class TestNoJudge: + def test_an_unconfigured_deployment_extracts_and_sends_nothing(self): + server = _Server() + llm = _FakeLLM('{"user":{},"history":{},"newFacts":[{"content":"User prefers dark mode","category":"preference","confidence":0.9,"scope":"user","durability":"durable","authority":"descriptive"}],"factsToRemove":[]}') + updater = _updater(judge=None, llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 1 + assert server.count == 0 + assert [fact["content"] for fact in updater._storage.memory["facts"]] == ["User prefers dark mode"] + + def test_no_judge_means_no_judge_payload_in_the_metrics(self): + recorded: list = [] + updater = _updater(recorded=recorded) + + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") + + assert recorded + assert "prescreen" not in recorded[0] + assert "signal_classification" not in recorded[0] + + +# --- phase A eligibility (L3 / L4 / L7 / L8 / L5) ------------------------- + + +class TestEligibility: + def test_deterministic_signals_force_extraction_without_a_request(self): + # The updater re-detects signals on the post-watermark feed, so L3 reads + # the same set the extraction hints use -- not the admission-time set the + # queue already consumed. + text = "No, that's wrong - use tabs not spaces" + server = _Server() + llm = _FakeLLM() + recorded: list = [] + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm, recorded=recorded) + + assert updater.update_memory(_conversation(text), thread_id="thread-1", user_id="user-1") is True + + assert server.count == 0, "L3: deterministic signals answer for the pre-screen" + assert llm.calls == 1 + # §5 counts local fallbacks, so the reason has to reach the extraction record + # the shadow evaluation reads. + assert recorded[0]["prescreen"]["fallback_reason"] == "deterministic_signals" + + def test_a_signal_outside_the_hint_window_still_keeps_the_batch_out_of_judging(self): + """L3 reads the whole batch, not the 6-message window the extraction hint uses. + + A skip advances the watermark past every turn fed to this batch, so an + explicit correction in its *first* turn must keep the batch out of judging: + scanning only the trailing window let an eight-message batch be skipped, and + the correction was then never extracted. + """ + correction = "No, that's wrong - use tabs not spaces" + batch = [HumanMessage(content=correction), AIMessage(content="Understood.")] + for index in range(3): + batch += [HumanMessage(content=f"what about the file {index}"), AIMessage(content="It is unchanged.")] + assert len(batch) == 8, "the correction has to sit outside detect_signals' 6-message window" + server = _Server(lambda request: httpx.Response(200, json=_answer(0.05))) + llm = _FakeLLM() + recorded: list = [] + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm, recorded=recorded) + + assert updater.update_memory(batch, thread_id="thread-1", user_id="user-1") is True + + assert server.count == 0, "L3: a signal anywhere in the batch answers for the pre-screen" + assert llm.calls == 1, "the batch is not consumed by a skip" + assert recorded[0]["prescreen"]["fallback_reason"] == "deterministic_signals" + + def test_the_emergency_flush_path_never_judges(self): + server = _Server() + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1", bypass_watermark=True) is True + + assert server.count == 0, "L4: the batch is about to be removed by summarization" + assert llm.calls == 1 + assert _watermark(updater) is None, "the emergency path never advances the conversation watermark" + + def test_the_shutdown_drain_never_judges(self): + server = _Server() + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1", judge=False) is True + + assert server.count == 0, "L7: the drain budget belongs to persistence" + assert llm.calls == 1 + + @pytest.mark.parametrize("flag", ["staleness_review_enabled", "consolidation_enabled"]) + def test_maintenance_review_disables_skipping(self, flag): + server = _Server() + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm, **{flag: True}) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert server.count == 0, "L8: a skip would also skip that batch's maintenance review" + assert llm.calls == 1 + + def test_a_cjk_batch_is_limited_by_characters_not_wire_bytes(self): + """Shared-client design §2.4: the limit stays a character count over the judged text.""" + text = "我们决定用标签" * 5 # 35 characters, 105 UTF-8 bytes + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9))) + judge = _judge(_prescreen(server, mode=MODE_ENFORCE, max_state_chars=len(text)), prescreen_mode=MODE_ENFORCE) + + verdict = judge.judge(_context(text)) + + assert verdict.payload["prescreen"]["fallback_reason"] is None, "the batch fits its character limit" + assert server.count == 1, "counting UTF-8 bytes would have refused this request three times too early" + + def test_an_over_limit_batch_extracts_without_a_request_or_truncation(self): + server = _Server() + llm = _FakeLLM() + text = "x" * 4000 + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE, max_state_chars=10), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(text), thread_id="thread-1", user_id="user-1") is True + + assert server.count == 0, "L5: no request, extract as usual" + assert llm.calls == 1 + # No second truncation: the extractor sees exactly the formatter's output. + formatted = format_conversation_for_update(_conversation(text)) + prompt_text = "\n".join(str(getattr(message, "content", message)) for message in llm.prompts[0]) + assert formatted in prompt_text + + +# --- L2: every failure direction extracts --------------------------------- + + +class TestFailureDirection: + @pytest.mark.parametrize( + "responder", + [ + lambda request: httpx.Response(200, content=b"not json at all"), + lambda request: httpx.Response(200, content=b'{"model": "jev-1.13.0"}'), + lambda request: httpx.Response(200, json=_answer("not a number")), + lambda request: httpx.Response(500, json={}), + lambda request: (_ for _ in ()).throw(httpx.ConnectError("down", request=request)), + ], + ) + def test_a_failing_verdict_extracts_as_usual(self, responder): + server = _Server(responder) + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE, retry_backoff=0.0), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 1, "L2: pre-screening never loses a memory" + assert _watermark(updater) is not None, "extraction success still advances the watermark" + + def test_a_provider_exception_extracts_as_usual(self): + class _Exploding: + name = "exploding" + + def decide(self, request): + raise RuntimeError("provider bug") + + def release_policy_parameters(self): + return {} + + llm = _FakeLLM() + updater = _updater(judge=_judge(_Exploding(), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 1 + + @pytest.mark.parametrize( + "responder", + [ + lambda request: httpx.Response(200, content=b"not json at all"), + lambda request: httpx.Response(400, json={}), + lambda request: (_ for _ in ()).throw(httpx.ConnectError("down", request=request)), + ], + ) + def test_a_request_level_failure_is_recorded_as_request_failed(self, responder): + """The record separates a failed request from a provider with no opinion. + + Both directions extract as usual, but only one of them means the endpoint is + unreachable — which is what an operator reads the fallback population for, so + the two must not both arrive as ``no_verdict``. + """ + recorded: list = [] + updater = _updater(judge=_judge(_prescreen(_Server(responder), mode=MODE_ENFORCE, retry_backoff=0.0), prescreen_mode=MODE_ENFORCE), llm=_FakeLLM(), recorded=recorded) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert recorded and recorded[0]["prescreen"]["fallback_reason"] == "request_failed" + assert recorded[0]["prescreen"]["verdict"] is None + + def test_an_answered_but_unusable_response_is_not_a_request_failure(self): + """A question-level failure is the request *working*: "no verdict", not "failed".""" + recorded: list = [] + + def responder(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json=_answer("not a number")) + + updater = _updater(judge=_judge(_prescreen(_Server(responder), mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=_FakeLLM(), recorded=recorded) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert recorded[0]["prescreen"]["fallback_reason"] == "no_verdict" + + +# --- modes: shadow / enforce ---------------------------------------------- + + +class TestShadowAndEnforce: + def test_shadow_records_the_verdict_and_extracts_anyway(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + recorded: list = [] + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_SHADOW), prescreen_mode=MODE_SHADOW), llm=llm, recorded=recorded) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 1, "shadow changes nothing" + assert _watermark(updater) is not None, "the watermark advances on extraction success, as today" + record = recorded[0]["prescreen"] + assert (record["mode"], record["verdict"], record["probability"]) == (MODE_SHADOW, "skip", 0.01) + assert record["digest"] and record["skip_threshold"] == 0.2 + + def test_enforce_skip_drops_the_call_and_advances_the_watermark(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + recorded: list = [] + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm, recorded=recorded) + + assert updater.update_memory(_conversation(_CHATTER_TEXT), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 0, "enforce + skip saves the extraction call" + assert _watermark(updater) is not None, "the batch is consumed, so it is not re-judged every turn" + assert recorded, "a skip still emits its record" + assert recorded[0]["success"] is True and recorded[0]["token_usage"] is None, "no LLM call, so no token usage, but the record is still emitted" + assert "mutations_accepted" not in recorded[0], "no apply ran" + assert recorded[0]["prescreen"]["verdict"] == "skip" + + def test_enforce_extract_matches_today(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.99))) + llm = _FakeLLM() + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=llm) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert llm.calls == 1 + assert _watermark(updater) is not None + + def test_threshold_boundary_extracts(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.2))) + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE, skip_threshold=0.2), prescreen_mode=MODE_ENFORCE), llm=_FakeLLM()) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert updater._llm.calls == 1, "p == skip_threshold extracts (when unsure, extract)" + + def test_an_extraction_failure_leaves_the_watermark_alone(self): + """The fourth watermark branch: a failed extraction advances nothing, so the + batch is re-fed next turn -- and the judge's verdict did not consume it + either (it said extract, and extraction did not succeed).""" + server = _Server(lambda request: httpx.Response(200, json=_answer(0.99))) + + class _ExplodingLLM: + def invoke(self, prompt, config=None): + raise RuntimeError("extraction provider down") + + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=_ExplodingLLM()) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is False + assert _watermark(updater) is None + + def test_the_cache_makes_the_second_batch_free(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + recorded: list = [] + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE), llm=_FakeLLM(), recorded=recorded) + + updater.update_memory(_conversation(_CHATTER_TEXT), thread_id="thread-1", user_id="user-1") + updater.update_memory(_conversation(_CHATTER_TEXT), thread_id="thread-2", user_id="user-2") + + assert server.count == 1, "same digest, same verdict" + assert recorded[0]["prescreen"]["cached"] is False + assert recorded[1]["prescreen"]["cached"] is True, "a cache hit is recorded as one" + + def test_an_expired_cache_entry_is_requested_again(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + updater = _updater(judge=_judge(_prescreen(server, mode=MODE_ENFORCE, cache_ttl_seconds=0.01), prescreen_mode=MODE_ENFORCE), llm=_FakeLLM()) + + updater.update_memory(_conversation(_CHATTER_TEXT), thread_id="thread-1", user_id="user-1") + import time + + time.sleep(0.05) + updater.update_memory(_conversation(_CHATTER_TEXT), thread_id="thread-2", user_id="user-2") + + assert server.count == 2 + + +# --- signal classification: hints, per-question failure, the veto --------- + + +class TestSignalClassification: + def test_hints_are_merged_into_the_extraction_prompt(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9, 0.1, questions=(QUESTION_AFFIRMATION, QUESTION_NEGATION)))) + recorded: list = [] + llm = _FakeLLM() + updater = _updater(judge=_judge(classifier=_classifier(server), classifier_mode=MODE_HINTS), llm=llm, recorded=recorded) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + prompt_text = "\n".join(str(getattr(message, "content", message)) for message in llm.prompts[0]) + assert "Positive reinforcement signals were detected" in prompt_text + assert "Explicit correction signals were detected" not in prompt_text, "negation stayed below the threshold" + assert recorded[0]["signal_classification"]["labels"] == [LABEL_REINFORCEMENT] + + def test_a_failed_direction_keeps_the_other_one(self): + server = _Server(lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": "not a number"}}})) + recorded: list = [] + updater = _updater(judge=_judge(classifier=_classifier(server), classifier_mode=MODE_HINTS), recorded=recorded) + + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") + + record = recorded[0]["signal_classification"] + assert record["labels"] == [LABEL_REINFORCEMENT], "failure is counted per question, never per side" + assert LABEL_CORRECTION not in record["probabilities"] + + def test_a_model_hint_vetoes_a_skip_only_under_enforce(self): + answering = _Server( + lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.01}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + ) + recorded: list = [] + llm = _FakeLLM() + updater = _updater(judge=_combined_judge(answering), llm=llm, recorded=recorded) + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is True + + assert answering.count == 1, "one shared request serves both sides" + assert set(answering.bodies()[0]["questions"]) == {_PRESCREEN_Q, QUESTION_AFFIRMATION, QUESTION_NEGATION} + assert llm.calls == 1, "the veto recovered the extraction" + assert _watermark(updater) is not None + assert recorded[0]["skip_vetoed_by_model_signal"] is True + + def test_a_hint_does_not_veto_without_enforce(self): + answering = _Server(lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.01}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}}})) + recorded: list = [] + llm = _FakeLLM() + updater = _updater(judge=_combined_judge(answering, prescreen_mode=MODE_SHADOW), llm=llm, recorded=recorded) + + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") + + assert llm.calls == 1, "shadow has no skip, so there is nothing to veto (S11)" + assert "skip_vetoed_by_model_signal" not in recorded[0] + + def test_classifier_only_asks_its_own_questions(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9, 0.1, questions=(QUESTION_AFFIRMATION, QUESTION_NEGATION)))) + updater = _updater(judge=_judge(classifier=_classifier(server), classifier_mode=MODE_HINTS)) + + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") + + assert set(server.bodies()[0]["questions"]) == {QUESTION_AFFIRMATION, QUESTION_NEGATION} + + +# --- the coordinator's contract (unit level) ------------------------------ + + +class TestCoordinatorRules: + def test_both_off_is_a_no_op(self): + verdict = _judge().judge(_context()) + + assert verdict.payload == {} + assert verdict.skip is False and verdict.hints == frozenset() + + def test_combine_never_sends_two_requests(self): + server = _Server( + lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + ) + judge = _combined_judge(server, combine=COMBINE_NEVER) + + judge.judge(_context()) + + assert server.count == 2, "each side keeps its own request and its own cache" + + def test_a_limit_mismatch_splits_the_requests(self): + server = _Server( + lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + ) + factory = server.transport + judge = _judge(_prescreen(server, mode=MODE_ENFORCE, transport_factory=factory), prescreen_mode=MODE_ENFORCE, classifier=_classifier(server, max_state_chars=7000, transport_factory=factory), classifier_mode=MODE_HINTS) + + judge.judge(_context()) + + assert server.count == 2, "auto separates rather than silently widening the stricter side's limit" + + def test_combine_always_refuses_a_mismatched_pair(self): + server = _Server() + factory = server.transport + with pytest.raises(ValueError, match="combine='always'"): + _judge( + _prescreen(server, mode=MODE_ENFORCE, transport_factory=factory), + prescreen_mode=MODE_ENFORCE, + classifier=_classifier(server, max_state_chars=7000, transport_factory=factory), + classifier_mode=MODE_HINTS, + combine=COMBINE_ALWAYS, + ) + + def test_a_full_cache_hit_sends_nothing(self): + server = _Server( + lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + ) + judge = _combined_judge(server) + + first = judge.judge(_context()) + second = judge.judge(_context()) + + assert server.count == 1, "the second round is a full hit" + assert first.payload["prescreen"]["cached"] is False + assert second.payload["prescreen"]["cached"] is True + assert second.payload["signal_classification"]["labels"] == [LABEL_REINFORCEMENT] + + def test_a_partial_response_is_topped_up_into_the_same_bucket(self): + """One direction missing in a response is asked again next time; the other is not.""" + bodies: list[dict] = [] + + def responder(request: httpx.Request) -> httpx.Response: + bodies.append(json.loads(request.content)) + questions = bodies[-1]["questions"] + answers = {question: {"type": "noul", "noul": 0.9} for question in questions if question != QUESTION_NEGATION} + return httpx.Response(200, json={"model": "jev-1.13.0", "answers": answers}) + + server = _Server(responder) + judge = _combined_judge(server) + + judge.judge(_context()) + second = judge.judge(_context()) + + assert set(bodies[0]["questions"]) == {_PRESCREEN_Q, QUESTION_AFFIRMATION, QUESTION_NEGATION}, "the combined request carries every side's questions" + assert set(bodies[1]["questions"]) == {QUESTION_NEGATION}, "only the unanswered question is retried" + assert second.payload["signal_classification"]["labels"] == [LABEL_REINFORCEMENT], "the cached direction is reused" + + def test_the_shared_bucket_answers_before_eligibility_narrows_the_question_set(self): + """§2.2.3 step 1: the key holds the full logical set, so an ineligible side costs no request.""" + server = _Server( + lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + ) + judge = _combined_judge(server) + + judge.judge(_context()) + signalled = judge.judge(_context(signals=frozenset({"correction"}))) + + assert server.count == 1, "the answers were already bucketed for this digest" + assert signalled.payload["prescreen"]["fallback_reason"] == "deterministic_signals" + assert signalled.payload["signal_classification"]["labels"] == [LABEL_REINFORCEMENT], "the still-eligible side consumes the held answers" + + def test_an_enabled_side_records_why_it_could_not_judge(self): + """§5 counts local fallbacks, so a round with no eligible side still emits its record.""" + server = _Server() + judge = _judge(_prescreen(server, mode=MODE_ENFORCE), prescreen_mode=MODE_ENFORCE) + + verdict = judge.judge(_context(signals=frozenset({"correction"}))) + + assert server.count == 0 + assert verdict.decided is True + assert verdict.payload["prescreen"]["verdict"] is None + assert verdict.payload["prescreen"]["fallback_reason"] == "deterministic_signals" + assert verdict.skip is False + + def test_a_failed_shared_request_marks_every_eligible_side(self): + """One failed request leaves both eligible sides without a verdict, and says so. + + The endpoint answering ``400`` (not a retryable status) fails the round + immediately: no retry, no second request, and no verdict for either side. + """ + server = _Server(lambda request: httpx.Response(400, json={})) + judge = _combined_judge(server) + + verdict = judge.judge(_context()) + + assert server.count == 1 + assert verdict.payload["prescreen"]["fallback_reason"] == "request_failed" + assert verdict.payload["signal_classification"]["fallback_reason"] == "request_failed" + assert verdict.skip is False and verdict.hints == frozenset() + + def test_a_failed_shared_request_keeps_an_ineligible_sides_own_reason(self): + """Only the side that actually asked can report a request failure.""" + server = _Server(lambda request: httpx.Response(400, json={})) + judge = _combined_judge(server) + + verdict = judge.judge(_context(signals=frozenset({"correction"}))) + + assert server.count == 1, "the still-eligible classifier carried the request" + assert verdict.payload["prescreen"]["fallback_reason"] == "deterministic_signals", "L3 removed this side; it did not ask" + assert verdict.payload["signal_classification"]["fallback_reason"] == "request_failed" + + def test_a_failed_shared_request_is_logged_where_it_happens(self, caplog): + """The durable trace is the record; an unreachable endpoint also needs a line now.""" + server = _Server(lambda request: httpx.Response(400, json={})) + judge = _combined_judge(server) + + with caplog.at_level("WARNING"): + judge.judge(_context()) + + assert any("Memory judge request failed" in record.message for record in caplog.records) + + def test_a_reused_verdict_is_not_reported_as_a_network_sample_when_the_other_side_fetches(self): + """Per-answer provenance: a verdict reused from the bucket stays a cache hit. + + Rounds differ in which questions the network answered. When the pre-screen + answer is already held and only the classifier question is fetched, stamping + the reused verdict ``cached=False`` (with the new response's model) makes the + shadow evaluation count it as another network sample, inflating the + confidence-bound denominator. + """ + bodies: list[dict] = [] + + def responder(request: httpx.Request) -> httpx.Response: + body = json.loads(request.content) + bodies.append(body) + if len(bodies) == 1: + return httpx.Response(200, json={"model": "jev-1.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}}}) + return httpx.Response(200, json={"model": "jev-2.0", "answers": {question: {"type": "noul", "noul": 0.9} for question in body["questions"]}}) + + server = _Server(responder) + judge = _combined_judge(server) + + first = judge.judge(_context()) + second = judge.judge(_context()) + + assert first.payload["prescreen"]["cached"] is False, "the first round fetched the pre-screen answer" + assert set(bodies[1]["questions"]) == {QUESTION_AFFIRMATION, QUESTION_NEGATION}, "only the missing classifier questions are asked" + assert second.payload["prescreen"]["cached"] is True, "the pre-screen answer came from the bucket" + assert second.payload["prescreen"]["model"] == "jev-1.0", "the reused verdict keeps the model that produced it" + assert second.payload["signal_classification"]["cached"] is False, "the classifier answer was fetched this round" + + def test_a_full_cache_hit_reports_each_answers_original_model(self): + """Model provenance is per answer, not per bucket. + + Filling a bucket with a newer model must not re-attribute the answers it + already held, or a later full cache hit audits the older model's verdict + under the newer model's name. + """ + bodies: list[dict] = [] + + def responder(request: httpx.Request) -> httpx.Response: + body = json.loads(request.content) + bodies.append(body) + if len(bodies) == 1: + return httpx.Response(200, json={"model": "jev-A", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.9}}}) + return httpx.Response(200, json={"model": "jev-B", "answers": {QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}}) + + server = _Server(responder) + judge = _combined_judge(server) + + judge.judge(_context()) # buckets the pre-screen answer, served by jev-A + judge.judge(_context()) # fills the classifier answers, served by jev-B + third = judge.judge(_context()) # full cache hit, no request + + assert server.count == 2, "the third round is a full cache hit" + assert third.payload["prescreen"]["cached"] is True and third.payload["prescreen"]["model"] == "jev-A" + assert third.payload["signal_classification"]["cached"] is True and third.payload["signal_classification"]["model"] == "jev-B" + + def test_per_side_limits_are_enforced_separately(self): + judged_chars = len(_KEEPING_TEXT) # the limit is a character count, not the wire size (design §2.4) + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9, 0.1, questions=(QUESTION_AFFIRMATION, QUESTION_NEGATION)))) + factory = server.transport + judge = _judge( + _prescreen(server, mode=MODE_ENFORCE, max_state_chars=judged_chars - 1, transport_factory=factory), + prescreen_mode=MODE_ENFORCE, + classifier=_classifier(server, max_state_chars=judged_chars + 100, transport_factory=factory), + classifier_mode=MODE_HINTS, + ) + + verdict = judge.judge(_context()) + + assert verdict.payload["prescreen"]["fallback_reason"] == "over_limit" + assert verdict.payload["signal_classification"]["fallback_reason"] is None + assert set(server.bodies()[0]["questions"]) == {QUESTION_AFFIRMATION, QUESTION_NEGATION}, "the side under its limit still asks" + + def test_only_the_side_over_its_limit_falls_back(self): + """Scenario row 11: the classifier is over *its* limit while the pre-screen is not.""" + judged_chars = len(_KEEPING_TEXT) # the limit is a character count, not the wire size (design §2.4) + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + factory = server.transport + judge = _judge( + _prescreen(server, mode=MODE_ENFORCE, max_state_chars=judged_chars + 100, transport_factory=factory), + prescreen_mode=MODE_ENFORCE, + classifier=_classifier(server, max_state_chars=judged_chars - 1, transport_factory=factory), + classifier_mode=MODE_HINTS, + ) + + verdict = judge.judge(_context()) + + assert verdict.payload["signal_classification"]["fallback_reason"] == "over_limit" + assert verdict.payload["prescreen"]["fallback_reason"] is None + assert set(server.bodies()[0]["questions"]) == {_PRESCREEN_Q}, "only the eligible side asks" + + def test_the_veto_requires_both_conditions(self): + server = _Server(lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {_PRESCREEN_Q: {"type": "noul", "noul": 0.01}, QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}}})) + hints_only = _judge(classifier=_classifier(server, mode=MODE_HINTS), classifier_mode=MODE_HINTS) + shadow_classifier = _combined_judge(server, classifier_mode="shadow") + + assert hints_only.judge(_context()).skip is False, "no pre-screen enforce, no skip to veto" + shadowed = shadow_classifier.judge(_context()) + assert shadowed.skip is True and shadowed.vetoed_by_model_signal is False, "a shadow classifier records only, so the skip stands" + assert shadowed.hints == frozenset(), "shadow hints are not merged" + + def test_release_policy_parameters_carry_no_credential(self): + server = _Server() + declared = _combined_judge(server).release_policy_parameters() + + serialized = json.dumps(declared) + assert _API_KEY not in serialized + assert "api_key_env" not in serialized + assert declared["combine"] == "auto" + + +# --- the adapters' own contracts ------------------------------------------ + + +class TestAdapterContracts: + def test_the_prescreen_decides_without_a_coordinator(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + decision = _prescreen(server).decide(MemoryPrescreenRequest(batch_text=_CHATTER_TEXT, digest="d1")) + + assert decision is not None and decision.verdict == "skip" and decision.cached is False + assert _prescreen(server).decide(MemoryPrescreenRequest(batch_text=_CHATTER_TEXT, digest="d1")) is not None + + def test_the_prescreen_sends_only_the_batch_text(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9))) + _prescreen(server).decide(MemoryPrescreenRequest(batch_text=_KEEPING_TEXT, digest="d1", signals=frozenset({"correction"}), user_id="u1")) + + body = server.bodies()[0] + assert body["state"] == {"conversation_tail": {"text": _KEEPING_TEXT}}, "no memory, no signals, no tool arguments" + assert set(body["questions"]) == {_PRESCREEN_Q} + + def test_a_recordable_model_token_is_bounded(self): + hostile = "rm -rf /tmp/secret " + "x" * 4000 + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01, model=hostile))) + decision = _prescreen(server).decide(MemoryPrescreenRequest(batch_text=_CHATTER_TEXT, digest="d1")) + + assert decision is not None + assert decision.model.startswith("unrecorded:sha256:") + assert len(decision.model) == len("unrecorded:sha256:") + 16 + assert hostile not in decision.model + + def test_a_request_level_failure_propagates_for_the_coordinator_to_record(self): + """``decide`` reports a transport failure by raising, not by returning ``None``. + + Only the adapter that made the request can tell "the request failed" from "the + provider answered with no opinion", and the round's audit record needs the + difference, so the failure is not swallowed here. + """ + server = _Server(lambda request: httpx.Response(400, json={})) + + with pytest.raises(TypeSafeError): + _prescreen(server).decide(MemoryPrescreenRequest(batch_text=_CHATTER_TEXT, digest="d1")) + with pytest.raises(TypeSafeError): + _classifier(server).decide(MemorySignalRequest(batch_text=_CHATTER_TEXT, digest="d1")) + + def test_a_question_level_failure_is_still_no_opinion(self): + """The envelope was usable, so this is not a request failure: no verdict, no raise.""" + unparsable_answer = _Server(lambda request: httpx.Response(200, json=_answer("not a number"))) + empty_envelope = _Server(lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {}})) + + assert _prescreen(unparsable_answer).decide(MemoryPrescreenRequest(batch_text=_CHATTER_TEXT, digest="d1")) is None + assert _classifier(empty_envelope).decide(MemorySignalRequest(batch_text=_CHATTER_TEXT, digest="d1")) is None + + def test_the_classifier_maps_both_directions(self): + server = _Server(lambda request: httpx.Response(200, json=_answer(0.9, 0.9, questions=(QUESTION_AFFIRMATION, QUESTION_NEGATION)))) + decision = _classifier(server).decide(MemorySignalRequest(batch_text=_KEEPING_TEXT, digest="d1")) + + assert decision is not None and decision.labels == frozenset({LABEL_REINFORCEMENT, LABEL_CORRECTION}) + + def test_a_missing_direction_is_asked_again_rather_than_read_as_a_hit(self): + """S18: a partial answer is not a full hit, and the answered direction is still consumed.""" + bodies: list[dict] = [] + + def responder(request: httpx.Request) -> httpx.Response: + bodies.append(json.loads(request.content)) + answers = {question: {"type": "noul", "noul": 0.9} for question in bodies[-1]["questions"] if question != QUESTION_NEGATION} + return httpx.Response(200, json={"model": "jev-1.13.0", "answers": answers}) + + server = _Server(responder) + classifier = _classifier(server) + request = MemorySignalRequest(batch_text=_KEEPING_TEXT, digest="d1") + + first = classifier.decide(request) + second = classifier.decide(request) + + assert set(bodies[0]["questions"]) == {QUESTION_AFFIRMATION, QUESTION_NEGATION} + assert set(bodies[1]["questions"]) == {QUESTION_NEGATION}, "the unanswered direction is sent again" + assert first is not None and second is not None + assert second.labels == frozenset({LABEL_REINFORCEMENT}), "the cached direction is reused" + + def test_a_retry_with_no_valid_answers_keeps_the_cached_answers_model(self): + """A retry that validates nothing supplied none of the consumed evidence. + + The bucket's answer was served by the first model; reporting the empty + retry's model would attribute the verdict to a model that returned no + usable answer, corrupting the classification audit record. + """ + bodies: list[dict] = [] + + def responder(request: httpx.Request) -> httpx.Response: + bodies.append(json.loads(request.content)) + if len(bodies) == 1: + return httpx.Response(200, json={"model": "jev-A", "answers": {QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}}}) + return httpx.Response(200, json={"model": "jev-B", "answers": {}}) + + server = _Server(responder) + classifier = _classifier(server) + request = MemorySignalRequest(batch_text=_KEEPING_TEXT, digest="d1") + + first = classifier.decide(request) + second = classifier.decide(request) + + assert first is not None and first.cached is False and first.model == "jev-A" + assert set(bodies[1]["questions"]) == {QUESTION_NEGATION}, "only the missing direction is retried" + assert second is not None and second.cached is True + assert second.model == "jev-A", "the empty retry's model must not replace the cached answer's model" + assert second.labels == first.labels + + def test_invalid_configuration_fails_at_construction(self): + with pytest.raises(ValueError, match="skip_threshold"): + TypeSafeMemoryPrescreen(mode=MODE_SHADOW, api_key=_API_KEY, skip_threshold=1.5) + with pytest.raises(ValueError, match="hint_threshold"): + TypeSafeSignalClassifier(mode=MODE_HINTS, api_key=_API_KEY, hint_threshold=True) + with pytest.raises(ValueError, match="max_state_chars"): + TypeSafeMemoryPrescreen(mode=MODE_SHADOW, api_key=_API_KEY, max_state_chars=0) + with pytest.raises(ValueError, match="mode"): + TypeSafeMemoryPrescreen(mode="sometimes", api_key=_API_KEY) + + def test_mode_off_resolves_nothing(self, monkeypatch): + monkeypatch.delenv("TYPESAFE_API_KEY", raising=False) + assert resolve_memory_prescreen(mode=MODE_OFF, use=_PRESCREEN) is None + assert resolve_memory_signal_classifier(mode=MODE_OFF, use=_CLASSIFIER) is None + + def test_an_enabled_mode_without_a_usable_class_path_fails(self): + with pytest.raises(ValueError, match="needs 'use'"): + resolve_memory_prescreen(mode=MODE_SHADOW, use=None) + with pytest.raises(ValueError, match="could not be resolved"): + resolve_memory_prescreen(mode=MODE_SHADOW, use="not.a.real.module:Cls") + + +# --- the audit metric ------------------------------------------------------ + + +class TestMutationsAccepted: + @pytest.mark.parametrize( + ("payload", "expected", "signal"), + [ + ('{"user":{},"history":{},"newFacts":[{"content":"User prefers dark mode","category":"preference","confidence":0.9,"scope":"user","durability":"durable","authority":"descriptive"}],"factsToRemove":[]}', 1, frozenset()), + ('{"user":{"workContext":{"shouldUpdate":true,"summary":"Works on memory","scope":"user","durability":"durable","authority":"descriptive"}},"history":{},"newFacts":[],"factsToRemove":[]}', 0, frozenset()), + ], + ) + def test_only_real_mutations_count(self, payload, expected, signal): + recorded: list = [] + updater = _updater(llm=_FakeLLM(payload), recorded=recorded) + + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1", signals=signal) + + assert recorded[0]["mutations_accepted"] == expected, "a summary rewrite is not a kept memory" + + def test_confirmation_and_removal_count(self): + memory = _memory() + memory["facts"] = [ + {"id": "fact_a", "content": "User prefers dark mode", "category": "preference", "confidence": 0.9, "createdAt": "2026-01-01T00:00:00Z", "source": "t"}, + {"id": "fact_b", "content": "Old context", "category": "context", "confidence": 0.8, "createdAt": "2026-01-01T00:00:00Z", "source": "t"}, + ] + storage = _Storage() + storage.memory = memory + config = DeerMemConfig() + config.fact_eviction_policy = "hybrid-v1" + llm = _FakeLLM('{"user":{},"history":{},"newFacts":[],"factsToReinforce":[{"id":"fact_a","scope":"user","reason":"explicit confirmation"}],"factsToRemove":[{"id":"fact_b","scope":"user","reason":"explicit retraction"}]}') + metrics: dict = {} + updater = MemoryUpdater(config, storage, llm) + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(llm.payload), metrics=metrics, signals=frozenset({"reinforcement"}), user_id="user-1") + + assert result["facts"][0]["confirmationCount"] == 1 + assert "fact_b" not in {fact["id"] for fact in result["facts"]} + assert metrics["mutations_accepted"] == 2 + + def test_a_capacity_evicted_new_fact_is_not_an_accepted_mutation(self): + """A proposal that passes the confidence gate but is evicted leaves the facts unchanged. + + Reporting the append would make a shadow skip look like a lost memory: + ``eval_memory_prescreen`` reads any positive ``mutations_accepted`` on a + skip as a miss, so the counter must track the persistent fact set. + """ + existing = [_fact(index, confidence=0.9) for index in range(10)] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "A brand new fact that capacity will discard", "category": "context", "confidence": 0.8, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), max_facts=10) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert [fact["id"] for fact in result["facts"]] == [fact["id"] for fact in existing], "the proposal was evicted, so the persistent set is unchanged" + assert metrics["mutations_accepted"] == 0, "an evicted proposal did not land" + + def test_a_surviving_new_fact_at_capacity_is_counted(self): + """The counter still reports a genuine addition when the trim actually runs.""" + existing = [_fact(index, confidence=0.75) for index in range(10)] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "A higher-confidence fact that survives the trim", "category": "context", "confidence": 0.95, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), max_facts=10) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + contents = {fact["content"] for fact in result["facts"]} + assert len(result["facts"]) == 10 + assert "A higher-confidence fact that survives the trim" in contents + assert metrics["mutations_accepted"] == 1, "the surviving addition landed" + + def test_nonexistent_removal_and_reinforcement_targets_do_not_count(self): + """A scope-valid op on an id that does not exist changes nothing. + + Counting it would report an accepted mutation that never landed, and the + pre-screen evaluation reads any positive count on a skip as a lost + memory. + """ + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [], + "factsToReinforce": [{"id": "does-not-exist", "scope": "user", "reason": "explicit confirmation"}], + "factsToRemove": [{"id": "also-missing", "scope": "user", "reason": "explicit retraction"}], + } + ) + updater = _updater(llm=_FakeLLM(payload), fact_eviction_policy="hybrid-v1") + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, signals=frozenset({"reinforcement"}), user_id="user-1") + + assert result["facts"] == [] + assert metrics["mutations_accepted"] == 0, "neither target existed, so nothing landed" + + def test_a_failed_persist_does_not_publish_the_accepted_mutation_counter(self): + """A rejected commit persisted nothing, so it must be censored, not scored. + + The apply counted a mutation, but ``save`` refused the commit; publishing the + counter would let the shadow evaluation read a failed extraction as a lost + memory instead of a missing sample (persistent-mutation contract). + """ + recorded: list = [] + payload = '{"user":{},"history":{},"newFacts":[{"content":"User prefers dark mode","category":"preference","confidence":0.9,"scope":"user","durability":"durable","authority":"descriptive"}],"factsToRemove":[]}' + updater = _updater(llm=_FakeLLM(payload), recorded=recorded) + updater._storage.save = lambda *args, **kwargs: False # persistence rejects the commit + + assert updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") is False + + assert recorded and recorded[0]["success"] is False + assert "mutations_accepted" not in recorded[0], "nothing persisted, so nothing was accepted" + + def test_a_persisted_near_duplicate_confidence_raise_counts(self): + """A merge that raises an existing fact's confidence persisted a change. + + Counting only appends made a dedup-only batch look like it wrote nothing, + so a shadow skip for that batch read as a safe saving instead of lost + information. + """ + existing = [{"id": "fact_existing", "content": "User prefers dark mode", "category": "preference", "confidence": 0.5, "createdAt": "2026-01-01T00:00:00Z", "source": "old"}] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "User prefers dark mode always", "category": "preference", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), fact_dedup_enabled=True) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert len(result["facts"]) == 1, "the proposal merged into the existing fact" + assert result["facts"][0]["confidence"] == 0.9 + assert metrics["mutations_accepted"] == 1, "the merge persisted a confidence change" + + def test_a_no_op_near_duplicate_merge_does_not_count(self): + """A merge whose proposed confidence is not higher changes nothing.""" + existing = [{"id": "fact_existing", "content": "User prefers dark mode", "category": "preference", "confidence": 0.95, "createdAt": "2026-01-01T00:00:00Z", "source": "old"}] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "User prefers dark mode always", "category": "preference", "confidence": 0.8, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), fact_dedup_enabled=True) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert result["facts"][0]["confidence"] == 0.95, "an unconfirmed restatement never lowers the stored confidence" + assert metrics["mutations_accepted"] == 0, "a no-op merge persisted nothing" + + def test_a_merge_target_evicted_by_capacity_does_not_count(self): + """A changed merge target counts only if it survives, like an appended fact.""" + existing = [{"id": "fact_existing", "content": "User prefers dark mode", "category": "preference", "confidence": 0.5, "createdAt": "2026-01-01T00:00:00Z", "source": "old"}] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [ + {"content": "User prefers dark mode always", "category": "preference", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}, + {"content": "User ships on Fridays", "category": "context", "confidence": 0.99, "scope": "user", "durability": "durable", "authority": "descriptive"}, + ], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), fact_dedup_enabled=True, max_facts=1) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert [fact["content"] for fact in result["facts"]] == ["User ships on Fridays"], "capacity kept only the highest-confidence fact" + assert metrics["mutations_accepted"] == 1, "only the surviving new fact counts; the evicted merge target's change was discarded" + + def test_a_persisted_merge_into_an_id_less_legacy_fact_counts(self): + """An id-less legacy target cannot be matched by id, so it is tracked by identity. + + Otherwise the same undercount returns for hand-edited/legacy facts, which + the staleness path already tolerates. + """ + existing = [{"content": "User prefers dark mode", "category": "preference", "confidence": 0.5, "createdAt": "2026-01-01T00:00:00Z", "source": "old"}] + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "User prefers dark mode always", "category": "preference", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload), fact_dedup_enabled=True) + updater._storage.memory = _memory_with_facts(existing) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert len(result["facts"]) == 1 and "id" not in result["facts"][0], "the target is an id-less legacy fact" + assert result["facts"][0]["confidence"] == 0.9 + assert metrics["mutations_accepted"] == 1, "the merge into the id-less target persisted a change" + + def test_an_upload_event_fact_discarded_at_finalization_does_not_count(self): + """A fact accepted by the apply but scrubbed before persistence has not landed. + + Upload events are session-scoped and scrubbed on the way to storage, so the + shadow evaluation must not read the rejected proposal as a lost memory. + """ + payload = json.dumps( + { + "user": {}, + "history": {}, + "newFacts": [{"content": "User uploaded a test file for verification purposes", "category": "context", "confidence": 0.9, "scope": "user", "durability": "durable", "authority": "descriptive"}], + "factsToRemove": [], + } + ) + updater = _updater(llm=_FakeLLM(payload)) + metrics: dict = {} + + result = updater._apply_updates(updater.get_memory_data(user_id="user-1"), json.loads(payload), metrics=metrics, user_id="user-1") + + assert result["facts"] == [], "the upload-event fact is scrubbed before persistence" + assert metrics["mutations_accepted"] == 0, "a proposal removed before persistence did not land" + + +def _patterns_dir(tmp_path, **patterns: str) -> pathlib.Path: + """A complete custom pattern set: ``detect_signals`` loads every class file. + + ``patterns_dir`` must hold all of them, so a test that overrides one class + writes an inert pattern for the rest. + """ + directory = tmp_path / "patterns" + directory.mkdir(exist_ok=True) + for name in ("correction", "reinforcement", "preference", "identity", "goal", "decision"): + (directory / f"{name}.yaml").write_text(patterns.get(name, "- pattern: 'never matches anything'\n"), encoding="utf-8") + return directory + + +# --- the production wiring: memory config -> host hooks -> DeerMem --------- + + +class TestHostWiring: + """The path the app actually takes: config -> factory hooks -> backend slot. + + The other tests inject the judge straight into ``DeerMemConfig``; this one + exercises ``_collect_host_hooks`` + ``DeerMem.from_config``, where a wiring + mistake would silently disable the feature. + """ + + def _hooks_with(self, memory: dict) -> dict: + from deerflow.config.memory_config import get_memory_config, load_memory_config_from_dict, set_memory_config + + previous = get_memory_config() + load_memory_config_from_dict(memory) + try: + from deerflow.agents.memory.manager import _collect_host_hooks + + return _collect_host_hooks() + finally: + set_memory_config(previous) + + def test_both_sides_off_injects_no_judge(self, tmp_path): + from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem + + hooks = self._hooks_with({"enabled": True}) + + assert hooks["judge"] is None, "L1: an unconfigured deployment gets no judge at all" + dm = DeerMem.from_config({"storage_path": str(tmp_path)}, **hooks) + assert dm._config.judge is None + + def test_a_configured_pre_screen_reaches_the_backend_slot(self, tmp_path): + from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem + + hooks = self._hooks_with( + { + "enabled": True, + "prescreen": { + "mode": MODE_SHADOW, + "use": _PRESCREEN, + "config": {"api_key": "SECRET-MEMORY-KEY", "skip_threshold": 0.3}, + }, + } + ) + + assert isinstance(hooks["judge"], MemorySignalCoordinator) + dm = DeerMem.from_config({"storage_path": str(tmp_path)}, **hooks) + assert dm._config.judge is hooks["judge"], "the hook the factory built is the one the updater calls" + assert dm.backend_config == {"storage_path": str(tmp_path)}, "the judge stays out of the serializable backend_config" + declared = json.dumps(hooks["judge"].release_policy_parameters()) + assert '"skip_threshold": 0.3' in declared + assert "SECRET-MEMORY-KEY" not in declared and "api_key_env" not in declared + + def test_an_enabled_mode_without_use_fails_at_config_load(self): + from deerflow.config.memory_config import MemoryConfig + + with pytest.raises(ValueError, match="memory.prescreen.use is required"): + MemoryConfig(prescreen={"mode": MODE_SHADOW}) + with pytest.raises(ValueError, match="memory.signal_classification.use is required"): + MemoryConfig(signal_classification={"mode": MODE_HINTS}) + + +class TestJudgeHotReload: + """A judging-config edit reaches the cached manager's judge (no restart). + + The manager singleton is built once, so without a refresh an ``enforce`` -> + ``off`` switch would keep skipping extraction and sending conversation text + externally, and ``off`` -> ``shadow`` would silently never record. + """ + + @pytest.fixture(autouse=True) + def _isolate(self, monkeypatch, tmp_path): + import deerflow.config.runtime_paths as runtime_paths + from deerflow.agents.memory.manager import reset_memory_manager + from deerflow.config.memory_config import get_memory_config, set_memory_config + from deerflow.config.typesafe_config import reset_typesafe_config + + monkeypatch.setattr(runtime_paths, "runtime_home", lambda: tmp_path) + previous = get_memory_config() + reset_memory_manager() + reset_typesafe_config() + yield + set_memory_config(previous) + reset_memory_manager() + reset_typesafe_config() + + def test_a_mode_toggle_rebuilds_the_injected_judge(self): + from deerflow.agents.memory.manager import get_memory_manager + from deerflow.config.memory_config import MemoryConfig, set_memory_config + + set_memory_config(MemoryConfig(manager_class="deermem")) + manager = get_memory_manager() + assert manager._config.judge is None, "both sides default off" + + set_memory_config(MemoryConfig(manager_class="deermem", prescreen={"mode": MODE_SHADOW, "use": _PRESCREEN, "config": {"api_key": "SECRET-HOT-RELOAD-KEY"}})) + reloaded = get_memory_manager() + assert reloaded is manager, "the stateful manager singleton is reused, not rebuilt" + assert isinstance(reloaded._config.judge, MemorySignalCoordinator), "off -> shadow takes effect" + assert reloaded._config.judge._prescreen_mode == MODE_SHADOW + assert "SECRET-HOT-RELOAD-KEY" not in json.dumps(reloaded._config.judge.release_policy_parameters()) + + set_memory_config(MemoryConfig(manager_class="deermem")) + assert get_memory_manager()._config.judge is None, "shadow -> off stops judging (and any external call)" + + def test_an_unchanged_judging_config_keeps_the_same_judge(self): + from deerflow.agents.memory.manager import get_memory_manager + from deerflow.config.memory_config import MemoryConfig, set_memory_config + + set_memory_config(MemoryConfig(manager_class="deermem", prescreen={"mode": MODE_ENFORCE, "use": _PRESCREEN, "config": {"api_key": _API_KEY}})) + first = get_memory_manager()._config.judge + assert get_memory_manager()._config.judge is first, "no config change, no rebuild" + + def test_a_shared_typesafe_edit_rebuilds_the_judge(self, monkeypatch): + """A judge inheriting its connection from the top-level ``typesafe`` block + must pick up an endpoint/model/credential/deadline edit there, and a + credential rotation, not just a memory-slot edit.""" + from deerflow.agents.memory.manager import get_memory_manager + from deerflow.config.memory_config import MemoryConfig, set_memory_config + from deerflow.config.typesafe_config import load_typesafe_config_from_dict + + monkeypatch.setenv("TYPESAFE_API_KEY", "first-key") + set_memory_config(MemoryConfig(manager_class="deermem", prescreen={"mode": MODE_SHADOW, "use": _PRESCREEN})) + load_typesafe_config_from_dict({"base_url": "https://first.example"}) + first = get_memory_manager()._config.judge + assert first is not None + assert "https://first.example" in json.dumps(first._prescreen.release_policy_parameters()) + + # An inherited endpoint edit invalidates without touching a memory slot. + load_typesafe_config_from_dict({"base_url": "https://second.example"}) + second = get_memory_manager()._config.judge + assert second is not first, "a shared typesafe edit must invalidate the cached judge" + assert "https://second.example" in json.dumps(second._prescreen.release_policy_parameters()) + + # A rotated env credential invalidates even though no config text changed: + # the signature carries the resolved credential fingerprint, not just the block. + monkeypatch.setenv("TYPESAFE_API_KEY", "second-key") + assert get_memory_manager()._config.judge is not second, "a rotated credential must invalidate the cached judge" + + # A credential set in the block is picked up the same way. + load_typesafe_config_from_dict({"api_key": "block-key"}) + assert get_memory_manager()._config.judge is not second, "a block credential edit must invalidate the cached judge" + + def test_an_enabled_build_does_not_land_after_a_rollback(self, monkeypatch): + """A build that started before a rollback must not be published after it. + + The judging config can be rolled back (enforce -> off) while another thread + is still building the enabled judge; installing that build afterwards would + resurrect the superseded mode for already-queued batches. Publication + revalidates the config generation, so the stale build is dropped. + """ + import threading + + from deerflow.agents.memory import manager as manager_module + from deerflow.agents.memory.manager import get_memory_manager + from deerflow.config.memory_config import MemoryConfig, set_memory_config + + set_memory_config(MemoryConfig(manager_class="deermem")) + manager = get_memory_manager() + assert manager._config.judge is None + + set_memory_config(MemoryConfig(manager_class="deermem", prescreen={"mode": MODE_ENFORCE, "use": _PRESCREEN, "config": {"api_key": _API_KEY}})) + + started = threading.Event() + released = threading.Event() + real_build = manager_module._host_default_judge + + def build_then_hold(): + judge = real_build() # the enabled judge, built before the rollback lands + started.set() + released.wait(timeout=5) + return judge + + monkeypatch.setattr(manager_module, "_host_default_judge", build_then_hold) + refresher = threading.Thread(target=manager_module._refresh_judge_for_reloaded_config, args=(manager,)) + refresher.start() + assert started.wait(timeout=5), "the enabled build started" + set_memory_config(MemoryConfig(manager_class="deermem")) # roll back to off mid-build + released.set() + refresher.join(timeout=5) + + assert not refresher.is_alive() + assert manager._config.judge is None, "a superseded enabled judge must not be published after the rollback" + + +class TestConfirmationInvariant: + """O3 invariant 1: a model verdict must not change confirmation writes (S4). + + The reinforcement gate is evidence that the user really restated something, so + it reads the deterministic signal set only. Model hints reach the hint text and + (under enforce) the veto, never this gate. + """ + + def _updater_with_fact(self, *, judge=None, llm=None, patterns_dir: str | None = None) -> MemoryUpdater: + storage = _Storage() + storage.memory = _memory() + storage.memory["facts"] = [ + {"id": "fact_a", "content": "User prefers concise answers", "category": "preference", "confidence": 0.8, "createdAt": "2026-01-01T00:00:00Z", "source": "t"}, + ] + config = DeerMemConfig() + config.fact_eviction_policy = "hybrid-v1" + config.staleness_review_enabled = False + if patterns_dir is not None: + config.patterns_dir = patterns_dir + if judge is not None: + setattr(config, _JUDGE_HOOK, judge) + return MemoryUpdater(config, storage, llm or _FakeLLM('{"user":{},"history":{},"newFacts":[],"factsToReinforce":[{"id":"fact_a","scope":"user","reason":"explicit confirmation"}],"factsToRemove":[]}')) + + @staticmethod + def _confirmation(updater: MemoryUpdater): + updater.update_memory(_conversation(), thread_id="thread-1", user_id="user-1") + fact = next(fact for fact in updater._storage.memory["facts"] if fact["id"] == "fact_a") + return fact.get("confirmationCount") + + def test_a_model_hint_alone_never_confirms(self): + hinting = _Server(lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {QUESTION_AFFIRMATION: {"type": "noul", "noul": 0.9}, QUESTION_NEGATION: {"type": "noul", "noul": 0.1}}})) + with_hints = self._updater_with_fact(judge=_combined_judge(hinting)) + without_judge = self._updater_with_fact() + + assert self._confirmation(with_hints) is None, "a model hint is not confirmation evidence" + assert self._confirmation(without_judge) is None, "and the baseline confirms nothing either" + + def test_the_deterministic_channel_still_confirms(self, tmp_path): + """Teeth: the gate is live -- it just reads the other channel. + + A custom pattern set makes the feed's re-detected reinforcement signal + match, which is the only thing that may satisfy this gate. + """ + patterns = _patterns_dir(tmp_path, reinforcement="- pattern: 'release candidate'\n") + + assert self._confirmation(self._updater_with_fact(patterns_dir=str(patterns))) == 1 + + +class TestThroughTheQueue: + """End to end through the real backend: add -> debounce queue -> updater -> judge.""" + + def test_a_queued_chatter_batch_is_skipped_without_an_extraction_call(self, tmp_path): + from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem + + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + recorded: list = [] + deer_mem = DeerMem.from_config( + {"storage_path": str(tmp_path), "staleness_review_enabled": False}, + judge=_combined_judge(server), + extraction_callback=recorded.append, + ) + llm = _FakeLLM() + deer_mem._llm = llm + deer_mem._updater._llm = llm + + deer_mem.add(thread_id="t1", messages=_conversation(_CHATTER_TEXT), user_id="u1") + deer_mem._queue.flush() + + assert llm.calls == 0, "enforce + skip saved the extraction call" + assert recorded and recorded[0]["prescreen"]["verdict"] == "skip" + assert "mutations_accepted" not in recorded[0], "no apply ran, so there is nothing to count" + # DeerMem resolves a default agent bucket, so match the thread/user scope. + assert [key for key in deer_mem._updater._watermarks if key[:2] == ("t1", "u1")], "the skipped batch is consumed, not re-judged every turn" + + def test_a_queued_batch_with_a_signal_still_extracts(self, tmp_path): + from deerflow.agents.memory.backends.deermem.deer_mem import DeerMem + + server = _Server(lambda request: httpx.Response(200, json=_answer(0.01))) + recorded: list = [] + deer_mem = DeerMem.from_config( + {"storage_path": str(tmp_path), "staleness_review_enabled": False}, + judge=_combined_judge(server), + extraction_callback=recorded.append, + ) + llm = _FakeLLM() + deer_mem._llm = llm + deer_mem._updater._llm = llm + + # A correction phrase trips detect_signals, which answers for the pre-screen. + deer_mem.add(thread_id="t2", messages=[HumanMessage(content="No, that's wrong - use tabs not spaces"), AIMessage(content="Understood.")], user_id="u1") + deer_mem._queue.flush() + + assert llm.calls == 1, "L3: a deterministic signal forces extraction" + # L3 removes only the *pre-screen's* eligibility: the classifier has no such + # rule and still asks its own questions (design §2.2.3, scenario row 6). + assert set(server.bodies()[0]["questions"]) == {QUESTION_AFFIRMATION, QUESTION_NEGATION} + assert _PRESCREEN_Q not in server.bodies()[0]["questions"], "the pre-screen was never asked" diff --git a/backend/tests/test_memory_prescreen_eval_script.py b/backend/tests/test_memory_prescreen_eval_script.py new file mode 100644 index 0000000000..dee628c948 --- /dev/null +++ b/backend/tests/test_memory_prescreen_eval_script.py @@ -0,0 +1,624 @@ +"""Tests for scripts/eval_memory_prescreen.py. + +The script is the pre-``enforce`` evidence for the memory pre-screen, so its +arithmetic has to stay honest when the record set is thin or contradictory: a +cache hit and a local fallback must never vote on the model's accuracy, a skip +whose outcome is unknown must be censored rather than counted as saved, and the +miss-rate bound must be the exact Clopper-Pearson interval it claims to be. + +Only the pure analysis logic is covered here — population classification, the +saved/miss arithmetic, the no-network baseline comparison, the confidence bound +and the gate outcomes. The JSONL loading and the report printing are plumbing. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import math +import sys +from pathlib import Path +from typing import Any + +import pytest + +SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "eval_memory_prescreen.py" + +spec = importlib.util.spec_from_file_location("deerflow_eval_memory_prescreen", SCRIPT_PATH) +assert spec is not None and spec.loader is not None +eval_script = importlib.util.module_from_spec(spec) +# dataclasses resolve ``cls.__module__`` through sys.modules, so register first. +sys.modules[spec.name] = eval_script +spec.loader.exec_module(eval_script) + + +def _payload(**overrides: Any) -> dict[str, Any]: + """A shadow record shaped exactly like the DeerMem ``extraction_callback`` payload.""" + payload: dict[str, Any] = { + "thread_id": "thread-1", + "success": True, + "token_usage": {"input_tokens": 100, "output_tokens": 20}, + "mutations_accepted": 0, + "prescreen": { + "mode": "shadow", + "verdict": "skip", + "probability": 0.05, + "skip_threshold": 0.2, + "model": "jev-latest", + "cached": False, + "digest": "digest-0", + "signals": [], + "message_count": 4, + "batch_chars": 400, + "duration_ms": 40.0, + "fallback_reason": None, + }, + } + prescreen_override = overrides.pop("prescreen", "absent") + if prescreen_override != "absent": + if prescreen_override is None: + del payload["prescreen"] + else: + payload["prescreen"].update(prescreen_override) + for key, value in overrides.items(): + if value is None: + payload.pop(key, None) + else: + payload[key] = value + return payload + + +def _record(**overrides: Any): + return eval_script.parse_record(1, _payload(**overrides)) + + +def _fallback(reason: str): + return _record(prescreen={"verdict": None, "cached": None, "fallback_reason": reason}, mutations_accepted=None, success=True) + + +def _population(records, name: str) -> dict[str, Any]: + return eval_script.summarize_population(records, name) + + +# --- classification ------------------------------------------------------- + + +def test_classify_population_is_mutually_exclusive_across_the_design_populations(): + assert eval_script.classify_population(_payload()) == eval_script.NETWORK + assert eval_script.classify_population(_payload(prescreen={"cached": True})) == eval_script.CACHE_HIT + assert eval_script.classify_population(_payload(prescreen={"verdict": None, "cached": None, "fallback_reason": "over_limit"})) == eval_script.LOCAL_FALLBACK + assert eval_script.classify_population(_payload(prescreen={"verdict": None, "cached": None, "fallback_reason": "deterministic_signals"})) == eval_script.LOCAL_FALLBACK + assert eval_script.classify_population(_payload(prescreen=None)) == eval_script.UNJUDGED + assert eval_script.classify_population({"success": True}) == eval_script.UNJUDGED + for name in (eval_script.NETWORK, eval_script.CACHE_HIT, eval_script.LOCAL_FALLBACK, eval_script.UNJUDGED): + assert name in eval_script.POPULATIONS + + +def test_local_fallback_reasons_map_to_their_design_locks(): + records = [_fallback("deterministic_signals"), _fallback("over_limit"), _fallback("emergency_flush"), _fallback("shutdown_drain"), _fallback("staleness_or_consolidation"), _fallback("no_verdict")] + summary = _population(records, eval_script.LOCAL_FALLBACK) + assert summary["records"] == 6 + assert summary["verdicts"] == {"skip": 0, "extract": 0, "none": 6} + assert summary["fallback_reasons"]["deterministic_signals"] == 1 + assert summary["fallback_locks"] == {"L3": 1, "L4": 1, "L5": 1, "L7": 1, "L8": 1, "failure": 1} + + +def test_local_fallbacks_and_cache_hits_stay_out_of_the_classifier_rates(): + records = [ + _record(mutations_accepted=0), + _record(prescreen={"verdict": "extract"}, mutations_accepted=3), + _record(prescreen={"cached": True}, mutations_accepted=5), + _fallback("over_limit"), + ] + network = _population(records, eval_script.NETWORK) + assert network["records"] == 2 + assert network["verdicts"] == {"skip": 1, "extract": 1, "none": 0} + assert network["scored_skips"] == 1 + assert network["misses"] == 0 + assert network["miss_rate"] == 0.0 + assert _population(records, eval_script.CACHE_HIT)["misses"] == 1 + assert _population(records, eval_script.LOCAL_FALLBACK)["scored_skips"] == 0 + + +def test_sensitive_labels_are_read_from_stratum_and_labels_and_normalised(): + record = eval_script.parse_record(1, _payload(stratum="Preference", labels=["identity", "chatter", 7])) + assert record.sensitive_labels == {"preference", "identity"} + assert record.sensitive is True + assert eval_script.parse_record(1, _payload(stratum="chatter")).sensitive is False + + +# --- saved / miss arithmetic --------------------------------------------- + + +def test_saved_and_miss_rates_are_complementary_over_scored_skips(): + records = [_record(mutations_accepted=0), _record(mutations_accepted=0), _record(mutations_accepted=0), _record(mutations_accepted=2), _record(prescreen={"verdict": "extract"}, mutations_accepted=1)] + summary = _population(records, eval_script.NETWORK) + assert summary["skip_rate"] == 4 / 5 + assert summary["scored_skips"] == 4 + assert summary["saved_calls"] == 3 + assert summary["saved_call_rate"] == 0.75 + assert summary["misses"] == 1 + assert summary["miss_rate"] == 0.25 + assert summary["saved_call_rate"] + summary["miss_rate"] == 1.0 + assert summary["saved_call_rate_over_all_verdicts"] == 3 / 5 + + +def test_a_skip_without_an_outcome_is_censored_not_counted_as_saved(): + censored_skip = _record(mutations_accepted=None) + failed_extract = _record(prescreen={"verdict": "extract"}, mutations_accepted=None, success=False) + scored_skip = _record(mutations_accepted=0) + records = [censored_skip, failed_extract, scored_skip] + summary = _population(records, eval_script.NETWORK) + assert censored_skip.censored is True + assert censored_skip.censored_reason == eval_script.CENSORED_OUTCOME_NOT_RECORDED + assert failed_extract.censored is True + assert failed_extract.censored_reason == eval_script.CENSORED_EXTRACTION_FAILED + assert scored_skip.censored is False + assert summary["censored"] == 2 + assert summary["censored_reasons"] == {eval_script.CENSORED_OUTCOME_NOT_RECORDED: 1, eval_script.CENSORED_EXTRACTION_FAILED: 1} + assert summary["scored_skips"] == 1 + assert summary["saved_call_rate"] == 1.0 + + +def test_a_manual_review_is_a_second_ground_truth_witness_for_a_skip(): + reviewed_saved = _record(mutations_accepted=None, reviewed=True, review_outcome="none_worth_remembering") + reviewed_missed = _record(mutations_accepted=0, reviewed=True, review_outcome="worth_remembering") + contradicted = _record(mutations_accepted=2, reviewed=True, review_outcome="none_worth_remembering") + summary = _population([reviewed_saved, reviewed_missed, contradicted], eval_script.NETWORK) + assert reviewed_saved.censored is False and reviewed_saved.miss is False + assert reviewed_missed.miss is True + assert contradicted.miss is True, "a review saying not-worth never clears a counter that says otherwise" + assert summary["scored_skips"] == 3 + assert summary["saved_calls"] == 1 + assert summary["misses"] == 2 + + +def test_a_negative_count_is_absent_not_a_saved_call(): + """A corrupt counter must censor the sample rather than read as "nothing lost". + + Scoring ``-3`` as a saved call (it is not a miss) added a phantom saving to both + the numerator and the scored-skip denominator, tilting every gate toward PASS on + input that could not have come from the apply site. + """ + record = _record(mutations_accepted=-3) + + assert record.mutations_accepted is None + assert record.scorable is False + assert record.censored is True + assert record.censored_reason == eval_script.CENSORED_OUTCOME_NOT_RECORDED + + report = eval_script.build_report([record], source="corrupt.jsonl") + assert report["model"]["scored_skips"] == 0 + assert report["model"]["saved_calls"] == 0 + assert report["model"]["miss_rate"] is None + + +def test_missed_digests_are_reported_for_audit(): + records = [_record(prescreen={"digest": "lost-1"}, mutations_accepted=1), _record(prescreen={"digest": "kept-0"}, mutations_accepted=0)] + summary = _population(records, eval_script.NETWORK) + assert summary["missed_digests"] == ["lost-1"] + + +# --- heuristic baseline --------------------------------------------------- + + +def test_baseline_comparison_scores_both_policies_on_the_same_records(): + records = [ + _record(trivial_only=True, mutations_accepted=0), # baseline skip, saved + _record(trivial_only=True, mutations_accepted=1), # baseline skip, missed + _record(trivial_only=False, mutations_accepted=0), # baseline extracts + _record(prescreen={"signals": ["preference"]}, trivial_only=True, mutations_accepted=0), # a signal blocks the baseline + ] + baseline = eval_script.baseline_summary(records) + assert baseline["evaluated_verdicts"] == 4 + assert baseline["computable"] == 4 + assert baseline["coverage"] == 1.0 + assert baseline["skips"] == 2 + assert baseline["saved_calls"] == 1 + assert baseline["misses"] == 1 + assert baseline["saved_call_rate"] == 0.5 + assert baseline["miss_rate"] == 0.5 + model = baseline["model"] + assert model["skips"] == 4 and model["saved_calls"] == 3 and model["misses"] == 1 + assert model["saved_call_rate"] == 0.75 + assert baseline["incremental_saved_calls"] == 2 + assert baseline["incremental_saved_call_rate"] == 2 / 4, "incremental saving is measured over the shared scored set" + + +def test_incremental_saving_uses_the_shared_denominator_not_the_conditional_rates(): + """Different skip rates: subtracting conditional rates hides real call savings. + + Both policies saved every batch they skipped, so both conditional rates are + 1.0 and their difference is 0 — yet the model skipped 70 more batches. The + incremental rate must be (model_saved - baseline_saved) / scored. + """ + records = ( + [_record(trivial_only=True, mutations_accepted=0) for _ in range(10)] # both skip, harmless + + [_record(trivial_only=False, mutations_accepted=0) for _ in range(70)] # model-only skip, harmless + + [_record(trivial_only=False, prescreen={"verdict": "extract"}, mutations_accepted=1) for _ in range(20)] # neither skips + ) + baseline = eval_script.baseline_summary(records) + + assert baseline["scored"] == 100 + assert baseline["saved_calls"] == 10 + assert baseline["model"]["saved_calls"] == 80 + assert baseline["saved_call_rate"] == 1.0 and baseline["model"]["saved_call_rate"] == 1.0, "the conditional rates stay in the report" + assert baseline["incremental_saved_calls"] == 70 + assert baseline["incremental_saved_call_rate"] == 70 / 100, "70 extra saved calls over 100 scored records" + + +def test_baseline_reports_coverage_instead_of_shrinking_the_denominator(): + records = [ + _record(mutations_accepted=0), + _record(trivial_only=True, mutations_accepted=0), + _record(prescreen={"signals": "unreadable"}, trivial_only=True), + ] + baseline = eval_script.baseline_summary(records) + assert baseline["evaluated_verdicts"] == 3 + assert baseline["computable"] == 1 + assert baseline["coverage"] == 1 / 3 + assert baseline["trivial_only_missing"] == 1 + assert baseline["signals_missing"] == 1 + assert baseline["skips"] == 1 + assert baseline["incremental_saved_calls"] == 0 + + empty = eval_script.baseline_summary([_record()]) + assert empty["computable"] == 0 + assert empty["coverage"] == 0.0 + assert empty["saved_call_rate"] is None and empty["miss_rate"] is None + + +# --- confidence bound ----------------------------------------------------- + + +def test_clopper_pearson_upper_matches_known_reference_values(): + # Zero misses: the bound is 1 - alpha ** (1 / n) exactly. + assert math.isclose(eval_script.clopper_pearson_upper(0, 10), 1 - 0.05**0.1, rel_tol=1e-9) + assert math.isclose(eval_script.clopper_pearson_upper(0, 10), 0.25887, abs_tol=1e-4) + # Published table value: the two-sided 95% upper limit of 0/20 is 0.16843. + assert math.isclose(eval_script.clopper_pearson_upper(0, 20, confidence=0.975), 0.16843, abs_tol=1e-4) + assert eval_script.clopper_pearson_upper(0, 0) is None + assert eval_script.clopper_pearson_upper(4, 4) == 1.0 + + +def test_clopper_pearson_upper_satisfies_its_defining_equation(): + alpha = 1 - eval_script.DEFAULT_CONFIDENCE + for misses, skips in ((0, 200), (1, 200), (3, 50), (7, 30), (2, 3)): + bound = eval_script.clopper_pearson_upper(misses, skips) + solved = eval_script._regularized_incomplete_beta(skips - misses, misses + 1, 1 - bound) + assert math.isclose(solved, alpha, rel_tol=1e-6), (misses, skips, solved) + + +def test_clopper_pearson_upper_is_monotone_in_samples_and_in_misses(): + assert eval_script.clopper_pearson_upper(0, 10) > eval_script.clopper_pearson_upper(0, 200) + assert eval_script.clopper_pearson_upper(0, 200) > eval_script.clopper_pearson_upper(0, 2000) + assert eval_script.clopper_pearson_upper(1, 200) > eval_script.clopper_pearson_upper(0, 200) + assert eval_script.clopper_pearson_upper(5, 200) > eval_script.clopper_pearson_upper(1, 200) + for misses, skips in ((0, 1), (1, 10), (9, 10), (0, 200)): + bound = eval_script.clopper_pearson_upper(misses, skips) + assert misses / skips <= bound <= 1.0 + + +def test_clopper_pearson_upper_rejects_an_impossible_confidence(): + with pytest.raises(ValueError): + eval_script.clopper_pearson_upper(0, 10, confidence=1.0) + + +# --- gates ---------------------------------------------------------------- + + +def _gates(evidence, coverage) -> dict[str, str]: + return {gate.gate_id: gate.status for gate in eval_script.evaluate_gates(evidence, coverage)} + + +def _evidence(**overrides: Any): + base = { + # 300 skips is the smallest zero-miss sample whose exact upper bound is at or + # below the 1% target: 1 - 0.05 ** (1 / 300) = 0.0099. + "scored_skips": 300, + "misses": 0, + "saved_calls": 300, + "sensitive_scored_skips": {"identity": 2, "preference": 2, "correction": 1}, + "sensitive_misses": {"identity": 0, "preference": 0, "correction": 0}, + "reviewed_skips": 200, + "reviewed_scored_skips": 200, + "reviewed_missed": 0, + "saved_skips_with_tokens": 300, + "latency_samples": 300, + "baseline_computable": 300, + # The paired comparison: the heuristic saves nothing here, so the model's 300 + # saved calls are all incremental (an equal count is a FAIL, and a paired + # subset smaller than the scored skips is INSUFFICIENT — see the tests below). + "baseline_paired_skips": 300, + "baseline_model_saved_calls": 300, + "baseline_heuristic_saved_calls": 0, + } + base.update(overrides) + return eval_script.Evidence(**base) + + +_CLEAN_COVERAGE = eval_script.Coverage(records=310, censored=10, malformed_lines=0) + + +def test_every_gate_passes_on_a_clean_run(): + assert _gates(_evidence(), _CLEAN_COVERAGE) == { + "network_miss_rate": eval_script.PASS, + "sensitive_stratum_zero_misses": eval_script.PASS, + "reviewed_skips_confirm_no_loss": eval_script.PASS, + "censoring_disclosed": eval_script.PASS, + "savings_recorded": eval_script.PASS, + "baseline_incremental_savings": eval_script.PASS, + } + + +@pytest.mark.parametrize( + ("overrides", "coverage", "expected"), + [ + ({"misses": 4, "saved_calls": 296}, None, "network_miss_rate"), + ({"sensitive_misses": {"identity": 1}}, None, "sensitive_stratum_zero_misses"), + ({"reviewed_skips": 199}, None, "reviewed_skips_confirm_no_loss"), + ({"reviewed_missed": 1}, None, "reviewed_skips_confirm_no_loss"), + ({}, eval_script.Coverage(records=310, censored=10, malformed_lines=2), "censoring_disclosed"), + ({"saved_calls": 0, "misses": 250}, None, "savings_recorded"), + ({"baseline_heuristic_saved_calls": 300}, None, "baseline_incremental_savings"), + ], +) +def test_a_failing_gate_is_reported_as_fail(overrides, coverage, expected): + statuses = _gates(_evidence(**overrides), coverage or _CLEAN_COVERAGE) + assert statuses[expected] == eval_script.FAIL + + +def test_gates_report_insufficient_instead_of_passing_vacuously(): + empty = eval_script.Evidence( + scored_skips=0, + misses=0, + saved_calls=0, + sensitive_scored_skips={}, + sensitive_misses={}, + reviewed_skips=0, + reviewed_scored_skips=0, + reviewed_missed=0, + saved_skips_with_tokens=0, + latency_samples=0, + baseline_computable=0, + baseline_paired_skips=0, + baseline_model_saved_calls=0, + baseline_heuristic_saved_calls=0, + ) + statuses = _gates(empty, eval_script.Coverage(records=0, censored=0, malformed_lines=0)) + assert set(statuses.values()) == {eval_script.INSUFFICIENT} + assert statuses["network_miss_rate"] == eval_script.INSUFFICIENT + + thin = eval_script.Evidence( + scored_skips=2, + misses=0, + saved_calls=2, + sensitive_scored_skips={}, + sensitive_misses={}, + reviewed_skips=0, + reviewed_scored_skips=0, + reviewed_missed=0, + saved_skips_with_tokens=2, + latency_samples=2, + baseline_computable=2, + baseline_paired_skips=2, + baseline_model_saved_calls=2, + baseline_heuristic_saved_calls=0, + ) + statuses = _gates(thin, eval_script.Coverage(records=2, censored=0, malformed_lines=0)) + assert statuses["network_miss_rate"] == eval_script.INSUFFICIENT, "two clean skips cannot establish a 1% target: the exact upper bound is 0.78" + assert statuses["sensitive_stratum_zero_misses"] == eval_script.INSUFFICIENT + assert statuses["reviewed_skips_confirm_no_loss"] == eval_script.INSUFFICIENT + + +def test_the_review_gate_needs_a_confirmed_review_not_only_a_count(): + """Design §5 gate 3: ≥200 reviewed skips *confirming* none was worth remembering.""" + gate_id = "reviewed_skips_confirm_no_loss" + assert _gates(_evidence(reviewed_missed=1), _CLEAN_COVERAGE)[gate_id] == eval_script.FAIL + assert _gates(_evidence(reviewed_scored_skips=199), _CLEAN_COVERAGE)[gate_id] == eval_script.INSUFFICIENT, "a reviewed skip with no outcome confirms nothing" + + +def test_the_savings_gate_needs_tokens_latency_and_the_baseline(): + """Design §5 gate 4: recorded calls are not the whole evidence, so missing pieces are INSUFFICIENT.""" + for overrides in ({"saved_skips_with_tokens": 0}, {"latency_samples": 0}, {"baseline_computable": 0}): + statuses = _gates(_evidence(**overrides), _CLEAN_COVERAGE) + assert statuses["savings_recorded"] == eval_script.INSUFFICIENT, overrides + assert _gates(_evidence(), _CLEAN_COVERAGE)["savings_recorded"] == eval_script.PASS + + +def test_the_sensitive_gate_needs_evidence_for_each_stratum_not_a_pooled_sample(): + """A pooled identity/preference/correction sample hides a stratum with no evidence. + + An identity-only run with zero misses must not pass the advertised zero-miss + check across all three strata; the uncovered strata make it INSUFFICIENT. + """ + identity_only = _evidence(sensitive_scored_skips={"identity": 5}, sensitive_misses={"identity": 0}) + assert _gates(identity_only, _CLEAN_COVERAGE)["sensitive_stratum_zero_misses"] == eval_script.INSUFFICIENT + + all_covered = _evidence(sensitive_scored_skips={"identity": 5, "preference": 5, "correction": 5}, sensitive_misses={"identity": 0, "preference": 0, "correction": 0}) + assert _gates(all_covered, _CLEAN_COVERAGE)["sensitive_stratum_zero_misses"] == eval_script.PASS + + one_miss = _evidence(sensitive_scored_skips={"identity": 5, "preference": 5}, sensitive_misses={"preference": 1}) + assert _gates(one_miss, _CLEAN_COVERAGE)["sensitive_stratum_zero_misses"] == eval_script.FAIL, "a recorded miss outranks missing strata" + + +def test_miss_rate_gate_reports_the_exact_bound_and_the_sample_size(): + gate = next(gate for gate in eval_script.evaluate_gates(_evidence(), _CLEAN_COVERAGE) if gate.gate_id == "network_miss_rate") + + assert gate.status == eval_script.PASS + assert "n=300" in gate.detail + assert f"{eval_script.clopper_pearson_upper(0, 300):.4f}" in gate.detail + + +def test_the_miss_rate_gate_needs_the_bound_not_only_the_point_estimate(): + """A zero-low point estimate on a thin sample must not be reported as proof. + + ``misses=1`` of 300 is a 0.33% point estimate, but its one-sided 95% upper bound + is about 2%, so the sample does not establish the 1% target: the gate says + INSUFFICIENT and the detail carries the bound, not just the estimate. + """ + gate = next(gate for gate in eval_script.evaluate_gates(_evidence(misses=1, saved_calls=299), _CLEAN_COVERAGE) if gate.gate_id == "network_miss_rate") + + assert gate.status == eval_script.INSUFFICIENT + assert f"{eval_script.clopper_pearson_upper(1, 300):.4f}" in gate.detail + assert "too few skips" in gate.detail + assert "n=300" in gate.detail + + +def test_the_baseline_gate_needs_a_gain_over_the_heuristic(): + """The heuristic is already in the stack, so a tie is not a reason to egress.""" + gate_id = "baseline_incremental_savings" + + assert _gates(_evidence(baseline_heuristic_saved_calls=300), _CLEAN_COVERAGE)[gate_id] == eval_script.FAIL, "the model saved exactly what the heuristic would have" + assert _gates(_evidence(baseline_heuristic_saved_calls=299), _CLEAN_COVERAGE)[gate_id] == eval_script.PASS, "one incremental call is a gain" + unannotated = _evidence(baseline_computable=0, baseline_paired_skips=0, baseline_model_saved_calls=0, baseline_heuristic_saved_calls=0) + assert _gates(unannotated, _CLEAN_COVERAGE)[gate_id] == eval_script.INSUFFICIENT, "without the baseline field there is no comparison to make" + + +def test_the_baseline_gate_refuses_a_comparison_over_a_subset_of_the_skips(): + """A paired comparison that misses most scored skips is not evidence of a gain. + + The two sides must be the same records: subtracting a subset's heuristic savings + from model savings over every scorable skip credits the model for batches the + baseline never saw. + """ + gate_id = "baseline_incremental_savings" + partial = _evidence(baseline_paired_skips=1, baseline_model_saved_calls=1, baseline_heuristic_saved_calls=0) + + assert _gates(partial, _CLEAN_COVERAGE)[gate_id] == eval_script.INSUFFICIENT, "1 paired skip cannot stand for 300" + detail = next(gate for gate in eval_script.evaluate_gates(partial, _CLEAN_COVERAGE) if gate.gate_id == gate_id).detail + assert "1 of 300" in detail + + +# --- report assembly ------------------------------------------------------ + + +def test_build_report_carries_the_gates_the_evidence_and_the_censoring(): + records = [ + _record(prescreen={"digest": "saved-1"}, mutations_accepted=0, trivial_only=True), + _record(prescreen={"digest": "lost-1"}, mutations_accepted=2, trivial_only=True), + _record(prescreen={"digest": "lost-2"}, mutations_accepted=1, stratum="preference"), + _record(prescreen={"verdict": "extract"}, mutations_accepted=0), + _record(mutations_accepted=None), + _record(prescreen=None, mutations_accepted=None), + ] + report = eval_script.build_report(records, source="test.jsonl", malformed_lines=1, filtered_out=0) + assert report["populations"][eval_script.NETWORK]["records"] == 5 + assert report["populations"][eval_script.UNJUDGED]["records"] == 1 + assert report["model"]["scored_skips"] == 3 + assert report["model"]["misses"] == 2 + assert report["model"]["miss_rate"] == pytest.approx(2 / 3) + assert report["model"]["missed_digests"] == ["lost-1", "lost-2"] + assert report["sensitive_stratum"]["by_label"]["preference"] == {"scored_skips": 1, "misses": 1} + assert report["censoring"]["censored"] == 1 + statuses = {gate["id"]: gate["status"] for gate in report["gates"]} + assert statuses["network_miss_rate"] == eval_script.FAIL + assert statuses["sensitive_stratum_zero_misses"] == eval_script.FAIL + assert statuses["censoring_disclosed"] == eval_script.FAIL + assert report["gate_summary"]["enable_recommended"] is False + assert report["baseline_heuristic"]["computable"] == 2 + + +def test_build_report_refuses_enforce_when_a_review_contradicts_a_skip(): + """The reviewer's counterexample: 200 reviewed skips with one worth remembering must not enable.""" + records = [_record(prescreen={"digest": f"d{i}"}, mutations_accepted=0, trivial_only=True, reviewed=True, review_outcome="none_worth_remembering", stratum="identity" if i < 5 else None) for i in range(199)] + records.append(_record(prescreen={"digest": "lost"}, mutations_accepted=0, trivial_only=True, reviewed=True, review_outcome="worth_remembering")) + report = eval_script.build_report(records, source="review.jsonl") + + assert report["review"]["reviewed_skips"] == 200 + assert report["review"]["reviewed_missed"] == 1 + gates = {gate["id"]: gate["status"] for gate in report["gates"]} + assert gates["reviewed_skips_confirm_no_loss"] == eval_script.FAIL + assert report["gate_summary"]["enable_recommended"] is False + + +def test_build_report_treats_a_record_set_without_savings_evidence_as_insufficient(): + """The reviewer's counterexample: 250 saved calls with no tokens, latency or baseline must not enable.""" + records = [_record(prescreen={"digest": f"d{i}", "duration_ms": None}, token_usage=None, trivial_only=None, mutations_accepted=0) for i in range(250)] + report = eval_script.build_report(records, source="thin.jsonl") + + assert report["model"]["saved_calls"] == 250 + assert report["baseline_heuristic"]["computable"] == 0 + gates = {gate["id"]: gate["status"] for gate in report["gates"]} + assert gates["savings_recorded"] == eval_script.INSUFFICIENT + assert report["gate_summary"]["enable_recommended"] is False + + +def test_build_report_enables_when_every_gate_passes(): + """The enabling shape: enough skips for the bound, and calls the heuristic cannot save. + + ``trivial_only=False`` is what makes the baseline comparison meaningful — the + heuristic skips only trivial, signal-free batches, so a batch it would have + extracted is exactly the saving the model adds. + """ + strata = ("identity", "preference", "correction") + records = [_record(prescreen={"digest": f"d{i}"}, mutations_accepted=0, trivial_only=False, reviewed=True, review_outcome="none_worth_remembering", stratum=strata[i] if i < 3 else None) for i in range(300)] + report = eval_script.build_report(records, source="clean.jsonl") + + assert report["model"]["scored_skips"] == 300 + assert report["model"]["misses"] == 0 + assert report["baseline_heuristic"]["saved_calls"] == 0 + assert report["gate_summary"]["enable_recommended"] is True + assert report["model"]["saved_call_rate_over_all_verdicts"] == 1.0 + + +def test_build_report_refuses_enforce_without_a_gain_over_the_heuristic(): + """The reviewer's counterexample: a harmless all-trivial sample saves nothing extra.""" + strata = ("identity", "preference", "correction") + records = [_record(prescreen={"digest": f"d{i}"}, mutations_accepted=0, trivial_only=True, reviewed=True, review_outcome="none_worth_remembering", stratum=strata[i] if i < 3 else None) for i in range(300)] + report = eval_script.build_report(records, source="trivial.jsonl") + + assert report["baseline_heuristic"]["saved_calls"] == 300 + gates = {gate["id"]: gate["status"] for gate in report["gates"]} + assert gates["baseline_incremental_savings"] == eval_script.FAIL + assert report["gate_summary"]["enable_recommended"] is False + + +def test_build_report_refuses_enforce_when_the_baseline_covers_only_a_subset(): + """The reviewer's follow-up: 299 unannotated skips must not read as a 300-vs-1 gain. + + Only one record carries ``trivial_only``, so the paired comparison is model=1 + against heuristic=1 — no incremental saving. Comparing model savings over *all* + scorable skips against that subset's heuristic savings read 300-1=299 and enabled + ``enforce`` on records the baseline never saw. + """ + strata = ("identity", "preference", "correction") + records = [_record(prescreen={"digest": f"d{i}"}, mutations_accepted=0, trivial_only=True if i == 0 else None, reviewed=True, review_outcome="none_worth_remembering", stratum=strata[i] if i < 3 else None) for i in range(300)] + report = eval_script.build_report(records, source="unannotated.jsonl") + + assert report["model"]["saved_calls"] == 300 + assert report["baseline_heuristic"]["saved_calls"] == 1, "the heuristic can only be scored on the one annotated skip" + assert report["baseline_heuristic"]["incremental_saved_calls"] == 0, "the paired comparison saves nothing" + gates = {gate["id"]: gate["status"] for gate in report["gates"]} + assert gates["baseline_incremental_savings"] == eval_script.INSUFFICIENT + assert report["gate_summary"]["enable_recommended"] is False + + +# --- label filtering ------------------------------------------------------ + + +def test_label_filter_matches_dotted_paths_and_typed_values(): + record = _payload(stratum="preference") + assert eval_script.record_matches(record, [("prescreen.mode", "shadow")]) is True + assert eval_script.record_matches(record, [("prescreen.mode", "enforce")]) is False + assert eval_script.record_matches(record, [("prescreen.cached", "false")]) is True + assert eval_script.record_matches(record, [("success", "true")]) is True + assert eval_script.record_matches(record, [("mutations_accepted", "0")]) is True + assert eval_script.record_matches(record, [("prescreen.mode", "shadow"), ("stratum", "preference")]) is True + assert eval_script.record_matches(record, [("prescreen.mode", "shadow"), ("stratum", "identity")]) is False + assert eval_script.record_matches(record, [("prescreen.missing", "shadow")]) is False + assert eval_script.record_matches(record, [("prescreen.mode", "not json")]) is False + + +def test_label_filter_rejects_a_missing_equals_sign(): + assert eval_script.parse_label_filter("prescreen.mode=shadow") == ("prescreen.mode", "shadow") + assert eval_script.parse_label_filter("stratum=") == ("stratum", "") + with pytest.raises(argparse.ArgumentTypeError): + eval_script.parse_label_filter("prescreen.mode") + + +def test_percentile_uses_the_nearest_rank(): + assert eval_script._percentile([], 0.95) is None + assert eval_script._percentile([5.0], 0.95) == 5.0 + assert eval_script._percentile([1.0, 2.0, 3.0, 4.0], 0.95) == 4.0 + assert eval_script._percentile(list(range(1, 101)), 0.95) == 95 diff --git a/backend/tests/test_memory_queue.py b/backend/tests/test_memory_queue.py index e9f5461f0f..0597a041bf 100644 --- a/backend/tests/test_memory_queue.py +++ b/backend/tests/test_memory_queue.py @@ -38,6 +38,7 @@ def test_process_queue_forwards_correction_flag_to_updater() -> None: user_id=None, trace_id=None, bypass_watermark=False, + judge=True, ) @@ -68,6 +69,7 @@ def test_process_queue_forwards_reinforcement_flag_to_updater() -> None: user_id=None, trace_id=None, bypass_watermark=False, + judge=True, ) @@ -224,8 +226,8 @@ def test_process_queue_updates_different_agents_in_same_thread_separately() -> N assert mock_updater.update_memory.call_count == 2 mock_updater.update_memory.assert_has_calls( [ - call(messages=["agent-a"], thread_id="thread-1", agent_name="agent-a", signals=frozenset(), user_id=None, trace_id=None, bypass_watermark=False), - call(messages=["agent-b"], thread_id="thread-1", agent_name="agent-b", signals=frozenset(), user_id=None, trace_id=None, bypass_watermark=False), + call(messages=["agent-a"], thread_id="thread-1", agent_name="agent-a", signals=frozenset(), user_id=None, trace_id=None, bypass_watermark=False, judge=True), + call(messages=["agent-b"], thread_id="thread-1", agent_name="agent-b", signals=frozenset(), user_id=None, trace_id=None, bypass_watermark=False, judge=True), ] ) @@ -246,6 +248,7 @@ def test_process_queue_forwards_trace_id_to_updater() -> None: user_id=None, trace_id="trace-memory-1", bypass_watermark=False, + judge=True, ) @@ -290,6 +293,7 @@ def test_flush_sync_drains_pending_queue_and_returns_true() -> None: user_id=None, trace_id=None, bypass_watermark=False, + judge=False, ) @@ -502,5 +506,6 @@ def test_cancel_by_agent_does_not_touch_in_flight_batch() -> None: user_id=in_flight[0].user_id, trace_id=None, bypass_watermark=False, + judge=True, ) mock_updater.update_memory.assert_called_once() diff --git a/backend/tests/test_typesafe_client.py b/backend/tests/test_typesafe_client.py new file mode 100644 index 0000000000..ad0ee91c7c --- /dev/null +++ b/backend/tests/test_typesafe_client.py @@ -0,0 +1,446 @@ +"""Tests for the shared ``deerflow.typesafe`` client. + +The three provider suites pin the tool gate's behaviour *through* the client; +these pin what the client adds on its own: + +* per-question partial success — one bad answer must not discard the good ones in + the same response (design §2.3), and the tool gate must keep reporting its own + question's failure exactly as it did before the parser was split; +* byte counting that replaces no character limit (design §2.4); +* the sharing identity (design §4 rule 2) — see ``test_typesafe_config.py`` for + the identity split, this file covers the transport contract. +""" + +from __future__ import annotations + +import asyncio +import gzip +import json +import time +import traceback +from collections.abc import AsyncIterator, Iterator + +import httpx +import pytest + +from deerflow.guardrails.provider import GuardrailRequest +from deerflow.guardrails.typesafe import TypeSafeGuardrailError, TypeSafeGuardrailProvider +from deerflow.typesafe.client import ( + CATEGORY_LABEL, + CATEGORY_MISSING, + CATEGORY_PROBABILITY, + CATEGORY_TYPE, + MAX_RESPONSE_BYTES, + QUESTION_CHOICE, + QUESTION_NOUL, + ChoiceAnswer, + NoulAnswer, + Question, + TypeSafeClient, + wire_size, +) +from deerflow.typesafe.connection import resolve_connection +from deerflow.typesafe.errors import CAUSE_DEADLINE, CAUSE_INVALID_RESPONSE, TypeSafeError + +_API_KEY = "shared-client-test-key" +_FIRST = "first_question" +_SECOND = "second_question" +_GATE_QUESTION = "risky_tool_call" + +_NOUL = Question(type=QUESTION_NOUL, instructions="Does this matter?") +_CHOICE = Question(type=QUESTION_CHOICE, instructions="Which label?", criteria={"keep": "worth keeping", "drop": "not worth keeping"}) + + +class _Server: + """Fake System One endpoint that records every request it receives.""" + + def __init__(self, responder=None) -> None: + self.requests: list[httpx.Request] = [] + self._responder = responder or (lambda request: httpx.Response(200, json=_response({_GATE_QUESTION: {"type": "noul", "noul": 0.9}}))) + + def handle(self, request: httpx.Request) -> httpx.Response: + self.requests.append(request) + return self._responder(request) + + def transport(self) -> httpx.MockTransport: + return httpx.MockTransport(self.handle) + + @property + def count(self) -> int: + return len(self.requests) + + +def _response(answers: dict, model: str = "jev-1.13.0") -> dict: + return {"model": model, "answers": answers} + + +def _ask(server: _Server, questions: dict[str, Question], **settings): + connection = resolve_connection(settings={"api_key": _API_KEY, **settings}, configuration_source="tests.typesafe") + return TypeSafeClient(connection, transport_factory=server.transport).ask({"tool_call": {"name": "bash", "arguments": "{}"}}, questions) + + +def _gate(server: _Server, **kwargs) -> TypeSafeGuardrailProvider: + return TypeSafeGuardrailProvider(api_key=_API_KEY, transport_factory=server.transport, **kwargs) + + +def _tool_call() -> GuardrailRequest: + return GuardrailRequest(tool_name="bash", tool_input={"command": "ls"}) + + +# --- per-question partial success ----------------------------------------- + + +class TestPartialSuccess: + def test_a_valid_answer_survives_an_invalid_one(self): + server = _Server(lambda request: httpx.Response(200, json=_response({_FIRST: {"type": "noul", "noul": 0.25}, _SECOND: {"type": "noul", "noul": "0.9"}}))) + + answer_set = _ask(server, {_FIRST: _NOUL, _SECOND: _NOUL}) + + assert answer_set.answers[_FIRST] == NoulAnswer(probability=0.25) + assert answer_set.noul(_SECOND) is None + error = answer_set.errors_by_question[_SECOND] + assert error.category == CATEGORY_PROBABILITY + assert "type=str" in error.message + assert "0.9" not in error.message, "the response body is never repeated into a message" + + def test_an_oversized_noul_integer_does_not_discard_a_valid_sibling(self): + """A JSON integer too large for float() is a per-question error, not a crash. + + ``float(10**400)`` raises OverflowError; letting it escape ``_parse`` would + discard the valid answer in the same response (design §2.3 isolation). + """ + server = _Server(lambda request: httpx.Response(200, json=_response({_FIRST: {"type": "noul", "noul": 0.25}, _SECOND: {"type": "noul", "noul": 10**400}}))) + + answer_set = _ask(server, {_FIRST: _NOUL, _SECOND: _NOUL}) + + assert answer_set.answers[_FIRST] == NoulAnswer(probability=0.25), "the valid sibling survives" + assert answer_set.noul(_SECOND) is None + assert answer_set.errors_by_question[_SECOND].category == CATEGORY_PROBABILITY + + @pytest.mark.parametrize( + ("answers", "questions", "category"), + [ + ({}, {_FIRST: _NOUL}, CATEGORY_MISSING), + ({_FIRST: "not an object"}, {_FIRST: _NOUL}, CATEGORY_TYPE), + ({_FIRST: {"type": "choice", "choice": "keep"}}, {_FIRST: _NOUL}, CATEGORY_TYPE), + ({_FIRST: {"type": "noul", "noul": 1.5}}, {_FIRST: _NOUL}, CATEGORY_PROBABILITY), + ({_FIRST: {"type": "noul", "noul": True}}, {_FIRST: _NOUL}, CATEGORY_PROBABILITY), + ({_FIRST: {"type": "choice", "choice": "unknown"}}, {_FIRST: _CHOICE}, CATEGORY_LABEL), + ({_FIRST: {"type": "choice", "choice": 7}}, {_FIRST: _CHOICE}, CATEGORY_LABEL), + ], + ) + def test_a_bad_answer_is_reported_per_question_not_per_request(self, answers, questions, category): + server = _Server(lambda request: httpx.Response(200, json=_response(answers))) + + answer_set = _ask(server, questions) + + assert answer_set.answers == {} + assert answer_set.errors_by_question[_FIRST].category == category + + def test_a_valid_choice_label_is_returned(self): + server = _Server(lambda request: httpx.Response(200, json=_response({_FIRST: {"type": "choice", "choice": "keep"}}))) + + answer_set = _ask(server, {_FIRST: _CHOICE}) + + assert answer_set.answers[_FIRST] == ChoiceAnswer(label="keep") + assert answer_set.errors_by_question == {} + + def test_an_answer_for_a_question_that_was_not_asked_is_ignored(self): + server = _Server(lambda request: httpx.Response(200, json=_response({_FIRST: {"type": "noul", "noul": 0.25}, "never_asked": {"type": "noul", "noul": "not a number"}}))) + + answer_set = _ask(server, {_FIRST: _NOUL}) + + assert set(answer_set.answers) == {_FIRST} + assert answer_set.errors_by_question == {} + + @pytest.mark.parametrize("content", [b"not json at all", b'[{"model": "jev-1.13.0"}]', b'{"model": "jev-1.13.0"}', b'{"answers": {"risky_tool_call": {"type": "noul", "noul": 0.1}}}', b'{"model": "", "answers": {}}']) + def test_the_envelope_still_fails_the_whole_request(self, content): + """A missing envelope is request level: the caller cannot use any of it.""" + server = _Server(lambda request, content=content: httpx.Response(200, content=content)) + + with pytest.raises(TypeSafeError) as excinfo: + _ask(server, {_FIRST: _NOUL}) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + + def test_a_missing_answer_is_data_where_a_missing_envelope_raises(self): + """Teeth for the layer split: the same questions, one envelope apart.""" + with_envelope = _Server(lambda request: httpx.Response(200, json=_response({_SECOND: {"type": "noul", "noul": 0.1}}))) + answer_set = _ask(with_envelope, {_FIRST: _NOUL}) + assert answer_set.errors_by_question[_FIRST].category == CATEGORY_MISSING + + without_envelope = _Server(lambda request: httpx.Response(200, content=b'{"model": "jev-1.13.0"}')) + with pytest.raises(TypeSafeError): + _ask(without_envelope, {_FIRST: _NOUL}) + + +# --- the tool gate's mapping of a question-level failure ------------------- + + +class TestGateQuestionMapping: + def test_the_gate_reports_a_question_level_failure_as_invalid_response(self): + server = _Server(lambda request: httpx.Response(200, json=_response({_GATE_QUESTION: {"type": "noul", "noul": "0.9"}}))) + + with pytest.raises(TypeSafeGuardrailError) as excinfo: + _gate(server).evaluate(_tool_call()) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + assert "type=str" in str(excinfo.value) + + def test_another_questions_failure_does_not_change_the_gates_verdict(self): + """The gate asks one question; an unusable answer for anything else is ignored.""" + server = _Server(lambda request: httpx.Response(200, json=_response({_GATE_QUESTION: {"type": "noul", "noul": 0.1}, "other_question": {"type": "noul", "noul": "not a number"}}))) + + decision = _gate(server).evaluate(_tool_call()) + + assert decision.allow is True + assert decision.metadata["probability"] == 0.1 + + +# --- wire size and the limits it must not move ----------------------------- + + +class TestWireSize: + def test_wire_size_matches_what_httpx_sends(self): + value = {"state": {"tool_call": {"name": "bash", "arguments": "{}"}}, "model": "jev-latest", "questions": {_FIRST: {"type": "noul", "instructions": "héllo 中文"}}} + + request = httpx.Request("POST", "https://api.typesafe.ai/v1/systemone", json=value) + + assert wire_size(value) == len(request.content) + + def test_wire_size_counts_utf8_bytes_not_characters(self): + value = {"command": "中" * 100} + compact = json.dumps(value, ensure_ascii=False, separators=(",", ":")) + + assert wire_size(value) == len(compact.encode("utf-8")) + assert wire_size(value) > len(compact), "one CJK character is three UTF-8 bytes" + + def test_the_gates_argument_limit_stays_a_character_count(self): + """Design §2.4: the client counts bytes, and moves no existing limit.""" + arguments = {"command": "中" * 100} + arguments_text = json.dumps(arguments, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + assert wire_size(arguments) > len(arguments_text), "the wire is larger than this character count" + + refused_server = _Server() + refused = _gate(refused_server, max_state_chars=len(arguments_text) - 1).evaluate(GuardrailRequest(tool_name="bash", tool_input=arguments)) + + assert refused.allow is False + assert refused.metadata["length"] == len(arguments_text) + assert refused_server.count == 0 + + sent_server = _Server() + judged = _gate(sent_server, max_state_chars=len(arguments_text)).evaluate(GuardrailRequest(tool_name="bash", tool_input=arguments)) + + assert sent_server.count == 1, "the same text fits under the character limit, so it is evaluated" + assert judged.metadata["probability"] == 0.9 + + +# --- the failure chain never carries the credential ------------------------ + + +class TestTransportFailureChain: + """A rejected header is where a credential used to reach a log. + + h11 builds its ``LocalProtocolError`` message from the request it refused — the + whole ``Bearer `` value — and the guardrail's ``logger.exception`` prints the + cause chain, so the client raises without chaining it. The exception *type* stays + in the message: the failure is still diagnosable, only the value is gone. + """ + + @staticmethod + def _rejecting_transport(request: httpx.Request) -> httpx.Response: + raise httpx.LocalProtocolError(f"Illegal header value b'{request.headers['authorization']}'") + + def _chain_client(self) -> TypeSafeClient: + connection = resolve_connection(settings={"api_key": _API_KEY, "max_attempts": 1}, configuration_source="tests.typesafe") + return TypeSafeClient(connection, transport_factory=lambda: httpx.MockTransport(self._rejecting_transport)) + + def test_the_sync_failure_does_not_print_the_key_into_the_chain(self): + with pytest.raises(TypeSafeError) as excinfo: + self._chain_client().ask({"tool_call": {"name": "bash", "arguments": "{}"}}, {_GATE_QUESTION: _NOUL}) + + assert _API_KEY not in "".join(traceback.format_exception(excinfo.value)) + assert "LocalProtocolError" in str(excinfo.value), "the failure stays diagnosable" + + def test_the_async_failure_does_not_print_the_key_into_the_chain(self): + client = self._chain_client() + + async def ask(): + return await client.aask({"tool_call": {"name": "bash", "arguments": "{}"}}, {_GATE_QUESTION: _NOUL}) + + with pytest.raises(TypeSafeError) as excinfo: + asyncio.run(ask()) + + assert _API_KEY not in "".join(traceback.format_exception(excinfo.value)) + + +# --- the reply is bounded and never decoded -------------------------------- + + +def _client_for(handler, **settings) -> TypeSafeClient: + connection = resolve_connection(settings={"api_key": _API_KEY, "max_attempts": 1, **settings}, configuration_source="tests.typesafe") + return TypeSafeClient(connection, transport_factory=lambda: httpx.MockTransport(handler)) + + +class TestBoundedResponse: + """A 200 is not a reason to read a body into memory. + + The endpoint is treated as potentially hostile, so the two shapes from the review + are refused as request-level failures: a body that only becomes large after + decompression, and one that is large as sent. Both read paths (``ask`` / + ``aask``) go through their own helper, so both are covered here. + """ + + _CALL = {"tool_call": {"name": "bash", "arguments": "{}"}} + _ENVELOPE = json.dumps(_response({_GATE_QUESTION: {"type": "noul", "noul": 0.1}})).encode() + + @staticmethod + def _ask(client: TypeSafeClient, *, as_async: bool): + if as_async: + return asyncio.run(client.aask(TestBoundedResponse._CALL, {_GATE_QUESTION: _NOUL})) + return client.ask(TestBoundedResponse._CALL, {_GATE_QUESTION: _NOUL}) + + @pytest.fixture(autouse=True) + def _bounded_helpers(self): + """Both directions of the cap: a body it must refuse and one it must still read.""" + self.oversized = json.dumps({"model": "jev-1.13.0", "answers": {}, "pad": "x" * MAX_RESPONSE_BYTES}).encode() + self.deliberately_huge = gzip.compress(b" " * (4 << 20)) + assert len(self.deliberately_huge) < MAX_RESPONSE_BYTES, "the wire size is small; only the decoded size is not" + + def test_the_request_asks_for_an_identity_body(self): + server = _Server() + _ask(server, {_GATE_QUESTION: _NOUL}) + + assert server.requests[0].headers["accept-encoding"] == "identity" + + @pytest.mark.parametrize("as_async", [False, True], ids=["sync", "async"]) + def test_a_compressed_body_is_refused_even_when_it_decodes_to_a_valid_envelope(self, as_async): + """Refusing by encoding, not by parse failure: the decoded body here is valid.""" + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, content=gzip.compress(self._ENVELOPE), headers={"Content-Encoding": "gzip"}) + + with pytest.raises(TypeSafeError) as excinfo: + self._ask(_client_for(handler), as_async=as_async) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + assert "gzip" in str(excinfo.value), "the encoding is reported without being decoded" + + @pytest.mark.parametrize("as_async", [False, True], ids=["sync", "async"]) + def test_a_body_over_the_cap_is_refused(self, as_async): + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, content=self.oversized) + + with pytest.raises(TypeSafeError) as excinfo: + self._ask(_client_for(handler), as_async=as_async) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + assert str(MAX_RESPONSE_BYTES) in str(excinfo.value) + + @pytest.mark.parametrize("as_async", [False, True], ids=["sync", "async"]) + def test_a_body_that_only_expands_is_refused_without_expanding(self, as_async): + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, content=self.deliberately_huge, headers={"Content-Encoding": "gzip"}) + + with pytest.raises(TypeSafeError) as excinfo: + self._ask(_client_for(handler), as_async=as_async) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + + def test_a_body_at_the_cap_is_still_answered(self): + envelope = json.loads(self._ENVELOPE) + envelope["pad"] = "" + envelope["pad"] = "x" * (MAX_RESPONSE_BYTES - len(json.dumps(envelope).encode())) + at_cap = json.dumps(envelope).encode() + assert len(at_cap) == MAX_RESPONSE_BYTES + + answer_set = self._ask(_client_for(lambda request: httpx.Response(200, content=at_cap)), as_async=False) + + assert answer_set.noul(_GATE_QUESTION).probability == 0.1 + + @pytest.mark.parametrize("as_async", [False, True], ids=["sync", "async"]) + def test_deep_nesting_is_a_request_level_failure(self, as_async): + """``json.loads`` raises ``RecursionError`` for deep nesting, not ``ValueError``.""" + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, content=b"[" * 20000 + b"]" * 20000) + + with pytest.raises(TypeSafeError) as excinfo: + self._ask(_client_for(handler), as_async=as_async) + + assert excinfo.value.cause == CAUSE_INVALID_RESPONSE + + +# --- the deadline is a budget on the read too ------------------------------ + + +class _DrippingStream(httpx.SyncByteStream): + """A body that arrives in pieces, each one a fraction of the deadline. + + ``deadline_seconds`` is a budget, and the sync path cannot preempt the read it + is already blocked in — so an endpoint that stays just under the read timeout + holds the caller for as long as it keeps dripping. Counting the pieces the + client asked for is what separates "stopped at the deadline" from "read the + whole body and only then noticed the budget was gone". + """ + + def __init__(self, pieces: list[bytes], delay: float) -> None: + self.pieces = pieces + self.delay = delay + self.pulled = 0 + + def __iter__(self) -> Iterator[bytes]: + for piece in self.pieces: + time.sleep(self.delay) + self.pulled += 1 + yield piece + + +class _AsyncDrippingStream(httpx.AsyncByteStream): + """The async twin of :class:`_DrippingStream` (its deadline is a cancellation).""" + + def __init__(self, pieces: list[bytes], delay: float) -> None: + self.pieces = pieces + self.delay = delay + self.pulled = 0 + + async def __aiter__(self) -> AsyncIterator[bytes]: + for piece in self.pieces: + await asyncio.sleep(self.delay) + self.pulled += 1 + yield piece + + +class TestDeadlineBudget: + """The deadline covers the body read, not only the response headers. + + The body is a valid envelope delivered in slow pieces, so the outcome can only + be the deadline: a parse failure would mean the client read past its budget. + """ + + _DEADLINE = 0.2 + _DELAY = 0.05 + _PIECES = 24 + + @pytest.mark.parametrize("as_async", [False, True], ids=["sync", "async"]) + def test_a_dripping_body_stops_at_the_deadline(self, as_async): + envelope = json.dumps(_response({_GATE_QUESTION: {"type": "noul", "noul": 0.1}})).encode() + body = envelope + b" " * (-len(envelope) % 8) + pieces = [body[index : index + 8] for index in range(0, len(body), 8)] + pieces += [b" "] * (self._PIECES - len(pieces)) + assert len(pieces) == self._PIECES + stream = _AsyncDrippingStream(pieces, self._DELAY) if as_async else _DrippingStream(pieces, self._DELAY) + client = _client_for(lambda request: httpx.Response(200, stream=stream), deadline_seconds=self._DEADLINE) + state = {"tool_call": {"name": "bash", "arguments": "{}"}} + + started = time.monotonic() + with pytest.raises(TypeSafeError) as excinfo: + if as_async: + asyncio.run(client.aask(state, {_GATE_QUESTION: _NOUL})) + else: + client.ask(state, {_GATE_QUESTION: _NOUL}) + elapsed = time.monotonic() - started + + assert excinfo.value.cause == CAUSE_DEADLINE + assert stream.pulled < self._PIECES, "the read stopped at the deadline instead of consuming the body" + assert elapsed < self._PIECES * self._DELAY, "the caller was held for the whole body" diff --git a/backend/tests/test_typesafe_config.py b/backend/tests/test_typesafe_config.py new file mode 100644 index 0000000000..3e4ac89c93 --- /dev/null +++ b/backend/tests/test_typesafe_config.py @@ -0,0 +1,318 @@ +"""Tests for the unified ``typesafe:`` configuration and its two identities. + +Design §4: precedence is consumer ``config`` > top-level ``typesafe`` > built-in +defaults; ``mode: off`` resolves nothing at all; and the internal sharing identity +(``sharing_key``) stays separate from each consumer's public policy identity +(``release_policy_parameters``) — behaviour parameters reach only the latter, the +credential fingerprint only the former. +""" + +from __future__ import annotations + +import hashlib +import json + +import httpx +import pytest +import yaml +from pydantic import ValidationError + +from deerflow.config.app_config import AppConfig +from deerflow.config.typesafe_config import TypeSafeConfig, get_typesafe_config, load_typesafe_config_from_dict, reset_typesafe_config +from deerflow.guardrails.provider import GuardrailRequest +from deerflow.guardrails.typesafe import TypeSafeGuardrailProvider +from deerflow.typesafe.client import TypeSafeClient +from deerflow.typesafe.connection import resolve_connection, resolve_connection_for_mode + + +@pytest.fixture(autouse=True) +def _isolated_block(): + """The block is a process singleton: keep one test's block out of the next.""" + reset_typesafe_config() + yield + reset_typesafe_config() + + +class _Server: + """Fake System One endpoint that records every request it receives.""" + + def __init__(self, responder=None) -> None: + self.requests: list[httpx.Request] = [] + self._responder = responder or (lambda request: httpx.Response(200, json={"model": "jev-1.13.0", "answers": {"risky_tool_call": {"type": "noul", "noul": 0.1}}})) + + def handle(self, request: httpx.Request) -> httpx.Response: + self.requests.append(request) + return self._responder(request) + + def transport(self) -> httpx.MockTransport: + return httpx.MockTransport(self.handle) + + def bodies(self) -> list[dict]: + return [json.loads(request.content) for request in self.requests] + + +def _tool_call() -> GuardrailRequest: + return GuardrailRequest(tool_name="bash", tool_input={"command": "ls"}) + + +def _client(**settings) -> TypeSafeClient: + connection = resolve_connection(settings=settings, configuration_source="tests.typesafe") + return TypeSafeClient(connection) + + +# --- precedence ------------------------------------------------------------ + + +class TestPrecedence: + def test_a_field_only_the_block_sets_reaches_the_request(self, monkeypatch): + """The block supplies the model and the credential source; the provider sets neither.""" + load_typesafe_config_from_dict({"api_key_env": "SHARED_TYPESAFE_KEY", "model": "block-model", "base_url": "https://block.invalid"}) + monkeypatch.setenv("SHARED_TYPESAFE_KEY", "block-key") + server = _Server() + + TypeSafeGuardrailProvider(transport_factory=server.transport).evaluate(_tool_call()) + + assert server.bodies()[0]["model"] == "block-model" + assert server.requests[0].headers["authorization"] == "Bearer block-key" + + def test_the_consumers_own_config_wins_over_the_block(self, monkeypatch): + load_typesafe_config_from_dict({"api_key": "block-key", "model": "block-model", "timeout": 30.0}) + server = _Server() + + TypeSafeGuardrailProvider(api_key="gate-key", model="gate-model", transport_factory=server.transport).evaluate(_tool_call()) + + assert server.bodies()[0]["model"] == "gate-model" + assert server.requests[0].headers["authorization"] == "Bearer gate-key" + + def test_each_field_resolves_independently(self): + """A consumer overriding one field leaves the block's other values in place.""" + resolved = resolve_connection( + settings={"api_key": "own-key", "model": "own-model"}, + defaults={"api_key": "block-key", "model": "block-model", "base_url": "https://block.example", "timeout": 9.0, "max_attempts": 4}, + configuration_source="tests.typesafe", + ) + + assert resolved.api_key == "own-key" + assert resolved.model == "own-model" + assert resolved.url == "https://block.example/v1/systemone" + assert (resolved.timeout, resolved.max_attempts) == (9.0, 4) + + def test_the_block_is_optional(self): + """With no block at all the built-in defaults apply, as before the block existed.""" + resolved = resolve_connection(settings={"api_key": "own-key"}, configuration_source="tests.typesafe") + + assert resolved.url == "https://api.typesafe.ai/v1/systemone" + assert resolved.api_key_env == "TYPESAFE_API_KEY" + + def test_app_config_parses_the_block_and_feeds_the_singleton(self, tmp_path, monkeypatch): + """The documented config.yaml shape: a top-level ``typesafe:`` mapping.""" + extensions_path = tmp_path / "extensions_config.json" + extensions_path.write_text(json.dumps({"mcpServers": {}, "skills": {}}), encoding="utf-8") + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "typesafe": {"api_key_env": "SHARED_TYPESAFE_KEY", "model": "block-model", "timeout": 30.0}, + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(extensions_path)) + + block = AppConfig.from_file(str(config_path)).typesafe + # AppConfig loading publishes this block to the process singleton; the + # provider reads it when its own config leaves a field unset. + load_typesafe_config_from_dict(block.model_dump()) + + assert block.connection_defaults() == {"api_key_env": "SHARED_TYPESAFE_KEY", "model": "block-model", "timeout": 30.0} + assert get_typesafe_config().connection_defaults() == block.connection_defaults() + + def test_an_unset_field_is_not_a_configured_field(self): + """``None`` means "not configured", so the next source still applies.""" + resolved = resolve_connection(settings={"api_key": "own-key", "model": None}, defaults={"model": "block-model"}, configuration_source="tests.typesafe") + + assert resolved.model == "block-model" + + def test_a_consumers_api_key_env_is_not_overridden_by_the_blocks_literal_key(self, monkeypatch): + """One credential source per layer: the consumer's own variable decides, and + everything else still comes from the block. + + The block's literal key would otherwise win — and it may not be valid for the + consumer's endpoint at all — while ``connection.api_key_env`` kept naming the + variable the consumer configured. + """ + load_typesafe_config_from_dict({"api_key": "block-literal-key", "model": "block-model"}) + monkeypatch.setenv("GATE_ONLY_KEY", "gate-env-key") + server = _Server() + + TypeSafeGuardrailProvider(api_key_env="GATE_ONLY_KEY", transport_factory=server.transport).evaluate(_tool_call()) + + assert server.requests[0].headers["authorization"] == "Bearer gate-env-key" + assert server.bodies()[0]["model"] == "block-model" + + def test_inside_one_layer_a_literal_key_beats_that_layers_env_name(self, monkeypatch): + """The standalone semantics are kept: a layer that sets both uses its literal key.""" + monkeypatch.setenv("IGNORED_KEY", "env-key") + server = _Server() + + TypeSafeGuardrailProvider(api_key="literal-key", api_key_env="IGNORED_KEY", transport_factory=server.transport).evaluate(_tool_call()) + + assert server.requests[0].headers["authorization"] == "Bearer literal-key" + + +# --- mode: off ------------------------------------------------------------- + + +class TestModeOff: + def test_mode_off_resolves_no_credentials_and_no_class_path(self, monkeypatch): + monkeypatch.delenv("MISSING_TYPESAFE_KEY", raising=False) + consumer_config = {"mode": "off", "use": "not.a.real.module:Cls", "base_url": "https://block.invalid", "api_key_env": "MISSING_TYPESAFE_KEY"} + + resolved = resolve_connection_for_mode(mode="off", settings=consumer_config, defaults=None, configuration_source="memory.prescreen.config") + + assert resolved is None, "an off consumer resolves nothing, constructs nothing and sends nothing" + + def test_mode_off_needs_no_configuration_at_all(self): + assert resolve_connection_for_mode(mode="off", settings=None, defaults=None, configuration_source="memory.prescreen.config") is None + + def test_an_enabled_mode_still_fails_loudly(self, monkeypatch): + """Teeth: ``off`` is what suppressed the failure, not a lenient resolver.""" + monkeypatch.delenv("MISSING_TYPESAFE_KEY", raising=False) + consumer_config = {"mode": "shadow", "api_key_env": "MISSING_TYPESAFE_KEY"} + + with pytest.raises(ValueError) as excinfo: + resolve_connection_for_mode(mode="shadow", settings=consumer_config, defaults=None, configuration_source="memory.prescreen.config") + + assert "MISSING_TYPESAFE_KEY" in str(excinfo.value) + assert "memory.prescreen.config" in str(excinfo.value) + + +# --- the two identities ---------------------------------------------------- + + +class TestIdentities: + def test_the_sharing_key_tracks_every_sharing_dimension(self): + base = {"api_key": "key-a", "model": "jev-latest"} + baseline = _client(**base).sharing_key(max_state_chars=100) + + assert _client(**base).sharing_key(max_state_chars=100) == baseline, "the same settings share" + assert _client(api_key="key-b", model="jev-latest").sharing_key(max_state_chars=100) != baseline, "a different credential is a different request" + assert _client(api_key="key-a", model="jev-other").sharing_key(max_state_chars=100) != baseline, "a different model is a different request" + assert _client(**base).sharing_key(max_state_chars=200) != baseline, "a different input limit is a different request" + + def test_a_different_transport_factory_never_shares(self): + connection = resolve_connection(settings={"api_key": "key-a"}, configuration_source="tests.typesafe") + first = TypeSafeClient(connection, transport_factory=_Server().transport) + second = TypeSafeClient(connection, transport_factory=_Server().transport) + + assert first.sharing_key() != second.sharing_key() + + def test_the_sharing_key_never_carries_the_credential_itself(self): + key = _client(api_key="key-a").sharing_key() + + assert "key-a" not in key + + def test_the_policy_identity_carries_behaviour_but_no_credential(self): + declared = TypeSafeGuardrailProvider(api_key="key-a", threshold=0.25, tools=["bash"]).release_policy_parameters() + serialized = json.dumps(declared) + + assert declared["threshold"] == 0.25 + assert declared["tools"] == ["bash"] + assert "key-a" not in serialized + assert hashlib.sha256(b"key-a").hexdigest()[:16] not in serialized, "the credential fingerprint is internal to sharing_key" + + +# --- values no request could use fail where they are configured ------------ + + +class TestConnectionValuesFailAtConstruction: + """A connection value that no request can use must fail at construction, not per call. + + ``base_url`` used to be checked only for being non-blank and the credential only + for being non-empty, so ``ftp://…``, a base URL with a query string, and a key + mounted from a file with a trailing newline all built fine and then failed on + every request — the last one by putting the whole ``Bearer`` value into h11's + ``LocalProtocolError`` message. + """ + + @pytest.mark.parametrize( + "base_url", + [ + "ftp://api.typesafe.ai", + "api.typesafe.ai", + "https://", + "https://api.typesafe.ai?tenant=1", + "https://api.typesafe.ai#frag", + "https://user:pass@api.typesafe.ai", + "https://api.typesafe.ai\n", + "https://api.typesafe.ai:notaport", + ], + ) + def test_a_base_url_no_request_could_use_is_refused(self, base_url): + with pytest.raises(ValueError, match="base_url"): + resolve_connection(settings={"api_key": "own-key", "base_url": base_url}, configuration_source="tests.typesafe") + + def test_a_usable_base_url_still_resolves_with_its_path_prefix(self): + resolved = resolve_connection(settings={"api_key": "own-key", "base_url": "http://127.0.0.1:8080/prefix"}, configuration_source="tests.typesafe") + + assert resolved.url == "http://127.0.0.1:8080/prefix/v1/systemone" + + @pytest.mark.parametrize("api_key", ["sk-live-abc\n", "sk-live-abc ", " sk-live-abc", "sk-live-abc\t", "sk-live-abc\u00a0"]) + def test_a_credential_that_cannot_be_a_header_value_is_refused_without_echoing_it(self, api_key): + with pytest.raises(ValueError) as excinfo: + resolve_connection(settings={"api_key": api_key}, configuration_source="tests.typesafe") + + assert api_key not in str(excinfo.value), "the value is never echoed" + assert "api_key" in str(excinfo.value), "the message names the setting to fix" + + def test_a_credential_read_from_the_environment_is_checked_the_same_way(self, monkeypatch): + """The realistic shape: a secret mounted from a file ends with a newline.""" + monkeypatch.setenv("MOUNTED_TYPESAFE_KEY", "sk-live-mount\n") + + with pytest.raises(ValueError) as excinfo: + resolve_connection(settings={"api_key_env": "MOUNTED_TYPESAFE_KEY"}, configuration_source="tests.typesafe") + + assert "sk-live-mount" not in str(excinfo.value) + + +class TestStrictBlockValues: + """The block's values are checked like the consumer ``config`` path's. + + In pydantic's lax mode ``max_attempts: true`` silently became ``1`` (``bool`` is + an ``int``) while the same value under ``guardrails.provider.config`` was + rejected, so one retry count meant two different things depending on where it was + written. + """ + + def test_a_boolean_retry_count_is_rejected(self): + with pytest.raises(ValidationError): + TypeSafeConfig.model_validate({"max_attempts": True}) + + def test_a_boolean_timeout_is_rejected(self): + with pytest.raises(ValidationError): + TypeSafeConfig.model_validate({"timeout": True}) + + def test_an_integer_is_still_a_usable_float(self): + block = TypeSafeConfig.model_validate({"timeout": 5, "max_attempts": 2}) + + assert block.connection_defaults() == {"timeout": 5.0, "max_attempts": 2} + + def test_a_config_file_with_a_boolean_retry_count_is_refused(self, tmp_path, monkeypatch): + """The operator sees a config error instead of a silently shortened budget.""" + extensions_path = tmp_path / "extensions_config.json" + extensions_path.write_text(json.dumps({"mcpServers": {}, "skills": {}}), encoding="utf-8") + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "typesafe": {"api_key_env": "SHARED_TYPESAFE_KEY", "max_attempts": True}, + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(extensions_path)) + + with pytest.raises(ValidationError): + AppConfig.from_file(str(config_path)) diff --git a/backend/tests/test_typesafe_guardrail_provider.py b/backend/tests/test_typesafe_guardrail_provider.py index 8a7e3ff055..42bc94d384 100644 --- a/backend/tests/test_typesafe_guardrail_provider.py +++ b/backend/tests/test_typesafe_guardrail_provider.py @@ -9,6 +9,7 @@ instance would be dead on the second call. from __future__ import annotations import asyncio +import gzip import hashlib import json import time @@ -327,6 +328,21 @@ class TestResponseValidation: provider.evaluate(_request()) assert excinfo.value.cause == "invalid_response" + def test_a_compressed_response_denies_instead_of_being_read(self): + """A body that only becomes large after decompression is a failed evaluation. + + The gate's failure direction is deny, so an endpoint that answers with an + encoded body must not be allowed through unexamined (the shared client refuses + to decode it and this provider maps that to ``invalid_response``). + """ + body = gzip.compress(b" " * (1 << 20)) + provider = _provider(_Server(lambda request: httpx.Response(200, content=body, headers={"Content-Encoding": "gzip"}))) + + with pytest.raises(TypeSafeGuardrailError) as excinfo: + provider.evaluate(_request()) + + assert excinfo.value.cause == "invalid_response" + def test_nan_never_reaches_the_threshold_comparison(self): """json.loads parses the NaN literal, so the guard has to be in the parser.""" provider = _provider(_Server(lambda request: httpx.Response(200, content=b'{"model": "jev-1.13.0", "answers": {"risky_tool_call": {"type": "noul", "noul": NaN}}}'))) diff --git a/config.example.yaml b/config.example.yaml index 21e016230e..8c892d4779 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -20,7 +20,7 @@ # ============================================================================ # Bump this number when the config schema changes. # Run `make config-upgrade` to merge new fields into your local config.yaml. -config_version: 49 +config_version: 50 # Optional operator-owned literal prompt extensions (see CONFIGURATION.md): # lead_prompt_overlay: @@ -2209,6 +2209,62 @@ memory: # exposing query-aware search (mem0 does this). Tool calls go through the # active MemoryManager; unsupported fact CRUD returns a JSON error. mode: middleware + # Pre-extraction cost gate (optional, OFF by default). Decides whether a batch + # of new conversation is worth paying for the extraction LLM call, using one + # `noul` question ("does this contain durable, cross-task, user-level + # information?"). + # off (default) - no request, no behavior change. + # shadow - decide and record the verdict, still extract (the safe first step). + # enforce - a low-probability batch skips the call AND advances the watermark, + # so that batch is consumed as "nothing durable here". Enable only after the + # measurement gate that scripts/eval_memory_prescreen.py implements and + # backend/docs/MEMORY_IMPROVEMENTS.md documents + # passes; `skip_threshold` below is provisional and unmeasured. + # The failure direction is always extraction: an error, timeout, unusable + # response, missing verdict, or an over-limit batch extracts as usual. A skip is + # not a safety boundary, gates no execution, and changes no post-extraction + # gate (confidence / scope / durability / authority / dedup / eviction). + # + # EGRESS: the formatted conversation text (user + final assistant turns, which + # may contain file paths or credentials) is sent to the TypeSafe API -- a second + # destination beyond your own extraction model. Existing memory is never sent. + # `pii_redaction_middleware` does not cover this path. + # + # NOTE: a batch is not judged at all while `staleness_review_enabled` (true + # below) or `consolidation_enabled` is on -- a skip would also skip that batch's + # maintenance review -- and never on the emergency (pre-summarization) flush or + # the shutdown drain. With the defaults shipped here, shadow/enforce therefore + # sees no batches until staleness_review_enabled is turned off. + # prescreen: + # mode: "off" + # use: deerflow.agents.memory.prescreen.typesafe:TypeSafeMemoryPrescreen + # config: + # skip_threshold: 0.2 # p below this skips; provisional, unmeasured + # max_state_chars: 6000 # characters of the judged batch; over it extracts + # cache_size: 256 # digest-keyed, successful verdicts only + # cache_ttl_seconds: 300.0 + # # Connection/model/timeouts come from the top-level `typesafe:` block + # # unless set here (api_key_env / api_key / base_url / model / timeout / + # # deadline_seconds / max_attempts / retry_backoff). + # + # Signal classification (optional, OFF by default): two `noul` questions per + # batch ("does this explicitly endorse / reject something the user stated + # earlier?") mapped to the deterministic `reinforcement` / `correction` hints. + # off (default) - no request. + # shadow - decide and record only. + # hints - merge the labels into the extraction hint text and, only while the + # pre-screen is in `enforce`, veto a skip the model endorsed. + # A model verdict never decides extraction, never drives deletion, and never + # satisfies the reinforcement evidence gate (that gate stays deterministic). + # signal_classification: + # mode: "off" + # use: deerflow.agents.memory.signals.typesafe:TypeSafeSignalClassifier + # combine: auto # auto | always | never: request sharing with the pre-screen + # config: + # hint_threshold: 0.5 # provisional, unmeasured + # max_state_chars: 6000 # a mismatch with prescreen => each side keeps its own request + # cache_size: 256 + # cache_ttl_seconds: 300.0 # Backend-private config (a dict), passed verbatim to the backend __init__. # Each backend self-interprets it (DeerMem parses it into DeerMemConfig). # @@ -3006,6 +3062,42 @@ run_ownership: # # max_state_chars / cache_size / cache_ttl_seconds / # # instructions / criteria +# ============================================================================ +# TypeSafe (Jev) Client Defaults (optional) +# ============================================================================ +# One block for every TypeSafe consumer: the guardrail risk gate +# (guardrails.provider.config) and, once enabled, memory pre-screening and +# signal classification. Each consumer overrides what it needs in its own +# `config`; precedence is consumer config > this block > built-in defaults. +# +# Not here, on purpose: failure policy (the gate denies on error, the memory +# paths carry on), cache settings, thresholds and question content. Those are +# the consumer's own; two consumers with different thresholds can still share +# one request, and each publishes its own policy identity. +# +# Credentials: prefer api_key_env. `api_key` here puts the key in this file. +# A consumer's `config` and this block are separate layers: the first layer that +# sets `api_key` or `api_key_env` decides the credential, so a consumer with its +# own `api_key_env` keeps reading that variable. Inside one layer a literal +# `api_key` wins over that layer's `api_key_env`. +# +# These values are checked when a consumer is built, not on the first call: a +# `base_url` that is not http(s) or that carries a query / fragment / embedded +# credentials, a key with surrounding whitespace (a secret mounted from a file), +# and a non-printable key character are all configuration errors. Numbers here are +# validated strictly, so write `max_attempts: 2` — `true` is rejected rather than +# silently read as `1`. +# +# typesafe: +# api_key_env: TYPESAFE_API_KEY # default; the variable holding the key +# # api_key: # alternative to api_key_env +# base_url: https://api.typesafe.ai # default; http(s) only, no query/fragment/credentials +# model: jev-latest # default; pin an exact version for traceability +# timeout: 5.0 # per-request timeout (seconds) +# deadline_seconds: 10.0 # whole-evaluation budget (seconds) +# max_attempts: 2 # includes the first; 429/529 and transport errors only +# retry_backoff: 0.5 # base seconds of exponential backoff + # ============================================================================ # Authorization Configuration # ============================================================================ diff --git a/deploy/helm/deer-flow/README.md b/deploy/helm/deer-flow/README.md index 79db3d2fe5..05381b3685 100644 --- a/deploy/helm/deer-flow/README.md +++ b/deploy/helm/deer-flow/README.md @@ -135,7 +135,7 @@ they resolve from the `secrets` map): ```yaml config: | - config_version: 49 + config_version: 50 models: - name: gpt-4 use: langchain_openai:ChatOpenAI diff --git a/deploy/helm/deer-flow/values.yaml b/deploy/helm/deer-flow/values.yaml index d790fd2c75..2744f020b1 100644 --- a/deploy/helm/deer-flow/values.yaml +++ b/deploy/helm/deer-flow/values.yaml @@ -249,7 +249,7 @@ ingress: # -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never # inline literal secret values here. The default enables provisioner sandbox. config: | - config_version: 49 + config_version: 50 log_level: info recursion_limit: 100 max_recursion_limit: 1000 diff --git a/examples/deerflow-extension-jev-classify/README.md b/examples/deerflow-extension-jev-classify/README.md index bdfa58554e..b0768e300e 100644 --- a/examples/deerflow-extension-jev-classify/README.md +++ b/examples/deerflow-extension-jev-classify/README.md @@ -118,6 +118,15 @@ carries a code such as `http_401`, `timeout`, `network`, `invalid_response`, and closes the client. An HTTP 401, 402, 403 or 429 stops further batches. A plugin bug fails its batch with `internal` and a type-name-only log line, never the whole call. +- **Not a production contract, and not sharing the host's transport.** This example + implements its own HTTP path, including its own `choice`-question shape, size + packing and `deadline_seconds` bound, and it does not retry a failed batch, + unlike the host's shared TypeSafe client (`packages/harness/deerflow/typesafe/`, + used by `deerflow.guardrails.typesafe`), which owns retry/backoff, one error + taxonomy and the two-layer response contract. The extension is a separately + distributed package and must not import host internals, so sharing that transport + needs a publishable client package — option A of the shared Jev/TypeSafe client + design, deferred to a later review. - Item text is data: the default instruction says so, and nothing from the items or responses is copied into error messages. The only backend-supplied text in a result is the served model name, kept to a short identifier. diff --git a/examples/deerflow-extension-jev-context/README.md b/examples/deerflow-extension-jev-context/README.md index bb9708ba37..4aee41a00f 100644 --- a/examples/deerflow-extension-jev-context/README.md +++ b/examples/deerflow-extension-jev-context/README.md @@ -88,6 +88,17 @@ only from the named environment variable, never from plugin settings or messages - Declares effective non-secret options through `release_policy_parameters()` so the host's assembly fingerprint changes when pruning policy changes. Neither the API key nor its environment variable name participates in that identity. +- **Not a production contract, and not sharing the host's transport.** This example + implements its own HTTP path: one request per evaluation, with `timeout_seconds` + as the only bound. It has **no retry** and **no whole-call deadline**, unlike the + host's shared TypeSafe client (`packages/harness/deerflow/typesafe/`, used by + `deerflow.guardrails.typesafe`), which owns retry/backoff, a deadline budget, one + error taxonomy and the two-layer response contract. The extension is a separately + distributed package and must not import host internals, so sharing that transport + needs a publishable client package — option A of the shared Jev/TypeSafe client + design, deferred to a later review. Until then, a hung Jev endpoint is bounded + only by `timeout_seconds`; treat this example as a demonstration rather than a + supported integration. Optional deployment fields (all validated; unknown fields rejected):