mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
d700ad9edf
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client - refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its configuration: one connection-precedence rule, credential fingerprint, wire-size counting and error taxonomy, shared by the guardrail gate and the memory hooks. - feat(memory): wire `memory.prescreen` and `memory.signal_classification` into DeerMem through one injected judge hook (both off by default, `enforce` gated by the shadow evaluation). - config: add the top-level `typesafe:` block and the two judging slots to `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync the Helm README example with it. - test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with per-stratum gates, censoring and the no-network baseline. Review follow-ups folded in: - `mutations_accepted` counts only facts that survive capacity enforcement and the upload-event scrub that precedes persistence, only removals/reinforcements of ids that existed, and persisted near-duplicate merges (matched by id, or by identity for legacy id-less facts); it is published only after persistence succeeds. - The cached manager hot-reloads its judge when a memory slot or the shared `typesafe` block changes (including a credential rotation); publication is serialized and revalidates the config generation, so a build started before an `enforce` -> `off` rollback cannot land after it. - Cache provenance is per answer: the `cached` flag and the reported model follow the answers a side actually consumed, across partial merges and answerless retries. - An oversized noul integer stays a per-question error instead of raising. - Incremental call savings use the shared scored denominator. - The sensitive-stratum gate requires evidence for each of identity, preference and correction. * fix(typesafe): resolve one credential per layer and refuse unusable connection values Review follow-ups on the shared TypeSafe client (#5906): - A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the shared `typesafe:` block: the first layer that sets either credential setting decides it, and inside one layer the literal key wins (the standalone semantics). A consumer whose endpoint does not accept the block's key no longer sends it on every call, and the missing-credential message keeps naming the variable that layer configured. - `base_url` is validated where it is configured: `http(s)` only, with no query, fragment, embedded credentials, whitespace or control characters, and a malformed port fails at construction instead of on every request (httpx raises `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been swallowed into the appended endpoint path). - A credential that cannot travel as a header value — surrounding whitespace from a secret mounted from a file, a non-printable character — is refused when the connection resolves. It used to construct fine and then put the whole `Bearer <key>` value into h11's `LocalProtocolError` message, which the guardrail middleware's `logger.exception` prints. - The client raises transport failures without chaining the original exception, so that message can no longer carry the credential through the cause chain; the exception type stays in the message for diagnosis. - `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config` path: `max_attempts: true` was silently read as `1`. An integer stays a usable float, so `timeout: 5` keeps working. Tests: 20 cases across `tests/test_typesafe_config.py` and `tests/test_typesafe_client.py`, each red on the previous revision and green now. 367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean. Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`, `config.example.yaml`. * fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence Second review round on #5906: the remaining items from the review thread and from the follow-up pass. Typesafe client (the reply is treated as hostile input too): - Send `Accept-Encoding: identity`, refuse a body that carries a non-identity `Content-Encoding`, and read through a bounded `iter_bytes` that stops past `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be decoded in full and, on the gate path, allowed the tool call unexamined; it is now `invalid_response` (a denial with `fail_closed: true`). - Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting escaped the module's taxonomy, so the memory consumers lost the round's record instead of recording a fallback. Memory judge: - A request-level failure is recorded as `request_failed`, never `no_verdict` (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable endpoint was indistinguishable from a provider with no usable answer). The adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record and log; `no_verdict` stays for a question-level failure, and an ineligible side keeps its own reason even when the round's request failed. Shadow evaluation (the enablement gate reads the evidence, not one number): - The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to meet the 1% target; a sample that only meets the point estimate is `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound). - New `baseline_incremental_savings` gate: a baseline that saves as many calls as the model — the shipped all-trivial sample — fails instead of enabling. - Negative counts (`mutations_accepted`, message / batch / token counts) are treated as absent rather than scored: `-3` read as "nothing lost", adding a phantom saving to the numerator and the denominator. Docs: the design documents cited but absent from the repository are replaced with in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`, `GUARDRAILS.md`), and the gate and limit wording now matches what the code enforces. Tests: 31 new or updated cases, each red on the previous revision (compressed body refused even when it decodes to a valid envelope, body over the cap refused, body at the cap answered, deep nesting mapped, `request_failed` on each request-level failure, the bound and baseline gates, the negative counter). 454 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` and `ruff format --check` are clean. * fix(memory-eval): pair the baseline comparison on the same records Review follow-up on the incremental-savings gate added in ed930478 (P2): both sides of the comparison are now the records the no-network baseline can score (`trivial_only` plus a readable signals list). Subtracting that subset's heuristic savings from model savings over *all* scorable skips credited the model with batches the baseline never saw: 300 reviewed skips with a single annotated record computed 300-1=299 "incremental", passed the gate and set `enable_recommended`, while the paired comparison was model=1 against heuristic=1 (incremental 0). - `Evidence` carries the paired numbers (`baseline_paired_skips`, `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one heuristic count, filled by `build_report` from the baseline summary. - The gate reports `INSUFFICIENT` when no scored skip carries the annotation and when the annotation covers only a subset of the scored skips (the message carries the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive paired gain, and `PASS` only on a positive gain over the full paired set. - Docs state the pairing/coverage requirement in `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring. Tests: the reviewer's counterexample as a report-level regression (it failed with `assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true), a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields. 344 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` / `ruff format --check` clean. * fix(typesafe): bound the sync body read by the deadline `deadline_seconds` only covered the sync path's headers, the completed body and the parse, so an endpoint that drips bytes just under the read timeout held the caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget. The read loop now checks the budget before every transport read, which bounds the overshoot to one read. The pieces are the transport's own reads, not re-chunked output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk, so a check at that granularity would never see a drip. `tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24 pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now (~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track the new granularity. * fix(memory): read the YAML off spelling, and veto a skip over the whole batch Two review findings on the memory judging path. `memory.prescreen.mode` / `memory.signal_classification.mode` rejected the `mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1 parses the unquoted value as a boolean, so both fields saw `False`. The failed reload left the previous judge running, so both fields now normalize `False` to `off` before the `Literal` runs. `True` stays a validation error (no mode it could mean). The examples quote the value, and the docs record why. The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch it answers for is the whole post-watermark feed, so an explicit correction in an earlier turn of an eight-message batch did not stop the skip that then advanced the watermark past it. `detect_signals` gains a `window` parameter (default unchanged) and the updater scans the full feed for the veto; the extraction hint and the reinforcement gate keep the window. Both regressions are red on the previous head: the config test reproduced the `input_value=False` rejection through `AppConfig.from_file`, and the batch test reproduced the prescreen judging (and skipping) a batch whose first turn carries the correction.
Jev context pruning — independent plugin example
An opt-in Python plugin that uses Jev to shorten clearly obsolete read-only tool results before DeerFlow's normal summarization. It uses the existing extension API 0.2.2 and full-stack plugin catalog, with no host code changes. Distributed under DeerFlow's MIT license.
Install and configure
From backend/ in a compatible DeerFlow deployment:
uv run deerflow extensions install ../examples/deerflow-extension-jev-context --yes
export TYPESAFE_API_KEY='your-deployment-secret'
Set the registered entry in the deployment configuration:
plugins:
- use: deerflow_extension_jev_context:install
enabled: true
config:
enabled: true
trigger_tokens: 60000
min_calls_between_attempts: 16
Provide the environment variable to the actual Gateway process (including its
container, if applicable). Restart Gateway after installing or changing deployment
configuration; refresh Capability Center to see Jev context pruning. The outer
switch loads the package; config.enabled enables its middleware and contribution.
It is disabled by default. The plugin catalog is read-only; there is no online
key editor, custom browser page, or new slash command. The authenticated status
backend action reports enabled/configured/threshold without exposing credentials.
configured means the key environment variable is nonempty, not that authentication
or connectivity has been verified.
No frontend rebuild is required.
Deployment opt-in sends bounded conversation excerpts, recent user goals,
read-tool arguments and result samples to https://api.typesafe.ai/v1/systemone.
The service uses jev-latest and typed noul decisions. This is a separate API
from the main chat model; a free main model does not make Jev free. Keys are read
only from the named environment variable, never from plugin settings or messages.
Behavior and limits
- Runs on the lead agent through
before_model/abefore_modelatMODEL_LOGICAL. No subagent interception or provider retry multiplier. - Only considers old, successful, plain-text results of
read_file,grep,globandls. Calls/results must match unambiguously with stable message IDs. Protects the first message, newest six messages, skill reads, errors, multimodal outputs, assistant text, users, and all tool-call records. Write/bash/task and other tools are never candidates. Deployments must preserve the read-only meaning of these built-in names. Host-stamped errors, partial results and unknown/malformed structured outcomes are protected even when LangChain's message status still sayssuccess. - A result is shortened only when its keep probability is below 0.2. Preserves 300 characters at each end plus an explicit omission note. Samples cannot prove that the omitted middle is irrelevant: this is lossy, and historical source contents may not be reproducible by reading again.
- Applies a batch only when estimated total context reduction reaches 10%. Sends at most eight candidates in one request, capped at 24,000 UTF-8 bytes including question instructions. Oversized candidates are skipped while later, smaller candidates can still fit in the same request. It does not repeatedly call Jev to process the entire conversation.
- Replaces result contents under their existing IDs in the persisted graph
state. Disabling the plugin does not restore omitted text. It creates no
transcript archive. Existing message metadata and call/result pairing remain.
Replacements carry
jev_context_shortened: truein their additional metadata, but no original-to-replacement hash mapping or detailed audit event is retained. Extension API 0.2.2 exposesCompactionEventand observer registration, but no public event-emission API for this middleware; this plugin does not notify those observers through host internals. The marker identifies a shortened message but cannot reconstruct omitted facts or explain the classifier's decision. - HTTP failures, missing/invalid answers, missing keys or insufficient reduction
leave messages unchanged. The host's usual summarization remains available.
Cancellation propagates. There are no automatic Jev retries or payload logs.
For caught request/response failures, the
deerflow_extension_jev_context.compactionlogger emits a DEBUG message containing only the exception class. Enable that logger to diagnose failures; the plugin never includes exception text, tracebacks, URLs, credentials or conversation data in these diagnostics. - After an attempt (including failure/no change), skips the next 16 logical model calls before trying again (attempts at calls 1, 18, 35, ...). This cooldown is checkpointed per thread, so a shared agent instance does not mix users' state. Already shortened results are not scored again. It does not intercept manual native compaction.
- Declares effective non-secret options through
release_policy_parameters()so the host's assembly fingerprint changes when pruning policy changes. Neither the API key nor its environment variable name participates in that identity. - Not a production contract, and not sharing the host's transport. This example
implements its own HTTP path: one request per evaluation, with
timeout_secondsas the only bound. It has no retry and no whole-call deadline, unlike the host's shared TypeSafe client (packages/harness/deerflow/typesafe/, used bydeerflow.guardrails.typesafe), which owns retry/backoff, a deadline budget, one error taxonomy and the two-layer response contract. The extension is a separately distributed package and must not import host internals, so sharing that transport needs a publishable client package — option A of the shared Jev/TypeSafe client design, deferred to a later review. Until then, a hung Jev endpoint is bounded only bytimeout_seconds; treat this example as a demonstration rather than a supported integration.
Optional deployment fields (all validated; unknown fields rejected):
| Field | Default | Meaning |
|---|---|---|
api_key_env |
TYPESAFE_API_KEY |
Environment variable name, never the key itself |
trigger_tokens |
60000 |
Approximate history tokens; choose for your main model |
preserve_recent_messages |
6 |
Protected suffix, minimum six |
min_result_chars |
4000 |
Minimum eligible result length |
min_calls_between_attempts |
16 |
Number of intervening calls to skip, including after failed attempts |
max_candidates |
8 |
Per-request candidate limit, maximum 16 |
keep_threshold |
0.2 |
Shorten only below this keep probability; maximum 0.5 |
min_reduction_ratio |
0.1 |
Minimum estimated reduction of the whole history |
timeout_seconds |
8.0 |
HTTP connect/read/write/pool timeout per phase |
Prompt caches and cost
Shortening an old result changes the prompt prefix from that point onward and can invalidate the following cached suffix. The stable prefix before the edit can still be reused; subsequent calls can reuse the new prefix. Thresholds and spacing reduce how often history is rewritten, but do not guarantee a net cost or latency saving. Include Jev charges, the cache refill, cache discounts, future turns and any extra rereads when measuring. Approximate tokens removed are not money saved; providers' actual usage/cache counters are the evidence.
Verify
From backend/:
uv run pytest tests/test_jev_context_extension.py
Coverage includes sync/async host isolation, checkpoint persistence, per-thread
cooldown, message pairing, protected content, malformed decisions, failure
fallback to real native summarization, cancellation and the plugin catalog.
The Jev Plugin Package CI workflow separately builds and installs the wheel,
then loads its declared entry point and registers its contributions through the
real host. scripts/verify_package.py --installed-dir PATH runs that check against
an isolated uv pip install --no-deps --target PATH installation; it does not
substitute a source-tree import for the installed distribution.
An opt-in synthetic smoke test calls real Jev and a caller-selected OpenAI-compatible chat endpoint, without using real user transcripts:
export TEST_CHAT_BASE_URL='http://localhost:8000'
export TEST_CHAT_MODEL='your-test-model'
uv run python ../examples/deerflow-extension-jev-context/scripts/verify_live.py
This script deliberately sends no chat Authorization header. Use a test endpoint
that supports unauthenticated access; only Jev receives the Jev bearer token.
It compares the baseline/pruned prompt usage and checks retention of two recent
facts. This small smoke test is not a general relevance or cost benchmark.