mirror of https://github.com/bytedance/deer-flow.git synced 2026-11-09 20:40:36 +00:00

History

feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906 )

* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client

- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
  configuration: one connection-precedence rule, credential fingerprint, wire-size
  counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
  DeerMem through one injected judge hook (both off by default, `enforce` gated by
  the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
  `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
  the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
  per-stratum gates, censoring and the no-network baseline.

Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
  upload-event scrub that precedes persistence, only removals/reinforcements of ids
  that existed, and persisted near-duplicate merges (matched by id, or by identity
  for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
  `typesafe` block changes (including a credential rotation); publication is
  serialized and revalidates the config generation, so a build started before an
  `enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
  the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
  and correction.

* fix(typesafe): resolve one credential per layer and refuse unusable connection values

Review follow-ups on the shared TypeSafe client (#5906):

- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
  shared `typesafe:` block: the first layer that sets either credential setting
  decides it, and inside one layer the literal key wins (the standalone
  semantics). A consumer whose endpoint does not accept the block's key no longer
  sends it on every call, and the missing-credential message keeps naming the
  variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
  fragment, embedded credentials, whitespace or control characters, and a
  malformed port fails at construction instead of on every request (httpx raises
  `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
  swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
  a secret mounted from a file, a non-printable character — is refused when the
  connection resolves. It used to construct fine and then put the whole
  `Bearer <key>` value into h11's `LocalProtocolError` message, which the
  guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
  so that message can no longer carry the credential through the cause chain; the
  exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
  path: `max_attempts: true` was silently read as `1`. An integer stays a usable
  float, so `timeout: 5` keeps working.

Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.

* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence

Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.

Typesafe client (the reply is treated as hostile input too):

- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
  `Content-Encoding`, and read through a bounded `iter_bytes` that stops past
  `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
  decoded in full and, on the gate path, allowed the tool call unexamined; it is
  now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
  escaped the module's taxonomy, so the memory consumers lost the round's record
  instead of recording a fallback.

Memory judge:

- A request-level failure is recorded as `request_failed`, never `no_verdict`
  (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
  endpoint was indistinguishable from a provider with no usable answer). The
  adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
  and log; `no_verdict` stays for a question-level failure, and an ineligible side
  keeps its own reason even when the round's request failed.

Shadow evaluation (the enablement gate reads the evidence, not one number):

- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
  meet the 1% target; a sample that only meets the point estimate is
  `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
  the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
  treated as absent rather than scored: `-3` read as "nothing lost", adding a
  phantom saving to the numerator and the denominator.

Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.

Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.

* fix(memory-eval): pair the baseline comparison on the same records

Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).

Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).

- `Evidence` carries the paired numbers (`baseline_paired_skips`,
  `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
  heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
  when the annotation covers only a subset of the scored skips (the message carries
  the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
  paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
  `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.

Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.

* fix(typesafe): bound the sync body read by the deadline

`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.

`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.

* fix(memory): read the YAML off spelling, and veto a skip over the whole batch

Two review findings on the memory judging path.

`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.

The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.

Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.

2026-09-27 20:28:05 +08:00

deermem

feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906 )

2026-09-27 20:28:05 +08:00

honcho

fix(memory): validate Honcho's allow_insecure_http as a boolean (#5887 )

2026-09-27 11:18:03 +08:00

mem0

fix(memory): report malformed backend_config values by key name (#5555 )

2026-09-20 16:43:54 +08:00

noop

feat(memory): add opt-in relevance-aware retrieval ranking (#5251 )

2026-09-18 11:32:04 +08:00

openviking

fix(memory): report malformed backend_config values by key name (#5555 )

2026-09-20 16:43:54 +08:00

__init__.py

feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122 )

2026-07-15 11:21:04 +08:00

README.md

fix(memory): validate Honcho timeout and character limits (#4783 )

2026-08-17 08:22:11 +08:00

README.md

Memory Backends

Each subfolder under agents/memory/backends/ is a pluggable memory backend. Swap the active one by changing one line in config.yaml - no deer-flow core changes required.

deermem/ - the default backend (deer-flow's own: structured facts + JSON storage).
noop/ - an empty backend and the template to copy when adding a new one.
openviking/ - optional remote backend using the official langchain-openviking package (single-user middleware mode).
honcho/ - optional remote Honcho backend over HTTP: user-model memory (representations built by Honcho's server-side deriver; no local LLM calls). Workspace-per-user isolation with per-user overrides.

This guide tells you which files to touch when you change, swap, or add a memory system. Paths are relative to backend/ unless noted.

Add a New Backend
Switch the Active Backend
Backend Contract
Do Not Modify
Common Pitfalls
Reference

Add a New Backend

Copy noop/ to backends/<yourname>/ and edit three files in this folder plus two outside it.

File	What to change
`backends/<yourname>/config.py`	Declare your config fields + `from_backend_config` (parse `backend_config`; read `storage_path` from it - do not import deer-flow path helpers)
`backends/<yourname>/<yourname>_manager.py`	Rename the class; parse config in `model_post_init`; implement `from_config` + the tier-1 abstracts (`add`/`get_context`); override tier-2/3 methods as needed (see Backend Contract)
`backends/<yourname>/__init__.py`	`MANAGER_CLASS = YourManager` (relative import)
`config.yaml` (repo root, parent of `backend/`)	`memory.manager_class: <yourname>` + your knobs under `memory.backend_config`
`packages/harness/pyproject.toml`	Only if the backend needs external libs: declare the dependency; add `[tool.uv.sources]` for vendored source. Otherwise `uv sync` purges it (see Common Pitfalls)

See the docstring at the top of noop/noop_manager.py for the full 6-step walkthrough.

Switch the Active Backend

Edit config.yaml (repo root) only:

memory:
  manager_class: <name>        # deermem / noop / <yourname>
  backend_config: { ... }      # that backend's private config

Then restart deer-flow - the memory manager is a process-level singleton; a running process does not hot-reload config or backend code.

Backend Contract

1. The three-tier contract

MemoryManager is a pydantic BaseModel (not a bare ABC). Methods are tiered:

Tier 1 (abstract) -- add + get_context: every backend MUST implement (write + read-inject are the backend's fundamental duties; missing one is caught at instantiation).
Tier 2 (management, with defaults) -- add_nowait (delegates to add), search / get_memory / clear_memory / import_memory / export_memory / delete_memory (default raise NotImplementedError), shutdown_flush (default True). Override the ones your backend supports.
Tier 3 (optional hooks, with defaults) -- warm (default True), reload_memory / create_fact / delete_fact / update_fact (default raise), on_pre_compress / on_turn_start (default no-op).

A new backend implements from_config + add + get_context and overrides only what it supports; the rest inherits defaults. Signatures must match (parameter names, keyword-only args). noop is the minimal reference.

2. Return shape (critical, easy to get wrong)

get_memory / export_memory / clear_memory / import_memory return a dict that the gateway casts to the DeerMem shape (MemoryResponse: version / lastUpdated / user / history / facts[]). Your backend must return a dict this shape accepts, or:

the data is silently dropped (pydantic ignores unknown fields);
the frontend gets empty defaults and lastUpdated="" crashes the date formatter.

A non-DeerMem backend maps its native records (e.g. {"results": [...]}) into this shape via a small adapter helper.

3. Tier-3 hooks (contracted, no `hasattr` probing)

create_fact / delete_fact / update_fact / reload_memory / warm are tier-3 hooks ON the base contract (with defaults). Callers (gateway / client / tools) invoke them directly and catch NotImplementedError for unsupported backends -- no more hasattr probing.

create_fact / delete_fact / update_fact - the frontend's add/delete/edit-fact buttons. Default raises (caller returns 501).
reload_memory - the frontend's reload button (caller falls back to get_memory on NotImplementedError).
warm - one-time warm-up at gateway startup (default True = nothing to warm).

Implement the ones your backend supports; the rest inherit the default raise.

4. Portability (the golden rule)

Important

A backend talks to the host through exactly two channels: (1) the ABC method arguments (manager.py), and (2) the backend_config dict. The only from deerflow import allowed anywhere in your backend folder is the ABC contract line in <name>_manager.py:

from deerflow.agents.memory.manager import MemoryManager

Change that one line (and only that line) to port the backend to another agent. Do not import deer-flow path helpers, config singletons, or models - get storage_path and everything else from backend_config.

5. What the host provides

The factory (manager.py::get_memory_manager) resolves the backend class, injects storage_path into backend_config, then calls cls.from_config(backend_config, mode=cfg.mode, **host_hooks). The host hooks (passed as from_config kwargs, NOT in backend_config):

backend_config["storage_path"] (str) - a writable state dir (the host's runtime_home by default, or whatever config.yaml sets). Use this as your storage root.
callbacks (MemoryCallbacks | None) - observability; on_memory_llm_call merges trace metadata before your LLM call (langfuse). Pass it to your LLM path; ignore if you don't trace.
should_keep_hidden_message / trace_context_manager / host_llm_factory - other host hooks; consume in from_config if relevant.
Plus whatever the user puts under config.yaml::memory.backend_config (your backend's own knobs).

Each backend's from_config consumes the hooks it needs (DeerMem does; noop ignores them).

Do Not Modify

These are backend-agnostic. Don't touch them when swapping backends (unless you're changing the shared contract, which affects every backend):

File	Role
`packages/harness/deerflow/agents/memory/manager.py`	ABC + factory + scanner
`packages/harness/deerflow/agents/middlewares/memory_middleware.py`	`after_agent` -> `manager.add`
`packages/harness/deerflow/agents/memory/summarization_hook.py`	summarization -> `manager.add_nowait`
`packages/harness/deerflow/agents/lead_agent/prompt.py`	`_get_memory_context` -> `manager.get_context`
`app/gateway/routers/memory.py`	HTTP endpoints -> `manager.*` (direct call + try/except `NotImplementedError`)
`packages/harness/deerflow/config/memory_config.py`	shared 4 fields (`enabled` / `injection_enabled` / `manager_class` / `backend_config`)
`frontend/src/components/workspace/settings/memory-settings-page.tsx`	frontend memory page (assumes DeerMem shape)

Note

The gateway and frontend are currently hard-coded to the DeerMem shape - that's why backends must return DeerMem-shape data (contract #2). Making them fully backend-agnostic is a larger refactor.

Common Pitfalls

Lessons from integrating external backends:

External deps must be declared in pyproject.toml. A bare uv pip install is purged on the next uv sync / langgraph dev. Declare the dep (and [tool.uv.sources] for vendored source).
Return the DeerMem shape. Otherwise the frontend crashes with Invalid time value and your data is silently dropped. Build a small adapter helper to map your native records into it.
Fact CRUD returns 501 if not implemented. The frontend's delete-fact button reports Operation 'delete fact' not supported. Implement delete_fact (and friends) to fix it.
Don't import runtime_home. Read storage_path from backend_config. (The noop template shows the correct pattern; importing deer-flow path helpers breaks portability - contract #4.)
Restart deer-flow after changes. The manager is a process-level singleton; a running process does not hot-reload config or backend code.
Cap get_context length yourself. The host applies no token budget; the backend must truncate (DeerMem has max_injection_tokens; noop does not).

Honcho Backend

The optional honcho/ backend is a remote-only HTTP adapter for user-model memory over Honcho (self-hosted or via api.honcho.dev). Middleware mode is the default; tool mode is also supported (search is implemented) and retains passive writes via MemoryMiddleware (requires_passive_writes_in_tool_mode = True) so Honcho's server-side deriver keeps building representations from every turn (no local LLM calls).

Configuration (under memory.backend_config):

Key	Type	Default	Description
`base_url`	str	`http://localhost:8000`	Honcho instance URL (e.g., `http://localhost:8000` or `https://api.honcho.dev`)
`api_key`	str	optional	API key for hosted Honcho; required if `base_url` is `api.honcho.dev`. Can use `$HONCHO_API_KEY` env var syntax. Requires `allow_insecure_http: true` when using plain HTTP
`allow_insecure_http`	bool	false	Allow HTTP (non-HTTPS) connections; needed for localhost development with api_key
`timeout_seconds`	float	`10.0`	HTTP client timeout (seconds) for calls to Honcho — read/write/pool; see `connect_timeout_seconds` for the connect phase. Must be finite and `> 0`
`connect_timeout_seconds`	float	`3.0`	HTTP connect timeout (seconds) for establishing the connection to Honcho. Must be finite and `> 0`
`workspace_prefix`	str	`deerflow-u-`	Prefix for isolated workspaces; each user gets one workspace named `{prefix}{sanitized_id}`
`workspace_overrides`	dict	`{}`	Map specific user ids to custom workspace names; overrides the prefix-based derivation. Values must be non-empty (parse error otherwise). Mapping several users to one workspace shares its search index across them (see Workspace Resolution)
`user_peer_overrides`	dict	`{}`	Map specific user ids to custom names for the user's own peer; overrides the stable-id derivation. Values must be non-empty (parse error otherwise)
`assistant_peer`	str	`deerflow`	Default peer name for the assistant when storing messages
`message_char_limit`	int	`8000`	Character limit per message; longer messages are truncated. Must be `> 0` (zero empties the write; a negative value is a Python suffix slice, not a cap)
`max_injection_chars`	int	`6000`	Character limit for injected memory into the system prompt. Must be `> 0`
`failure_policy.read`	str	`fail_open`	Recall failure handling: `fail_open` (log and return empty) or `fail_closed` (rethrow)

Workspace Resolution: Each DeerFlow user maps to one Honcho workspace. The workspace name is derived as: workspace_overrides[user_id] (if present) else workspace_prefix + sanitized_id, where sanitized_id is a collision-resistant hash suffix (sanitize[:48]-sha256[:8]). Missing user fails closed to no memory. The default derivation is isolated per user; a workspace_overrides entry that maps several users to one workspace deliberately shares that workspace's search index across them (search uses Honcho's workspace-scoped /search, which has no peer filter), while get_context / get_memory remain peer-scoped.

Tool Mode: While tool-mode memory tools are not fully supported, the backend implements requires_passive_writes_in_tool_mode = True to retain passive writes via MemoryMiddleware while also enabling memory search through the memory_search tool.

Reference

Template: noop/ - minimal implementation with full docstrings; copy and go.
Contract + factory: packages/harness/deerflow/agents/memory/manager.py (MemoryManager base, MemoryCallbacks, get_memory_manager factory).

README.md

Memory Backends

Table of Contents

Add a New Backend

Switch the Active Backend

Backend Contract

1. The three-tier contract

2. Return shape (critical, easy to get wrong)

3. Tier-3 hooks (contracted, no hasattr probing)

4. Portability (the golden rule)

5. What the host provides

Do Not Modify

Common Pitfalls

Honcho Backend

Reference

3. Tier-3 hooks (contracted, no `hasattr` probing)