18 Commits

Author SHA1 Message Date
ZephyrWang
b80191fad3
fix(memory): reject duplicate fact IDs during import (#8151)
* fix(memory): reject duplicate fact IDs during import

* fix(memory): explain duplicate IDs in import response

* docs(changelog): link duplicate import fix PR

* fix(memory): keep duplicate validation portable

---------

Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 11:42:55 +08:00
Wenchao An
9699f04cdc
feat(projects): add visible project memory with extraction controls (#8117)
* feat(projects): add visible project memory with extraction controls

* fix(projects): avoid replaying committed memory extractions

* chore(ci): retrigger project memory checks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-12 09:09:44 +08:00
Nan Gao
5ae953fc2f
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers

DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.

- New backend-private key memory.backend_config.retrieval_index_path: empty
  keeps {storage_path}/.retrieval, a relative path resolves against
  storage_path, an absolute path is used as-is. The full rebuild and the
  corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
  (mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
  scope whose live signature differs because another process wrote the user's
  memory. An own commit advances the recorded signatures of that user's scopes
  only when they were in sync at the previous revision, so own writes never
  rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
  declared multi-instance deployment keeps the DeerMem index inside
  storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
  the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
  it, drops the legacy memory.storage_path: memory.json line, and documents the
  memory layout next to the other multi-replica prerequisites.

* docs(changelog): link the memory retrieval index entry to #6494

* fix(memory): never advance the index signature on a no-op commit and resync whole scopes

Two review findings on the cross-replica retrieval re-sync:

- save() and apply_changes() inferred the replaced manifest revision as
  "returned revision - 1". The commit helper returns the existing manifest
  unchanged for a no-op (unchanged-value or empty change set), so after a peer
  bumped the manifest an own no-op write advanced this process's stale index
  signature to the peer's revision without carrying the peer's delta: a fact the
  peer deleted stayed searchable until the manifest changed again. Both paths
  now read the real pre-commit revision under the user lock and pass
  committed=(pre_revision, signature) only when the commit produced a new
  revision; a no-op passes None so the next search still rebuilds the scope.

- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
  listed facts through list_facts() with its default limit of 100 and then
  replaced every row of the scope, so a scope with more than 100 facts lost its
  tail on the first re-sync. It now reads the scope's complete fact list.

* fix(memory): fence index signature promotion against a concurrent scope rebuild

Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.

Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.

Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).

* fix(memory): publish a rebuild's signature under the lock that replaced its rows

Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.

Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).

Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.

Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.

* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error

Two suggestion-level review items on the cross-replica retrieval re-sync.

- The full rebuild cached manifest signatures keyed by (manifest path, agent
  name), but _scope_signature reads only the manifest file -- every agent
  bucket of a user shares one memory.json -- so the startup scan stat+read+
  parsed the same file once per bucket that has facts. The cache is now keyed
  by the manifest path alone while synced signatures are still recorded per
  (user, agent) scope: users manifest reads instead of users x agents.

- The per-search freshness compare put a shared-volume manifest read on every
  search_facts call, and _load_memory_file raises MemoryStorageCorruption on
  OSError and malformed JSON, so a transient network-filesystem read error or a
  corrupt memory.json failed a search the Pod-local index could still answer.
  The compare now catches OSError, ValueError and MemoryStorageCorruption: it
  logs a warning naming the scope (traceback at debug), serves the current
  local index for that scope and leaves the recorded synced signature
  untouched so the next search compares again. It deliberately does not mark
  the scope dirty: a scoped rebuild would hit the same failing read, forget the
  scope's signature and fail the search path too.

* docs(changelog): resolve merge markers left in the Unreleased section

* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible

Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.

Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.

Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.

Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-08 21:20:45 +08:00
PeaceMaker-best
b73a4e79bf
feat(memory): scope management API by agent (#5565)
* feat(memory): scope management API by agent

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

* fix(memory): preserve scoped management isolation

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>

---------

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-09-30 16:50:07 +08:00
Zeren Wang
d700ad9edf
feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906)
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client

- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
  configuration: one connection-precedence rule, credential fingerprint, wire-size
  counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
  DeerMem through one injected judge hook (both off by default, `enforce` gated by
  the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
  `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
  the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
  per-stratum gates, censoring and the no-network baseline.

Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
  upload-event scrub that precedes persistence, only removals/reinforcements of ids
  that existed, and persisted near-duplicate merges (matched by id, or by identity
  for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
  `typesafe` block changes (including a credential rotation); publication is
  serialized and revalidates the config generation, so a build started before an
  `enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
  the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
  and correction.

* fix(typesafe): resolve one credential per layer and refuse unusable connection values

Review follow-ups on the shared TypeSafe client (#5906):

- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
  shared `typesafe:` block: the first layer that sets either credential setting
  decides it, and inside one layer the literal key wins (the standalone
  semantics). A consumer whose endpoint does not accept the block's key no longer
  sends it on every call, and the missing-credential message keeps naming the
  variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
  fragment, embedded credentials, whitespace or control characters, and a
  malformed port fails at construction instead of on every request (httpx raises
  `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
  swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
  a secret mounted from a file, a non-printable character — is refused when the
  connection resolves. It used to construct fine and then put the whole
  `Bearer <key>` value into h11's `LocalProtocolError` message, which the
  guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
  so that message can no longer carry the credential through the cause chain; the
  exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
  path: `max_attempts: true` was silently read as `1`. An integer stays a usable
  float, so `timeout: 5` keeps working.

Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.

* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence

Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.

Typesafe client (the reply is treated as hostile input too):

- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
  `Content-Encoding`, and read through a bounded `iter_bytes` that stops past
  `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
  decoded in full and, on the gate path, allowed the tool call unexamined; it is
  now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
  escaped the module's taxonomy, so the memory consumers lost the round's record
  instead of recording a fallback.

Memory judge:

- A request-level failure is recorded as `request_failed`, never `no_verdict`
  (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
  endpoint was indistinguishable from a provider with no usable answer). The
  adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
  and log; `no_verdict` stays for a question-level failure, and an ineligible side
  keeps its own reason even when the round's request failed.

Shadow evaluation (the enablement gate reads the evidence, not one number):

- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
  meet the 1% target; a sample that only meets the point estimate is
  `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
  the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
  treated as absent rather than scored: `-3` read as "nothing lost", adding a
  phantom saving to the numerator and the denominator.

Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.

Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.

* fix(memory-eval): pair the baseline comparison on the same records

Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).

Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).

- `Evidence` carries the paired numbers (`baseline_paired_skips`,
  `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
  heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
  when the annotation covers only a subset of the scored skips (the message carries
  the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
  paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
  `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.

Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.

* fix(typesafe): bound the sync body read by the deadline

`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.

`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.

* fix(memory): read the YAML off spelling, and veto a skip over the whole batch

Two review findings on the memory judging path.

`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.

The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.

Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
2026-09-27 20:28:05 +08:00
wd_pan
cc27730348
feat(memory): add opt-in relevance-aware retrieval ranking (#5251)
* feat(memory): add opt-in relevance-aware retrieval ranking

Add a deterministic, network-free lexical relevance strategy for DeerMem
(issue #4495): memory_search ranks every fact in scope by idf-weighted
token overlap combined with confidence, with optional greedy-MMR diversity
against near-duplicate facts; prompt injection ranks facts against the
current-turn query threaded from DynamicContextMiddleware through the new
optional `query` keyword on MemoryManager.get_context/aget_context.

Defaults preserve the legacy confidence-only behavior exactly; no prompt,
storage-format, or vector/embedding-dependency changes.

Refs #4495
Signed-off-by: pwd11 <fvdsrc@163.com>

* fix(memory): bound relevance retrieval and apply review feedback

Bound tokenization and index shared stems, preserve mixed CJK tokens, warm jieba, and align missing confidence with legacy injection. Cache MMR token sets and stop selection at result or injection budgets. Document retrieval-adapter precedence and add regression coverage. Refs #4495.

Signed-off-by: pwd11 <fvdsrc@163.com>

* fix(memory): preserve backend compatibility and normalize relevance

Signed-off-by: pwd11 <fvdsrc@163.com>

* fix(memory): omit absent query hints and share injection IDF

Signed-off-by: pwd11 <fvdsrc@163.com>

* test(memory): retain timeout mock until injection worker exits

Signed-off-by: pwd11 <fvdsrc@163.com>

* docs(agents): drop root guidance compaction

Signed-off-by: pwd11 <fvdsrc@163.com>

* fix(memory): validate token prefixes and preserve upload queries

---------

Signed-off-by: pwd11 <fvdsrc@163.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-18 11:32:04 +08:00
Hao Zhe
ec274bdedb
fix(memory): enforce backend read failure policy (#4726)
* fix(memory): enforce backend read failure policy

* fix(memory): harden failure policy handling

* fix(memory): narrow strict read handling

* fix(memory): keep timeout handling off saturated executor

* fix(memory): preserve legacy fail-closed timeouts
2026-09-06 09:01:33 +08:00
SPEC
822c7bca4b
fix(memory): cancel buffered extraction when agent is deleted or cleared (#5123)
* fix(memory): cancel buffered extraction when agent is deleted or cleared

* fix(memory): cancel buffered work before agent delete

Address review: cancel before/after delete to close the rmtree race,
scope user_id=None cancels to the legacy root only, and import memory
helpers at module scope.

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

* fix(memory): close remaining cancel races from review

Post-clear cancel, legacy-only all_agents scope, always cancel even when
memory is disabled, and fold cancel+delete into one offloaded thread.

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

* docs(memory): align cancel_by_agent None-scope with legacy root

Document that user_id=None cancels only the legacy no-user bucket, matching
clear/storage semantics, not the whole process-local queue.

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

* test(memory): fix cancel_by_agent docstring regression assertion

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

* fix(memory): address final cancel review nits

Type the delete helper with AgentStore, replace docstring pinning with a
kwargs mapping test, and document scoped cancel + residual window in AGENTS.md.

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

* fix(memory): resolve agent store inside delete worker thread

get_agent_store() does blocking config/FS work; keep it off the event
loop so test_delete_agent_does_not_block_event_loop and backend-blocking-io CI pass.

Signed-off-by: SPEC <zt1y17@soton.ac.uk>

---------

Signed-off-by: SPEC <zt1y17@soton.ac.uk>
2026-09-03 08:00:25 +08:00
Hyeonsang Cho
9146bfa03d
feature(gateway): issue request trace ids unconditionally (#5119)
* refactor(gateway): issue request trace ids unconditionally

The request trace id was gated behind logging.enhance.enabled at every
entry point, so downstream code had to keep asking whether one existed:
a header-provenance flag in its own ContextVar, a precedence resolver,
and three-level carrier fallbacks at each consumer.

Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP;
ensure_trace_context covers the entry points that never touch ASGI --
scheduled occurrences, MCP task notification runs, IM channel messages,
and the embedded client -- each scoped to one unit of work so a
long-lived worker task cannot leak one occurrence's id into the next.
The ContextVar becomes the only source; the response header, runtime
context, run metadata and log records are derived outputs.

Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and
drop their presence guards. Removed: resolve_deerflow_trace_id, the
header-provenance flag and its three helpers, set/reset_current_trace_id,
is_trace_correlation_enabled and its gateway alias.

BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and
it cannot be turned off; logging.enhance.enabled controls log output
only. Installations on the default enabled: false will start seeing the
header. No config keys were added or removed.

* fix(gateway): stop persisting a caller-supplied trace id on the run record

body.metadata forks two ways: through build_run_config into the live run
config, which the run worker restamps, and through create_or_reject into
the run record that the runs API echoes verbatim. Only the first was
covered, so a client sending metadata.deerflow_trace_id made the most
durable and most visible surface of a run disagree with the X-Trace-Id
and the log lines the same request produced -- a correlation id that
does not match the logs is worse than none.

Stamp the server-issued id once at the trust boundary so both forks
receive it, preserving the caller's own metadata keys. Close the same
gap on config.context, which reaches the runtime context by a separate
path: _build_runtime_context no longer merges server-owned keys from the
caller, and _install_runtime_context assigns rather than setdefaults.

A thread's metadata is no longer seeded with the run-scoped id of
whichever run created it -- one thread spans many runs and as many
trace ids.

Found by driving a real run through the Gateway and reading the run back
from the runs API; every unit test built its metadata by hand and so
could not see it.

* fix(gateway): expose X-Trace-Id to split-origin browser clients

X-Trace-Id is not on the CORS safelist, so a browser client served from
a separate origin could not read it -- and those are exactly the clients
that cannot read the Gateway's logs either, leaving them with nothing to
quote in a bug report. Same-origin nginx deployments were unaffected,
which is why this stayed hidden.

Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing
TRACE_ID_HEADER rather than repeating the literal.

* fix(gateway): keep X-Trace-Id on unhandled-exception 500s

Starlette's ServerErrorMiddleware sits outside every user middleware and
emits unhandled-exception 500s through the raw send, so those responses
never pass TraceMiddleware's header-writing wrapper. The 500 for a server
bug is exactly the response a user most needs to correlate with a log line,
and it was the one response that shipped without the id.

TraceMiddleware now tracks whether http.response.start has been sent. On an
exception with no response started it emits its own plain 500 carrying the
header, then re-raises: the outer ServerErrorMiddleware sees the response
already started and only re-raises too, so the server's exception logging is
untouched. An exception mid-stream keeps propagating unchanged — a second
response start cannot be sent, and the already-written header stands.

The trace id is printable ASCII by construction (normalize_trace_id /
generate_trace_id), which is what makes the raw latin-1 header encoding
safe.

* fix(gateway): strip the forged trace id from the persisted request echo

The run-record fix stopped a forged metadata.deerflow_trace_id on the
authoritative metadata surface, but the raw request echo still carried one:
create_or_reject persists body.config verbatim as runs.kwargs_json, which
the runs API serves back. A client posting config.context.deerflow_trace_id
therefore still got its forged value stored and echoed on one API surface
while the header, logs, run metadata, and checkpoint all carried the real
id — the id is ignored as input there, so echoing it back only manufactures
disagreement.

Two changes close it. redact_config_secrets — already the shared scrub for
that echo, applied at admission and again at serve time, so historical
records are covered too — now also drops deerflow_trace_id from
config.metadata and config.context. And build_run_config now merges run
metadata onto a copy of the caller's config["metadata"] instead of updating
it in place: the nested values of the request config are reference copies,
so the in-place merge was writing the server-stamped key through into
body.config, contaminating the "what the client sent" record before it was
persisted (and incidentally masking the forged-value echo on the metadata
container).

The regression test posts a forged id through body.metadata,
config.metadata, and config.context at once and reads the kwargs echo back
off the run record, failing if either leak returns.

* docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence

The trace section of the harness AGENTS.md now covers the two fixes that
close the derived-output rule (the kwargs-echo scrub in
redact_config_secrets plus build_run_config's copy merge, and
TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains
their Fixed entries.

It also writes down the one accepted divergence: a crash-recovered
scheduled launch reuses the durable run through its idempotency key, and
start_run returns early on idempotency_reused without restamping — so the
run record keeps the first attempt's deerflow_trace_id while the retry's
own log lines carry the freshly minted id of its ensure_trace_context
binding. The divergence is confined to the crash-recovery window and is
accepted rather than fixed: restamping on reuse would rewrite a persisted
record for a run that already exists, which is worse than two ids that each
correlate their own attempt's logs. Written down so the next reader of the
scheduler recovery path does not diagnose it as a bug.

* docs(config): align the logging.enhance schema note with the unconditional trace id

The config-module AGENTS.md still described logging.enhance as the gate for
the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is
gone: ids are issued unconditionally and this block decides log output only.
Left as-is, the stale wording invites an agent to "restore" a header gate it
believes was lost. Reworded to match the sibling AGENTS.md files and
config.example.yaml, with a pointer to the Request Trace Context section
that owns the full model.

* docs(changelog): link the trace entries to #5119

The five new entries pointed at the ([#XXXX]) placeholder with no reference
definition, rendering as literal text instead of a link — and RELEASING.md
step 2 relies on those references when the section becomes release notes.
All five now point at #5119, with the definition appended to the reference
block.

* refactor(harness): rename _stream_without_trace_context to _stream_turn

The name asserted the opposite of what the method now does. It was accurate
while logging.enhance.enabled could route stream() around the trace scope;
with the gate gone it is the only stream implementation left, and it binds
the id itself via ensure_trace_id(). Private, so the rename touches only the
definition and the one stream() call site.

* docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget

The expanded Request Trace Context section pushed the effective AGENTS.md
chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit
scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the
section from 7,359 to 4592 bytes with no facts removed: the entry-point
table, the derived-output rule and its enforcement points, the accepted
scheduled-retry divergence, the two resolution helpers, the stream()
binding rationale, the log-output-only gate, the CORS listing, the 500
fallback, and the test map all remain.

Sized against the merge, not just the branch: current main grew the same
chain by ~724 bytes, so the check was verified on the merged tree as well
(97,772 bytes; branch tree 97,048).

* fix(gateway): declare content-length on the fallback 500

The pre-response 500 declared content-type but no content-length, leaving
the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on
HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response
it replaces, which sends content-length: 21. The explicit header keeps the
fallback byte-identical to what clients saw before.

* docs(readme): drop the trace-correlation condition from the translations

The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id
matches X-Trace-Id "when request trace correlation is enabled". The id now
always matches and that condition no longer exists, so each bullet states
the unconditional match and that logging.enhance.enabled only controls
whether the id is printed into logs — the one piece of the feature a user
can still configure.

* test(gateway): pin TraceMiddleware wiring through create_app()

Every X-Trace-Id test exercised a hand-built four-route app, so the real
stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting
it — or short-circuiting above it — passed CI while silently dropping both
the response header and the ambient id the run-record stamp and enhanced log
records derive from. One case now drives /health through create_app() and
asserts the inbound id round-trips; mutation-checked by removing the wiring
line, which fails exactly this test.

* docs(gateway): note the fallback 500 is CORS-opaque

The pre-response 500 is emitted outside CORSMiddleware — the exception has
already unwound past it — so it carries no Access-Control-Allow-Origin and
a split-origin browser client cannot read the id on this one response,
unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the
class and in the CHANGELOG entry rather than fixed: replicating the origin
allowlist outside CORSMiddleware would let the two policies drift.

* fix(harness): keep abandoned-stream cleanup inside the trace binding

stream() binds the turn's id around each next(inner) and resets it before
yielding, but the finally's inner.close() ran after that binding was gone.
Abandoning the stream therefore drove the inner LangGraph generator's
GeneratorExit/finally path with no trace id — or an unrelated ambient one
from whichever context ran the close — so cancellation and finalization
logs and callbacks did not correlate with the turn they belong to.

inner.close() is now wrapped in a local bind/reset of the same turn id. The
token is set and reset in the same frame, never across a yield, so the
per-step cross-context safety is preserved even when GC closes the
generator from another Context — pinned by the existing copy_context close
test, which now exercises this path. The regression test records the id
from the inner generator's finally and fails without the binding.

* test(harness): teach the worker-trace fake about RunManager.cleanup

Upstream #5112 (bound gateway memory after terminal runs) added a
run_manager.cleanup(run_id) call to run_agent's finalization, so the
merge-commit CI run failed all five worker-trace-binding tests with
AttributeError on this PR's _FakeRunManager. The fake gains the same no-op
shape as its other methods.

* docs(gateway): bring the gateway AGENTS.md back under its soft budget

Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over
the 40,960 soft budget that
test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes
enforces — its Unit Tests run on main was cancelled by push concurrency, so
main is currently red on that test and every PR merge-run inherits the
failure. Two whitespace/wording trims in the row #5092 touched (a doubled
space, and "its configured `context_window`" → "its `context_window`")
bring the file to 40,953 with no content change.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-01 16:49:39 +08:00
Nan Gao
7389331e65
feat(extensions): observe task lifecycle and system model calls (#4684)
* feat(extensions): observe task lifecycle and system model calls

PR 1 (#4636) gave extensions a middleware chain, and a middleware only sees
what passes through the agent graph. Two runtime surfaces stay invisible to
it: when a lead run or a subagent begins and ends, and the DeerFlow-owned
model calls made outside the graph. This slice adds both, with no new
Gateway surface -- routers, services, and the reference extension stay in
PR 3.

Contract (deerflow-extension-api 0.1.1)
---------------------------------------
Two contribution kinds join `middlewares` on the registry:
`task_lifecycle` (`on_task_start` / `on_task_stop`, receiving a `TaskInfo`
and a conservative `TaskOutcome` of completed / aborted / failed) and
`system_model_observer` (`on_system_model_call`, receiving a
`SystemOperationKind`, a `SystemModelRequest` snapshot, and a
`SystemModelResult` carrying either the response or the provider exception
plus a duration).

`SystemModelRequest.messages` normalizes to a tuple at construction. Goal
evaluation and memory extraction pass a message list while title generation
and summarization pass one prompt string, and a bare `str` already satisfies
`Sequence` -- without normalization an observer iterating `request.messages`
would silently walk characters. Copying also makes the frozen snapshot
immutable in fact rather than only by declaration, since observations may run
after the call site returns and keeps mutating its own list.

Registry marks and rollbacks become per-bucket and positional, so an
`install()` that fails after registering two different kinds cannot leave one
of them behind. `needs_task_store` now covers all three kinds: a deployment
that registers only lifecycle hooks still gets a task store.

Task lifecycle
--------------
The lead worker notifies start after the run has started and stop after
completion persistence and the completion hook, but before clearing the
finalizing barrier and publishing the stream end -- holding the barrier
across stop is what keeps a same-thread replacement run from overlapping this
task's lifecycle. Cancellation raised out of the stop notification is
deferred, not propagated in place, so a cancelled run still clears the
barrier and emits its end frame. A subagent with a parent `run_id` wraps its
execution in the same pair inside `finally`, reporting `parent_task_id` so a
delegation tree is reconstructable; a subagent without a `run_id` (embedded
client, standalone LangGraph Server) logs and skips rather than inventing a
parent. Contributors run in registration order inside one shared 3s budget
and every failure is logged and failed open.

System model calls
------------------
Four kinds cover the model calls the middleware chain cannot see: goal
evaluation, memory extraction, title generation, and summarization. Each site
reports both terminal paths without changing the provider exception the host
observes, short-circuits on `has_system_model_observers`, and passes the live
task store when the runtime has one (detached work gets an isolated store).
The sync summarization half stays unobserved on purpose -- it and its only
host caller are the sync side of an async-only runtime, so notifying there
would block a thread on a call site the host never reaches; the reason is
recorded at the call site.

The DeerMem backend must stay vendorable and cannot import the extension API,
so it reports through a new `MemoryCallbacks.on_memory_llm_result` host hook
that the DeerFlow-side callbacks translate into an observation.

Notification loop
-----------------
Extension resources must be touched on the loop that created them, but
subagents can execute on isolated loops and DeerMem runs on a worker thread.
The Gateway registers its serving loop before any runtime dependency starts
and resets it last through the exit stack, so every startup-failure and
cancellation path is covered. Awaited hooks raised on another loop are
dispatched across with `run_coroutine_threadsafe` and awaited under the same
budget; synchronous sites submit fire-and-forget work. Shutdown stops
accepting detached observations before the memory flush -- that flush runs on
a worker thread and can emit memory observations -- while keeping the loop
alive for awaited task hooks until run and subagent drain completes.

Tests
-----
`test_extension_task_lifecycle.py`, `test_extension_subagent_lifecycle.py`,
and `test_extension_system_model_calls.py` cover ordering, fail-open, budget
exhaustion, snapshot binding under a concurrent singleton replacement, the
loop-dispatch and shutdown-suspension paths, and both terminal paths at every
call site. `test_gateway_run_drain_shutdown.py` pins the stop-before-barrier
and drain ordering.

* fix(extensions): decide notification fail-open by origin, observe cancellation

`_notify_each` only guarded `Exception`, so a contributor letting a
`CancelledError` escape — an extension implementing an internal timeout with
cancellation, say — skipped its successors and reached the worker's
deferred-interrupt path, ending an otherwise successful run as cancelled.
Fail-open is about where a failure came from, not its base class: only a
genuine cancellation of the host task increments `Task.cancelling()`, so
propagate on that and contain everything else. `KeyboardInterrupt` /
`SystemExit` still propagate.

`observe_system_model_call` skipped observers on cancellation for the same
base-class reason, leaving goal / title / summarization silent on a terminal
path that is routine — interrupt/rollback admission and shutdown both cancel
the run task, with the provider tokens already spent. Awaiting observers there
is unreliable (a repeated cancel interrupts that await before any of them
runs), so report through the same non-blocking submission the synchronous
memory bridge uses, then propagate the cancellation untouched.

DeerMem keeps `BaseException` around its provider call, now with the reason
recorded: that path runs on a worker thread, where cancelling the awaiting
side never interrupts the running thread, so `CancelledError` cannot arrive
at all. Its host-hook wrapper narrows to `Exception` — only the hook's own
failures are non-fatal, and an observability path must not swallow a process
teardown signal.

* fix(extensions): warn on budget exhaustion, scope observer logs by task, propagate teardown

Review response on #4684:

- The memory observation bridge caught BaseException, which would swallow
  a teardown signal raised while dispatching; it now catches Exception,
  matching the boundary the DeerMem-side call site documents and tests.
- A notification-budget timeout raised mid-hook fell into the generic
  hook-failure path and logged an asyncio-internal traceback; it now logs
  a warning like the pre-hook budget skip, while a TimeoutError a
  contributor raises on its own stays classified as a hook failure.
- System model observer logs passed the operation kind as the task id,
  so log lines said "task goal/title/..."; they now carry the task
  scope id alongside the kind.
2026-08-11 16:33:22 +08:00
Aari
cccda35cc5
fix(memory): prevent task-scoped data from entering long-term memory (#4604)
* fix(memory): gate long-term updates by scope

* docs(memory): note custom prompts_dir migration; fix stale accept-filter comment

* fix(memory): harden scope-gate review paths
2026-08-01 08:39:28 +08:00
Vanzeren
352f247a81
feat(memory): add mem0 HTTP memory backend (#4528)
* feat(memory): add mem0 HTTP memory backend

* fix(memory): address mem0 review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-29 07:11:20 +08:00
qin-chenghan
795af20a6b
feat(memory): built-in FTS5/BM25 retrieval adapter (#4360)
* feat(memory): integrate FTS5 retrieval adapter

* deps: add jieba as default dependency for Chinese tokenization

Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.

* fix(memory): avoid treating hyphens as FTS5 operators

* feat(memory): make Chinese tokenization optional

* fix(memory): warm every requested retrieval scope

* fix(memory): close retrieval resources on shutdown

* fix(memory): close backend when shutdown flush fails

* fix(memory): recreate corrupt retrieval index

* fix(memory): tolerate partial retrieval rebuilds

* fix(memory): warm retrieval index in background

* fix(memory): preserve shutdown flush budget

* fix(memory): stop retrying partial lazy rebuilds

* fix(memory): close retrieval through storage

* refactor(memory): simplify retrieval scope limit

* docs(memory): clarify retrieval shutdown lifecycle

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-27 23:17:18 +08:00
lllyfff
8145d66a33
feat(memory): memory message processing (#4447)
* feat(memory): signals-based update pipeline + always-on watermark/trivial filter

Refactor the DeerMem memory update pipeline (message_processing -> queue ->
updater) around a signals frozenset seam, replacing the
(filtered, correction_detected, reinforcement_detected) 3-tuple with
(filtered, signals: frozenset[str]) end to end.

message_processing:
- Externalize signal-detection patterns to YAML (message_patterns/*.yaml).
- Extend signals from correction/reinforcement to a 6-class set
  (correction/reinforcement/preference/identity/goal/decision); detect_signals
  returns a frozenset aligned with the fact category enum.
- Pure-acknowledgment turns ("ok"/"好的"/...) are always filtered out before
  enqueue (whole-message fullmatch), saving an extraction LLM call.

queue (core/queue.py):
- In-memory list + debounce timer, with flush_sync (graceful-shutdown drain
  that joins an in-flight worker under a hard timeout) and queue_max_depth
  backpressure (signal-bearing updates always admitted; QueueFull otherwise).
- Same-key updates coalesce with a signal union; per-batch success/fail summary.

updater (core/updater.py):
- head500+tail500 message truncation (replaces the 1000-char head chop).
- Always-on per-thread watermark: feed only messages added since the last
  extraction. The watermark is in-memory and is not advanced on failure, so a
  failed/lost update is re-fed on the next conversation turn.
- [MANUAL] prompt marker for user-authored facts (source.type="manual").
- Post-invoke extraction_callback (host-injected) emitting facts_extracted /
  facts_accepted / rejected_low_confidence; the host default logs metrics and
  flags >60% rejection.

Confidence filtering remains in _apply_updates (the existing
fact_confidence_threshold check); there is no separate write gate.
Consolidation stays opt-in (lossy). The ABC add/add_nowait signature is
unchanged, so the summarization flush hook and host are unaffected.

Tests: add test_message_processing_signals, test_updater_truncation,
test_updater_watermark; update queue/updater/consolidation/staleness/pluggable
tests for the signals seam.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): harden update pipeline per PR review

- Catch QueueFull in DeerMem.add/add_nowait so backpressure degrades to
  'update skipped' instead of propagating into after_agent /
  summarization_hook and breaking the agent run (peer middlewares
  self-guard; MemoryMiddleware was the lone exception). Emergency
  (add_nowait) always admits under backpressure -- its data cannot be
  re-fed next turn.
- Rewrite the watermark from index-based to content/identity-based
  (_message_identity + _feed_after_watermark) so it stays correct when
  summarization removes the conversation front -- an index watermark
  pointed at the wrong message and silently skipped un-extracted tail
  turns. The emergency flush bypasses the watermark (bypass_watermark on
  ConversationContext, threaded through update_memory) and coexists with
  (does not replace) a pending normal update, so a flush cannot drop a
  pending update's un-extracted tail.
- Populate facts_accepted / rejected_low_confidence inside _apply_updates
  at the real confidence-filter site (passed_threshold) instead of
  re-deriving the threshold in _finalize_update -- eliminates metric drift.
- Emit extraction metrics in a finally with an 'attempted' flag so
  exception failures (parse error, apply_changes raise after retry) are
  observable, not only the happy path.
- Re-detect signals on the post-watermark feed for the extraction hint so
  it no longer references turns the LLM cannot see; admission-time signals
  still drive backpressure.
- Move the post-batch reschedule inside the queue lock to close a
  non-atomic self._timer race with a concurrent add.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address follow-up review nits (LRU, metric name, docstring)

- Bound the in-memory watermark cache with a configurable LRU
  (watermark_max_keys, default 4096, 0=unbounded). A dropped key re-extracts
  one batch on that thread's next turn (the documented restart behavior), so
  eviction is safe and preserves the content-identity watermark's
  front-removal guarantee. Adds _watermark_get/_watermark_set helpers and a
  bounded-LRU regression test.
- Rename the extraction metric facts_accepted -> facts_passed_confidence so
  the name matches what the >60% rejection-rate warning assumes (a
  confidence-gate signal, not a persisted-fact count); drop the stale
  "historical semantics" justification. Brand-new callback, one consumer.
- Fix the stale test_message_processing_signals module docstring: the signals
  seam is already swapped to frozenset, and a stale stage-numbering prefix is
  removed.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-26 21:16:36 +08:00
lllyfff
01a89f2379
[feat] memory: pluggable MemoryManager interface for backend onboarding (#4326)
* refactor(memory): pluggable MemoryManager interface for backend onboarding

Optimize the MemoryManager interface layer so new backends (mem0/openviking)
onboard with less code and the contract stays stable as capabilities are
added. A minimal backend now implements only from_config + add + get_context
(verified by test_memory_manager_interface.py::_MinimalBackend onboarding via
the factory); the factory no longer knows a backend's private hooks.

- MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1
  add/get_context abstract; tier-2 management defaults; tier-3 optional hooks
  warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving
  hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError.
- from_config classmethod: factory thins to resolve + inject storage_path +
  collect host hooks + call from_config; DeerMem-specific hook consumption
  moved from factory to DeerMem.from_config.
- Invariants: @model_validator (mode='tool' requires search via supports_search
  ClassVar); DeerMemConfig storage_path-is-file check moved here from factory.
- Async: aadd/aget_context/asearch default to the sync path (speculative).
- Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call
  subsumes tracing_callback (same signature/timing/mutation); deleted the
  tracing_callback field. DeerMem decoupled from langfuse (portability).
- noop keeps read-op empty overrides (avoids router 500s on the
  disable-memory-via-noop path); only delete/export inherit the base raise.

Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten
(three-tier + from_config + callbacks); samples README updated; removed stale
private doc paths.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): 501 on unsupported read/manage endpoints + accurate warm log

Review follow-up on the three-tier MemoryManager refactor.

- Read/manage endpoints (GET /memory, /memory/export, /memory/status,
  DELETE /memory, POST /memory/import) and the /memory/reload fallback now
  catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The
  hasattr->try/except migration had skipped these: they were @abstractmethod
  before (every backend implemented them, so they never raised), so once they
  became tier-2 default-raise a minimal backend (only add + get_context) hit a
  raw 500 -- there is no global NotImplementedError handler. get_memory is
  shared via _get_memory_or_501 (covers /memory, export, status, reload
  fallback). noop is unchanged: its read-op empty overrides never raise.
- warm() base default returns None (tri-state: True=warmed, False=failed,
  None=nothing to warm) so the Gateway lifespan logs "skipping" for a
  non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed
  successfully" it never earned. DeerMem.warm keeps True/False.
- Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan
  warm-log tests (None->skipping, False->warning); conformance/pluggable
  assert warm() is None.

705 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity

Address review feedback on the three-tier MemoryManager refactor:

- [Medium] supports_search/search drift: the invariant now requires the
  supports_search ClassVar flag to MATCH whether search() is actually
  overridden (type(self).search is not MemoryManager.search), so the flag
  can't drift from the impl. Catches both directions at instantiation: a
  backend that overrides search() but forgets supports_search=True (was a
  misleading tool-mode rejection), and one that sets the flag without
  overriding (was a runtime NotImplementedError on the first memory_search).
  noop sets supports_search=True to match its search() override. Conformance
  adds drift + consistent-backend tests.
- [Low] client.reload_memory fallback: wrap the get_memory fallback so a
  minimal backend (only add + get_context) surfaces a clean NotImplementedError
  ("implements neither reload_memory nor get_memory") instead of an uncaught
  propagation -- mirrors the router's 501. Test added.
- [Low] backend_config purity: DeerMem.from_config restores backend_config to
  the pure data the host passed after model_post_init parses the injected hooks
  into DeerMemConfig (self._config, PrivateAttr); the field stays serializable
  (no callables/LLM) and matches the README ("host hooks NOT in backend_config").
  Test asserts purity + hooks wired.
- [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend
  now fails fast at startup (was silently empty) so operators recognize it on
  upgrade.
- [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is
  local-only, not make test/CI).

709 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(changelog): correct memory tool-mode fail-fast note

The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)"
fails fast at startup, but noop overrides search() (returns []) and sets
supports_search=True (required by the consistency invariant), so noop IS
search-capable and noop+tool does NOT fail fast. The fail-fast only affects a
custom backend that onboards without overriding search(). Reworded to drop the
misleading noop example and state both shipping backends implement search().

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-22 14:40:57 +08:00
RongfuShuiping
4bf028d048
feat(memory): add incremental agent-scoped Markdown fact storage (#4279)
* feat(memory): add Markdown fact storage repository

* docs(memory): explain storage rewrite for beginners

* docs(memory): fix plan markdown formatting

* refactor(memory): separate global summaries from agent facts

* fix(memory): make Markdown fact updates incremental and safe

* Update STORAGE_REWRITE_CHANGES.md

* Delete docs/plans/STORAGE_REWRITE_PLAN.md

* Delete docs/plans/STORAGE_REWRITE_CHANGES.md

* fix(memory): address Markdown storage review feedback

* fix(memory): complete review follow-ups

* fix(memory): resolve storage review findings

* feat(memory): add proactive Markdown migration CLI

* fix(memory): harden Markdown storage concurrency

* fix(memory): harden markdown storage migration

* fix(memory): close migration review gaps

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: qin-chenghan <qinchenghan@Huawei.com>
2026-07-22 07:46:03 +08:00
Tu Naichao
959bf13406
fix(memory): flush memory queue on graceful shutdown to prevent loss (#4181)
* fix(memory): bounded shutdown flush via MemoryManager.shutdown_flush

Re-applies the memory-queue shutdown drain on top of the pluggable
MemoryManager abstraction (#4122): the old top-level MemoryUpdateQueue
singleton is gone, so the drain is now a backend contract instead of
host code reaching into the queue.

- MemoryManager ABC: shutdown_flush(timeout) -> bool. Every backend
  implements a bounded graceful-shutdown drain.
- DeerMem: queue.flush_sync (daemon-thread + Event.wait hard timeout
  for the uninterruptible sync LLM call; joins an in-flight worker
  first so contexts a debounce Timer already pulled out are not lost on
  exit; skips inter-item sleep on the drain path; per-item
  succeeded/failed count), exposed via shutdown_flush.
- noop: shutdown_flush is a clean no-op success.
- Gateway lifespan: call get_memory_manager().shutdown_flush(timeout)
  after channels/scheduler stop, via asyncio.to_thread, try/except
  bounded. No host-level pending/processing guard -- the backend
  short-circuits on an idle buffer, so the host cannot "forget" the
  in-flight case (structurally eliminates the guard race flagged on the
  prior revision).
- shutdown_flush_timeout_seconds added to the shared MemoryConfig
  (host-owned lifecycle budget, default 30, 1-300) + exposed on
  MemoryConfigResponse and the embedded client; config_version 25 -> 26.

Tests: queue flush_sync (7), lifespan drain incl. False-branch caplog
assertion + disabled gate (3), ABC contract noop/deermem (3).

* fix(chart): gateway grace period so memory drain is not SIGKILLed

K8s defaults terminationGracePeriodSeconds to 30s, shorter than the
Gateway's graceful-shutdown work (channel stop ~5s + memory queue drain
default 30s). Without an explicit grace period, K8s SIGKILLs the memory
drain mid-flight and silently re-introduces the loss shutdown_flush is
fixing (flagged on the prior revision).

- gateway pod: terminationGracePeriodSeconds (default 45, configurable).
- gateway container: preStop sleep (default 5, 0 disables) so the
  Service/ingress deregisters the pod before SIGTERM begins the drain.
- values.yaml + README: both configurable; README documents that the
  grace period must track memory.shutdown_flush_timeout_seconds.

* docs(memory): document shutdown_flush_timeout_seconds + lifespan drain

Add the host-shared field to the memory config list and Config Schema
summary in backend/AGENTS.md, noting the lifespan drain and the K8s
grace-period relationship.

* fix(chart): bump embedded config_version to 26

The chart's embedded `config:` block (values.yaml + README example) still
had config_version: 25 after commit f3ca8e9f raised config.example.yaml to
26, failing the validate-chart config_version drift check. Bump both to 26.
2026-07-15 20:25:41 +08:00
qin-chenghan
ad45f59d66
feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122)
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)

Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated

Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
  factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema

* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks

Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):

- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
  consolidation_min_facts / consolidation_max_groups_per_cycle /
  consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
  + CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
  _build_consolidation_section module helpers (matching the existing
  _select_stale_candidates style); consolidation normalization in
  _normalize_memory_update_data; consolidation apply in _apply_updates (after
  max_facts trim, with apply-time guardrails mirroring staleness); staleness
  KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
  staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
  incl. the staleness KeyError regression

Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): add noop backend template and backends guide

- backends/noop/: complete drop-in template (config.py with zero deer-flow
  imports, noop_manager.py with a 6-step new-backend walkthrough in its
  docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
  the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
  references).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): guard formatTimeAgo against invalid timestamps

Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(memory): wire tool-driven memory mode through the MemoryManager ABC

tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.

Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)

* docs: restore explanatory comments in config.example.yaml memory section

* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py

* fix(memory): address review + port dropped upstream memory fixes

Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
  MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
  Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
  new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
  over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
  knobs live under backend_config, not top-level - restoring flat would
  re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
  top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
  request-trace ContextVar on the Timer/executor worker via a new
  trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
  while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
  explicit manager_class instead of silently falling back to DeerMem (memory is
  persistent state - a wrong store is a silent data-integrity footgun)

Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
  (core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
  (deer_mem.py DeerMem.search)

host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)

* fix(memory): restore hide_from_ui clarification preservation, expose mode

Two memory-system fixes (F541/E501 lint was already fixed on this branch):

- filter_messages_for_memory: restore default preservation of well-formed
  human_input_response clarification answers (v2 regression). The
  self-containment refactor made the bare function skip ALL hide_from_ui when
  no hook was passed, but upstream preserves well-formed clarification
  responses by default (test_hide_from_ui_human_input_response_is_preserved).
  Inline a host-agnostic _is_human_clarification_response mirror of
  read_human_input_response as the default keep-decision; the host-injected
  should_keep_hidden_message hook still overrides (production path unchanged).
  Portable package stays zero `from deerflow`.

- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
  the config/status endpoints + client.get_memory_config. mode is a host-
  shared, behavior-determining field missing from the response projection.
  Sync tests (mock .mode; e2e assert mode present).

- Align manager_class field docstring with fail-fast behavior.

Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): resolve ruff format failures in memory module + tests

`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs

Address willem-bd's review on PR head bc8bf0d4 (risk:high, persistent state):

- config: auto-migrate pre-abstraction top-level memory.* DeerMem fields
  (storage_path, max_facts, debounce_seconds, model_name, token_counting,
  staleness_*, consolidation_*) into backend_config on load + warn, so an
  upgrade does NOT silently revert customized settings (was: silent
  extra='ignore' drop). model_name -> backend_config.model.model. Unknown
  top-level keys warned.
- factory: resolve a relative backend_config.storage_path against runtime_home()
  (base_dir-relative, CWD-independent) to preserve pre-abstraction semantics;
  paths.py stays portable (no runtime_home import).
- tools: memory_add uses the fact_id returned directly by create_fact instead of
  re-deriving it via content-key matching (coupled the tool to the backend's
  content normalization; could misreport a storage cap). create_fact now returns
  (memory_data, fact_id); gateway/client/tool updated. Fix terse
  {"error":"content"} -> {"error":"empty content"}.
- app.py: update stale token_counting=="char" warm-up comment to point at
  manager.warm (DeerMem.warm re-checks char and returns early).
- router: comment explaining reload_memory silent fallback vs fact 501 asymmetry
  (read-only degrade vs write fail-loud).
- CHANGELOG: document breaking changes (/memory/config + client.get_memory_config
  shape flat->backend_config; custom storage_class path moved + __init__ must
  accept config) and the legacy-field auto-migration.
- tests: add regression test pinning the per-user memory path
  ({storage_path}/users/{safe_user_id}/memory.json == host make_safe_user_id)
  across the abstraction; update create_fact mocks for (memory_data, fact_id).

Tests: 273 passed (memory suite); ruff check + format clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address PR review - storage_path, max_facts, tracing, parsing

Six review findings (willem-bd), each verified against upstream:

- storage_path semantics (file -> root dir): migration drops file-style
  (.json) legacy values with a warning; factory raises if storage_path
  resolves to an existing file (avoid silent NotADirectoryError write
  failure). CHANGELOG + config.example.yaml comment updated.
- create_memory_fact enforces max_facts again (via _trim_facts_to_max) and
  returns (memory, None) when the cap evicts the new fact; memory_add tool
  reports "not stored", client raises ValueError, POST /memory/facts -> 409.
- max_facts trim uses _coerce_source_confidence (was raw f.get("confidence",
  0) -> TypeError on non-float imported/legacy confidence, swallowed as
  silent update failure).
- memory-tracing assistant_id restored to "memory_agent" (was "lead-agent"
  copy-paste; matches upstream + DeerMem run_name).
- _is_human_clarification_response cross-checked against
  read_human_input_response (drift guard test).
- empty-string legacy values skipped silently in migration (narrow fix, not
  broad "if not value" which would skip explicit bool False).

8 new regression tests. make lint + 406 memory tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): address internal review - storage fail-fast, build_llm degrade, config warn, noop template

Addresses 4 findings from the PR #4122 internal supplemental review
(parallel to willem-bd's review, no overlap):

- create_storage fail-fast: a misspelled/unimportable storage_class now
  raises ValueError instead of silently falling back to FileMemoryStorage.
  Memory is persistent state, so a wrong store is a data-integrity footgun;
  mirrors the existing manager_class resolution policy. (storage.py)

- noop template create_fact signature: the commented template used
  keyword-only `content` and returned a bare dict, while DeerMem's actual
  create_fact takes positional `content` and returns tuple[dict, str|None]
  (the memory_add tool passes content positionally; gateway/client/tools all
  tuple-unpack). A backend copied from the template would 500 on fact-CRUD.
  Template fixed; delete_fact/update_fact templates left (callers compatible).
  (noop_manager.py)

- build_llm graceful degrade: wrap init_chat_model in try/except, degrade to
  None + WARNING on failure (mirroring _host_default_llm) so a misconfigured
  explicit model does not crash app startup -- non-LLM memory ops still work
  and an update raises at runtime with the error logged. (llm.py)

- from_backend_config unknown-key warning: log a WARNING for unknown
  backend_config keys (mirrors the host layer's load_memory_config_from_dict)
  so a typo like `storage_pat` does not silently fall back to the default and
  write memory to an unintended location. (config.py)

Tests: rewrote 3 create_storage fallback tests to expect ValueError; added 4
tests (build_llm zero-config/degrade, from_backend_config warn/silent).
make lint green; full memory suite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: lllyfff <2281215061@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: lllyfff <122260771+lllyfff@users.noreply.github.com>
2026-07-15 11:21:04 +08:00