mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-03 00:50:35 +00:00
124 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cordis798
|
09a98e5f7b
|
fix(memory): honor Mem0 read policies during context deduplication (#7561)
Co-authored-by: Cordis798 <284918518+Cordis798@users.noreply.github.com> |
||
|
Hyeonsang Cho
|
30b3a012cb
|
fix(memory): flush pre-compaction memory off the event loop (#7530)
* fix(memory): flush pre-compaction memory off the event loop The async compaction path fired the before-summarization hooks inline. memory_flush_hook hands the summarized messages to add_nowait, which is sync HTTP for mem0 and Honcho (inherited default -> add) and OpenViking (its own override), so every automatic compaction with an HTTP memory backend blocked the Gateway event loop for the whole request. acompact_state now runs the hooks through asyncio.to_thread, as MemoryMiddleware.aafter_agent already does for its writes. Ordering, failure isolation, summary-first timing and context propagation are unchanged; the sync compact_state path is untouched. * docs(changelog): reference #7530 for the pre-compaction memory flush fix * test(memory): anchor the OpenViking pre-compaction flush off the event loop --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
jianyx1
|
4b1e06bbf9
|
feat(runtime): honor the web UI language in replies and DeerMem updates (#7294)
* feat(runtime): honor the web UI language in replies and DeerMem updates Carry Settings > Appearance > Language as request-only run data: the Gateway validates context.locale against a finite allowlist, the lead agent prefers it for replies, and bundled DeerMem writes new facts and summary prose in it. Explicit user language requests still take precedence; checkpoint configurable, static prompts, and memory backend APIs are unchanged. Refs #7287 Co-Authored-By: Claude <noreply@anthropic.com> * fix(runtime): guard language context and pin locale maps * test(runtime): pin UI locales across frontend and Gateway Add the shared locale fixture and document the intentional language fallback for internal runs. Clarify live-locale precedence in the run context contract. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
PerryLink
|
338cd26dc2
|
fix(memory): drop unusable numbers in stale extensions and consolidation confidence (#7442)
* fix(memory): drop unusable numbers in stale extensions and consolidation confidence * fix(memory): log defaulted consolidation confidence and pin the int passthrough |
||
|
Lym308758
|
346ff7b253
|
fix(memory): rank capacity eviction per record, not per fact_id (#7454)
* fix(memory): rank capacity eviction per record, not per fact_id select_facts_for_capacity keyed every score by fact_id, so an imported batch that repeats an id collapsed into a single dict entry: the last record's score won, and both the ranking pass and the evicted audit read that one value for every position. A 0.9-confidence record could therefore be evicted while its own 0.1-confidence duplicate survived. The import path keeps caller-supplied ids without checking them for uniqueness, and any over-cap import lands here. Score by position instead: a parallel list next to the indexed facts, so ranking and the audit read each record's own score. The public ``scores`` dict stays keyed by fact_id with last-write-wins semantics, keeping the DeerMem benchmark contract and the callers that never read it unchanged. Red on main: test_confidence_policy_scores_duplicate_fact_ids_per_record fails with the highest-confidence record evicted; green on this branch. * fix(utils): keep share-less UNC roots out of the extended-length prefix extended_length_path prefixed every absolute UNC path with \\?\UNC\, but Win32 only accepts \\?\UNC\server\share\... — a share-less root (\\server, \\server\, or a bare \) is not a valid device path, so every Win32 call handed the prefixed spelling fails. Only prefix when both the server and the share are present; otherwise return the ordinary spelling unchanged. Red on main: the \\server, \\server\, //, and //server cases of test_windows_extended_spelling assert the forged prefix; green on this branch. |
||
|
Cordis798
|
3e9a417ea6
|
fix(memory): apply Mem0 read policies to record conversion failures (#7307)
* fix(memory): apply Mem0 read policies to record conversion failures * test(memory): cover Mem0 read policies through routes and tools * fix(memory): identify malformed Mem0 records in diagnostics --------- Co-authored-by: Cordis798 <284918518+Cordis798@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Fish
|
7097694abe
|
fix(memory): honor Mem0 failure policies for malformed JSON responses (#7202)
* fix(memory): honor Mem0 failure policies for malformed JSON responses * fix(memory): diagnose and normalize deep Mem0 response failures * fix(memory): include Mem0 page context in response errors --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Fish
|
97b12b1381
|
fix(memory): exclude hidden framework messages from Honcho writes (#7159)
* fix(memory): exclude hidden framework messages from Honcho writes * fix(memory): isolate hidden-message hook failures |
||
|
Nan Gao
|
dc6f0d7421
|
fix(memory): persist the DeerMem extraction watermark across restarts and replicas (#7205)
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
|
||
|
李刚 / Bryan Li
|
1fd266e0c0
|
feat(decisions): add an OpenAI Decisions backend for the decision slots (#7127)
* feat(decisions): add an OpenAI Decisions backend for the decision slots Issue #7005 asked for a second decision provider behind the interfaces the TypeSafe (Jev) integration already uses, not a new decision system. `deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question becomes a `predicate` whose true/false rubric folds into `instructions` (the endpoint has no criteria field, and dropping it would move the decision boundary while the adapter kept reporting the same criteria in its policy identity), answers arrive as an array rather than a mapping, and a refusal is an answer `type` rather than an HTTP failure -- so it lands in the existing per-question error channel and each consumer keeps its own failure direction. Transport is reused rather than copied. `provider_label` and the connection's `endpoint_path` are the only two seams a dialect needs, so retry and backoff, the deadline budget, the bounded never-decoded body read and the error taxonomy keep one owner; `resolve_decisions_connection` pairs the two so a Decisions connection cannot be built pointing at System One. The three adapters override their connection resolution, their client, and the value the endpoint judges (`input` text instead of a `state` object), and inherit thresholds, rubrics, caches and audit policy unchanged. Wiring stays a `use:` class path, so no config schema is added. * fix(decisions): isolate endpoint identities and correct diagnostics --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
LuckVd
|
2174c1d81e
|
fix(memory): quarantine unreadable memory summaries (#7076)
* fix(memory): quarantine unreadable memory.json in the read-error branch too MarkdownMemoryStorage._load_memory_file quarantined summaries that failed to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError) branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without quarantining. The leftover file was then silently overwritten by the next save, destroying the user's memories with no backup. Call the existing best-effort self._quarantine_unreadable(path) there as well, matching the documented quarantine contract, and add a regression test asserting the original bytes survive a subsequent load+save. Fixes #7072 * fix(memory): preserve summaries on transient read failures --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
LingYi
|
ee02e51df9
|
fix(memory): preserve consecutive pre-compaction snapshots (#7152)
* fix(memory): preserve consecutive pre-compaction snapshots * fix(memory): warn on growing emergency snapshot backlogs --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yetuge
|
18ca09b2da
|
fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption (#7116)
* fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption Facts are hand-editable Markdown files, so a Windows editor can save one as "UTF-8 with BOM" -- the same exposure #7083 fixed for memory.json. The fact parser checks `text.startswith("---\n")` against a plain-utf-8 read, so the BOM makes a valid fact read as "missing YAML front matter" and raises MemoryStorageCorruption, which the module docs describe as taking down the whole agent. Read the fact with utf-8-sig, matching the summary loader and the SOUL.md / SKILL.md convention already recorded in AGENTS.md. `_parse_listed_fact` delegates here, so the directory-scan path is covered by the same change. Measured: the new BOM test fails on the base revision with exactly that "missing YAML front matter" corruption and passes at head; the no-BOM control passes both. 49 tests in test_memory_storage.py, 106 across the memory suites, ruff check/format clean. Mutation-checked: reverting to plain utf-8 fails only the BOM case. * test(memory): cover BOM facts through the public storage API --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jay0130-a
|
0a1c86a371
|
fix(memory): parse a UTF-8 BOM in the DeerMem summary instead of raising corruption (#7083) (#7084) | ||
|
哈基米
|
816ebd5563
|
fix(memory): filter honcho search by category instead of stamping it on unfiltered results (#6981)
* fix(memory): filter honcho search by category instead of stamping it on unfiltered results HonchoMemoryManager.search sent an identical unfiltered query whether or not a category was requested, then stamped the requested category on every hit, so memory_search(category=...) returned unfiltered results attested to be that category (e.g. a work fact labeled "preference") — the opposite of the contract, where category filters before the top_k slice and only facts with that exact category are returned. Send the category as a real server-side metadata filter (Honcho applies "metadata" filters in its ranked query, before the limit) and label each result from the record's own metadata, falling back to "memory" as before. HonchoClient.search gains a keyword-only filters argument that is only put on the wire when set, so unscoped requests are byte-identical to before. * docs(memory): note honcho category filtering requires stored metadata |
||
|
ordinary being
|
cd0166ae76
|
fix(memory): drop out-of-range extraction confidence instead of failing the batch (#6971)
* fix(memory): drop out-of-range extraction confidence instead of failing the batch A model-emitted confidence outside [0, 1] (a percentage such as 95, or 1.5) passed extraction normalization, which only rejected non-finite values. The value then reached storage, whose own validation raises for out-of-range confidence, so one malformed field failed the whole update: the extraction watermark never advanced and the same paid call repeated on every turn, while every valid fact, summary and removal in the batch was lost each cycle. Treat the range like the other malformed shapes in _normalize_memory_update_fact: drop only that item and let the rest of the update apply. Read paths already clamp stored corrupt values (_coerce_source_confidence), so no read-side change is needed. * fix(memory): bound the confidence drop and make it observable (review) Review follow-ups on the out-of-range confidence fix: - range-check integer confidence before float(), so a model emitting 10**400 is dropped instead of raising OverflowError and failing the batch; - log the raw value when a fact is dropped, so a systematically percentage-scale model is visible instead of silently losing every fact; - document that a batch which also proposes factsToRemove still fails closed when any new fact is dropped (replacementFactIndex positions shift), and pin that boundary with a test. Values are documented as dropped, never rescaled. |
||
|
万家良
|
4e6c18ad9c
|
fix(memory): retry the scope lock open to absorb a Windows create race (#6870)
On Windows, two threads/processes racing to create a not-yet-existing lock file with an append open can transiently fail on open() itself with PermissionError (EACCES), before the byte-range lock is ever attempted. The open was outside the retry loop, so the error surfaced from load()/save() instead of being retried. Move the open inside the loop and close the handle on any OSError before retrying. |
||
|
哈基米
|
f4d10ef42d
|
fix(memory): read the complete scope in the substring search fallback (#6732)
* fix(memory): read the complete scope in the substring search fallback _search_substring iterated list_facts without paginating, so the adapter-less fallback only ever saw the default page of 100 facts. With 150 matching facts in a scope (max_facts allows up to 500), memory_search returned 100 results for top_k=150 and silently dropped every match past the first page; DeerMem.search keeps these results whenever they are non-empty, so its uncapped fallback never ran. Read the complete scope instead, as the scoped index rebuild already does. * test(memory): drop in-page fact-100 from the past-page assertion fact-100 was the last id inside list_facts' old first page (``matched[0:100]`` in insertion order), so its presence does not distinguish the fixed search from the pre-fix page; fact-101 and fact-150 (plus the ``len(results) == 150`` check) carry the regression signal. Keep a boundary note so the remaining ids are not misread. |
||
|
哈基米
|
1b7ca5ecf1
|
fix(memory): keep non-core fact categories instead of rewriting to other (#6733)
* fix(memory): keep non-core fact categories instead of rewriting to other
_normalize_category rewrote any category outside CORE_CATEGORIES to
"other" and parked the original label in a write-only
categoryExtension field, so memory_add(category="personal") -- an
example label in the tool's own docstring -- persisted as "other" and
memory_search(category="personal") ("Only facts with this exact category
are returned") never matched it. get_memory documents kept reporting
"other" as well.
Keep the caller's label as the stored category so it round-trips
through search, documents, and the retrieval index.
* fix(memory): finish migrations interrupted across the category change
A pre-native-category Gateway committed a per-agent legacy JSON migration
as category="other" + categoryExtension=<label> and could stop before
deleting the legacy JSON (crash / failed unlink). The retained JSON holds
the label natively, so _migrate_locked compares "personal" against its
own committed "other"/"personal" alias, raises MemoryStorageCorruption,
and fails every subsequent read, reload and migration retry for that
agent (HTTP 500 at the Gateway) -- a recoverable state the previous
build recovered from.
Fold the legacy alias into _migration_equivalent's category comparison
("other"/<label> equals <label>/no-extension) so retries recognize and
complete recovery, while a genuinely different label still stops the
migration with the legacy file preserved.
Also drop a stale categoryExtension when a fact is normalized away from
"other": _normalize_fact deep-copies unknown fields through, so a legacy
fact updated to its native category kept surfacing the leftover
extension, contradicting the Gateway contract that the field is only
set "when category is 'other'".
Both regressions are pinned by replaying the historical writer in tests.
---------
Co-authored-by: BlueX888 <BlueX888@users.noreply.github.com>
|
||
|
Jie Ji
|
bdc2d12367
|
fix(memory): prevent cross-conversation debounce starvation (#6672)
* fix(memory): prevent cross-conversation debounce starvation * fix(config): bump schema version for memory max wait |
||
|
Nan Gao
|
5ae953fc2f
|
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Hyeonsang Cho
|
c15ecb688d
|
fix(migration): read thread owners from the configured database (#6450)
* fix(migration): read thread owners from the configured database
migrate_user_isolation.py looked for thread owners in
{base_dir}/deer-flow.db, a file DeerFlow never creates. The database is
{sqlite_dir}/deerflow.db or PostgreSQL, so the owner map was always empty
and every legacy thread was moved to users/default/.
Read threads_meta through the configured database's sync SQLAlchemy URL
(SQLite or PostgreSQL, honoring postgres_schema). When legacy threads
exist but owners cannot be read (no database file, a memory backend, a
missing driver, or a failed query), stop before migrating anything;
--allow-missing-thread-owners keeps the old assign-to-default behavior
for installs that never recorded owners.
* docs(changelog): reference #6450
* fix(migration): document recovery and report unusable database URLs
Installs that already ran the old script have every legacy thread under
users/default/threads/, and re-running finds no threads/ to migrate. The
provisioner README and AUTH_DESIGN now give the recovery: move the owned
threads back to threads/ and re-run; an existing owner directory is never
overwritten and the old copy goes to migration-conflicts/. A test pins
that re-run behavior.
create_engine errors other than ImportError (e.g. ArgumentError from an
empty postgres_url) now stop with the same "Nothing was migrated." message
instead of a traceback.
The memory AGENTS.md note said threads go to --user-id; they go to their
threads_meta owner.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
ZJPex
|
220d62d79a
|
feat(memory): add scoped fact lookup by ID (#6443)
* feat(memory): add scoped fact lookup by ID * docs(memory): synchronize tool-mode lookup guidance * docs(memory): clarify named-agent read capability requirement |
||
|
Guo-Yixin
|
789bbaaa0d
|
fix(memory): redact invalid and legacy tool-call arguments (#6387) | ||
|
wgwfn3
|
c54883cce0
|
fix(memory): drop punctuation tokens from jieba relevance ranking (#6388)
* fix(memory): drop punctuation tokens from jieba relevance ranking jieba.cut emits punctuation as standalone tokens and tokenize() only filtered whitespace, so commas and full stops counted as query matches, inflated near-duplicate similarity, and consumed the 128-token budget. Keep only tokens with an alphanumeric character, matching the regex fallback and the FTS5 adapter's query filter. Document the rule in agents/memory/AGENTS.md. * docs(changelog): add the DeerMem jieba punctuation fix * docs(changelog): add the PR reference for the jieba punctuation fix (#6388) * docs(memory): document jieba relevance token filtering --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Guo-Yixin
|
1523a94551
|
fix(memory): exclude invalid and raw assistant tool calls (#6379) | ||
|
Hyeonsang Cho
|
1f7600e355
|
fix(memory): treat facts deleted during unlocked DeerMem scans as removed (#6255)
* fix(memory): treat facts deleted during unlocked DeerMem scans as removed load(), reload(), and the full rebuild_index() scan list agent fact files without the storage locks. A delete committed between the listing and opening a file raised MemoryStorageCorruption for intact data, which surfaced as HTTP 500 from the memory API and an empty memory block in the prompt. A listed fact that no longer exists is now skipped; an entry that is still present but unreadable is still corruption. * docs(changelog): reference #6255 for the DeerMem vanished-fact fix * docs(memory): note lock coverage and heuristic limits of the vanished-fact skip |
||
|
Hyeonsang Cho
|
e3064f541b
|
fix(memory): sign DeerMem reload before reading the document (#6238)
* fix(memory): sign DeerMem reload before reading the document reload() read the document before computing its cache signature, so a write committed in between cached the old document under the new signature and later load() calls served stale memory until the next write. Compute the signature first, matching load(). * test(memory): cover markdown storage and agent scope in reload race test * docs(changelog): reference #6238 for the DeerMem reload fix |
||
|
Grapette.L
|
f61372764e
|
fix(agents): anchor agent-name validation against a trailing newline (#5960)
* fix(agents): anchor agent-name validation against a trailing newline
Two copies of the shared ^[A-Za-z0-9-]+$ grammar still used re.match,
whose $ also matches before a final newline, while every other copy uses
fullmatch. POST /api/agents {"name": "reviewer\n"} therefore passed the
router guard and died inside the strict file store as an HTTP 500, and
DeerMem accepted the same name as an agents/{name}/facts directory.
* test(agents): state which trailing-newline params the regression actually covered
.match only accepted "reviewer\n" on main; "reviewer\n\n" and "reviewer \n"
were already rejected, so the docstrings should not read as if every param in
these parametrizations was broken before the fix.
|
||
|
PeaceMaker-best
|
b73a4e79bf
|
feat(memory): scope management API by agent (#5565)
* feat(memory): scope management API by agent Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(memory): preserve scoped management isolation Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
Zeren Wang
|
d700ad9edf
|
feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906)
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client - refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its configuration: one connection-precedence rule, credential fingerprint, wire-size counting and error taxonomy, shared by the guardrail gate and the memory hooks. - feat(memory): wire `memory.prescreen` and `memory.signal_classification` into DeerMem through one injected judge hook (both off by default, `enforce` gated by the shadow evaluation). - config: add the top-level `typesafe:` block and the two judging slots to `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync the Helm README example with it. - test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with per-stratum gates, censoring and the no-network baseline. Review follow-ups folded in: - `mutations_accepted` counts only facts that survive capacity enforcement and the upload-event scrub that precedes persistence, only removals/reinforcements of ids that existed, and persisted near-duplicate merges (matched by id, or by identity for legacy id-less facts); it is published only after persistence succeeds. - The cached manager hot-reloads its judge when a memory slot or the shared `typesafe` block changes (including a credential rotation); publication is serialized and revalidates the config generation, so a build started before an `enforce` -> `off` rollback cannot land after it. - Cache provenance is per answer: the `cached` flag and the reported model follow the answers a side actually consumed, across partial merges and answerless retries. - An oversized noul integer stays a per-question error instead of raising. - Incremental call savings use the shared scored denominator. - The sensitive-stratum gate requires evidence for each of identity, preference and correction. * fix(typesafe): resolve one credential per layer and refuse unusable connection values Review follow-ups on the shared TypeSafe client (#5906): - A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the shared `typesafe:` block: the first layer that sets either credential setting decides it, and inside one layer the literal key wins (the standalone semantics). A consumer whose endpoint does not accept the block's key no longer sends it on every call, and the missing-credential message keeps naming the variable that layer configured. - `base_url` is validated where it is configured: `http(s)` only, with no query, fragment, embedded credentials, whitespace or control characters, and a malformed port fails at construction instead of on every request (httpx raises `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been swallowed into the appended endpoint path). - A credential that cannot travel as a header value — surrounding whitespace from a secret mounted from a file, a non-printable character — is refused when the connection resolves. It used to construct fine and then put the whole `Bearer <key>` value into h11's `LocalProtocolError` message, which the guardrail middleware's `logger.exception` prints. - The client raises transport failures without chaining the original exception, so that message can no longer carry the credential through the cause chain; the exception type stays in the message for diagnosis. - `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config` path: `max_attempts: true` was silently read as `1`. An integer stays a usable float, so `timeout: 5` keeps working. Tests: 20 cases across `tests/test_typesafe_config.py` and `tests/test_typesafe_client.py`, each red on the previous revision and green now. 367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean. Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`, `config.example.yaml`. * fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence Second review round on #5906: the remaining items from the review thread and from the follow-up pass. Typesafe client (the reply is treated as hostile input too): - Send `Accept-Encoding: identity`, refuse a body that carries a non-identity `Content-Encoding`, and read through a bounded `iter_bytes` that stops past `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be decoded in full and, on the gate path, allowed the tool call unexamined; it is now `invalid_response` (a denial with `fail_closed: true`). - Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting escaped the module's taxonomy, so the memory consumers lost the round's record instead of recording a fallback. Memory judge: - A request-level failure is recorded as `request_failed`, never `no_verdict` (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable endpoint was indistinguishable from a provider with no usable answer). The adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record and log; `no_verdict` stays for a question-level failure, and an ineligible side keeps its own reason even when the round's request failed. Shadow evaluation (the enablement gate reads the evidence, not one number): - The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to meet the 1% target; a sample that only meets the point estimate is `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound). - New `baseline_incremental_savings` gate: a baseline that saves as many calls as the model — the shipped all-trivial sample — fails instead of enabling. - Negative counts (`mutations_accepted`, message / batch / token counts) are treated as absent rather than scored: `-3` read as "nothing lost", adding a phantom saving to the numerator and the denominator. Docs: the design documents cited but absent from the repository are replaced with in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`, `GUARDRAILS.md`), and the gate and limit wording now matches what the code enforces. Tests: 31 new or updated cases, each red on the previous revision (compressed body refused even when it decodes to a valid envelope, body over the cap refused, body at the cap answered, deep nesting mapped, `request_failed` on each request-level failure, the bound and baseline gates, the negative counter). 454 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` and `ruff format --check` are clean. * fix(memory-eval): pair the baseline comparison on the same records Review follow-up on the incremental-savings gate added in ed930478 (P2): both sides of the comparison are now the records the no-network baseline can score (`trivial_only` plus a readable signals list). Subtracting that subset's heuristic savings from model savings over *all* scorable skips credited the model with batches the baseline never saw: 300 reviewed skips with a single annotated record computed 300-1=299 "incremental", passed the gate and set `enable_recommended`, while the paired comparison was model=1 against heuristic=1 (incremental 0). - `Evidence` carries the paired numbers (`baseline_paired_skips`, `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one heuristic count, filled by `build_report` from the baseline summary. - The gate reports `INSUFFICIENT` when no scored skip carries the annotation and when the annotation covers only a subset of the scored skips (the message carries the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive paired gain, and `PASS` only on a positive gain over the full paired set. - Docs state the pairing/coverage requirement in `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring. Tests: the reviewer's counterexample as a report-level regression (it failed with `assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true), a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields. 344 focused tests, `make test-blocking-io` (167) and the eval suites pass; `ruff check` / `ruff format --check` clean. * fix(typesafe): bound the sync body read by the deadline `deadline_seconds` only covered the sync path's headers, the completed body and the parse, so an endpoint that drips bytes just under the read timeout held the caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget. The read loop now checks the budget before every transport read, which bounds the overshoot to one read. The pieces are the transport's own reads, not re-chunked output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk, so a check at that granularity would never see a drip. `tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24 pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now (~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track the new granularity. * fix(memory): read the YAML off spelling, and veto a skip over the whole batch Two review findings on the memory judging path. `memory.prescreen.mode` / `memory.signal_classification.mode` rejected the `mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1 parses the unquoted value as a boolean, so both fields saw `False`. The failed reload left the previous judge running, so both fields now normalize `False` to `off` before the `Literal` runs. `True` stays a validation error (no mode it could mean). The examples quote the value, and the docs record why. The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch it answers for is the whole post-watermark feed, so an explicit correction in an earlier turn of an eight-message batch did not stop the skip that then advanced the watermark past it. `detect_signals` gains a `window` parameter (default unchanged) and the updater scans the full feed for the veto; the extraction hint and the reinforcement gate keep the window. Both regressions are red on the previous head: the config test reproduced the `input_value=False` rejection through `AppConfig.from_file`, and the batch test reproduced the prescreen judging (and skipping) a batch whose first turn carries the correction. |
||
|
Shxiao
|
230d8f03d7
|
fix(memory): validate Honcho's allow_insecure_http as a boolean (#5887)
* fix(memory): validate Honcho's allow_insecure_http as a boolean * docs(memory): list the full Pydantic boolean vocabulary for allow_insecure_http |
||
|
xiaodu55
|
64e499be98
|
feat(memory): redact queued memory payloads (pii_redaction follow-up) (#5577)
* feat(memory): redact queued memory payloads (pii_redaction follow-up) Closes the remaining vector from #3190 (memory storage): request-scoped redaction leaves originals in thread state, and MemoryMiddleware queued that raw conversation into the durable extraction buffer — so the extraction model's input and the persisted facts carried raw PII even with pii_redaction.enabled. - MemoryMiddleware gains pii_redaction_config (same pattern as summarization / durable-context / title); the payload returned by _resolve_add_args() is redacted at the enqueue boundary before add()/aadd(), covering both after_agent and aafter_agent - Reuses the shared redact_text() seam from #5527 via active_pii_detectors/_Redactor/_redact_content; the buffered payload itself becomes clean, not just the extraction model's view - Wired at the lead-agent assembly site - Tests: queue-payload redaction enabled/disabled, detector toggles, message immutability, token stability across turns * fix(memory): reserve existing placeholder indices across the queued batch Review round 1 on #5577: thread state mixes raw user turns with already-redacted tool results and summaries, so a token in a later message must not collide with a new value in an earlier one. Pre-scan the whole queued batch with redactor.reserve() before allocating. Also extend the two MemoryMiddleware stubs in test_lead_agent_model_resolution.py to accept the new pii_redaction_config kwarg (CI backend-unit-tests red). * fix(memory): redact the compaction-triggered memory_flush_hook queue too Review round 2 on #5577: with memory + summarization enabled, memory_flush_hook() queues list(event.messages_to_summarize) directly into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely — and compaction removes those messages from state right after, so the after-agent redaction can never repair a raw batch queued there. Extract the batch redaction (reserve pre-scan + rebuild) into a shared redact_queued_messages() used by both MemoryMiddleware's enqueue boundary and the hook; the hook gains an optional pii_redaction_config wired via functools.partial at the summarization factory (subagents use skip_memory_flush=True and are unaffected). Regression covers the hook path enabled/disabled/absent. * test(summarization): unwrap the partial in the factory hook identity check edd916e wrapped memory_flush_hook in functools.partial to carry the pii_redaction config, but missed committing the matching update to test_factory_attaches_memory_flush_hook_by_default — a functools.partial is not equal to its wrapped function, so the identity assertion failed deterministically (caught by static review on #5577). * fix(memory): cover tool-call arguments and lenient text blocks in queued payloads Review round 4 on #5577 — two reproduced gaps still allowed raw PII into memory payloads with pii_redaction.enabled=true: - OpenViking retains the full message object, so parsed tool_calls args and provider-format arguments in additional_kwargs bypassed the content-only redaction; queued payloads now redact tool-call args recursively (and reserve them in the pre-scan). - DeerMem's format_conversation_for_update reads p.get("text") of any dict block, not only type=="text" — _redact_content now redacts every string-valued text field so a lenient consumer cannot bypass it. Both repro shapes are pinned as regressions. * fix(pii): value-derived placeholders for stable capture identity Review round 5 on #5577: sequential allocation is order-dependent — the same unchanged message re-redacted in a later batch can get a different token, which flips OpenViking's content-signature and replays already recorded turns into persistent memory sessions. Placeholders are now a pure function of the matched value (sha256-6 per category): identities stay stable across batches, enqueues, seams, and backends with no shared state and no stored mapping. - _Redactor becomes stateless; the reserve machinery (index pre-scans, _PLACEHOLDER_PATTERN) is removed along with its call sites - redact_texts keeps its signature, now a simple per-text redaction - reserve-semantics tests are superseded by cross-enqueue identity tests (two successive captures where the second batch carries higher existing placeholders); the compiled-graph echo regex matches the new format * fix(pii): 128-bit placeholder digests; redact tool-argument keys Review round 6 on #5577: - [P2] The 24-bit truncation collided distinct identities (reproduced: contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]). Retain 128 bits of the digest so separate people stay separate at any realistic volume; the compiled-graph echo regex and the AGENTS.md entry follow the new format. - [P2] _redact_strings redacted values but preserved mapping keys, so {"contacts": {"alice@example.com": "manager"}} kept the raw email as a key. Redact string keys along with values; deterministic 128-bit tokens make distinct-key collisions a hash-collision non-event, and a redacted key equal to another key merges the entries. Regression pins the email-keyed arguments shape. * docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording The chain grew past the 98304-byte AG002 hard limit again (main growth plus the round-6 format example). Trim the entry to fit, and replace the stale reserve-indices wording with the value-derived semantics that replaced it. * docs(agents): shrink PiiRedactionMiddleware entry below the chain budget main moved (#5609) and the base comparison flipped the nearly-over chain back to an AG002 error (98441 > 98304). Trim the entry to the facts that are not documented elsewhere: seams, allowlist, order, enqueue boundary. * fix(pii): letters-only value-derived tokens; wire the factory SDK path Review round 7 on #5577: - [P2] Hex digests carry digit runs that later digit-anchored detectors re-scan, corrupting minted tokens into nested placeholders (reproduced: user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit digest is now encoded over a-z, so a minted token survives the full pinned detector order. - [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware and DurableContextMiddleware without pii_redaction_config; the knob is now an explicit optional parameter threaded through _assemble_from_features (the factory reads no config files by design, so SDK callers pass it explicitly). - [Nit] hoist the Command rebuild comprehension out of the per-message loop in _redact_result. - [Nit] drop the dead _iter_tool_calls helper and Iterator import. * test(factory): pin pii_redaction_config wiring; document the parameter Review round 8 on #5577: - [P3] test_create_deerflow_agent.py had zero coverage of the pii_redaction_config parameter; two regressions now assert that create_deerflow_agent(pii_redaction_config=cfg) builds a MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that the default is None (redaction off). - [Nit] document the parameter in the create_deerflow_agent docstring: None leaves redaction off; a middleware= takeover bypasses the assembly that would apply it. * fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain Review round 10 on #5577: - [P2] sha256(category||value) is an unkeyed, globally deterministic fingerprint: a provider or memory backend can confirm candidate values offline and link identical values across users and threads, contradicting the documented no-re-identification guarantee. Placeholders are now HMAC-derived over a deployment-scoped token_secret (new config field); the docstring documents the linkability scope for both modes. - [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so the SDK knob enabled only the downstream seams while the primary model still received raw content. The middleware is now appended in the same relative position as the lead-agent base chain when the gate is on, with a model-call regression through the assembled SDK chain. * docs(pii): document token_secret; update the placeholder format examples Follow-up to the round-10 review on #5577: - config.example.yaml documents the new token_secret knob: a deployment-scoped secret makes placeholders HMAC-derived and only linkable within one deployment; leaving it unset keeps publicly computable fingerprints (documented trade). - the PiiRedactionConfig class docstring's placeholder example now matches the shipped format (128-bit HMAC digests over base-26 letters). * docs(pii): update the module docstring's placeholder format example The module docstring still showed the sequential [EMAIL_1] example; align it with the shipped value-derived format (128-bit HMAC digests over base-26 letters). * fix(pii): require a non-empty token_secret when redaction is enabled Review round 11 on #5577: with the secret optional, the default enabled configuration minted empty-key HMAC digests — publicly computable, globally linkable fingerprints of the raw values, confirmable by offline guessing for low-entropy identifiers such as phone numbers. - PiiRedactionConfig now rejects enabled=true without a non-empty token_secret (minimum 16 characters); the unkeyed mode is gone entirely. - The request-scoped redactor in _process_request now carries the configured key: it built its redactor without it, so user-message placeholders silently diverged from every other seam for the same value. The empty-key guard in _placeholder_token is the backstop that keeps every path keyed. - Module/config docstrings and config.example.yaml no longer describe the placeholders as irreversible or the secret as optional; the stale reserve-semantics wording and the "memory path not covered" note are corrected. - Regressions: validation errors for missing/blank/short secrets, and the user-message boundary minting the same keyed token as redact_text. * fix(memory): redact original_user_content on queued memory payloads Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in additional_kwargs.original_user_content, but redact_queued_messages rewrote only content, tool_calls, and provider-format tool_calls — the queued copy still carried the raw value, so the memory queue and backend received raw PII with redaction enabled. The shared helper now redacts the provenance string field on the queued copy (same batch redactor, merged with the existing additional_kwargs rewrite, rebuilt via model_copy — thread-state messages stay untouched), covering both the enqueue boundary and the compaction flush hook. Regressions pin the redacted provenance field through both paths and the untouched original. * fix(memory): redact human_input_response on queued memory payloads Review round 13 on #5577: clarification replies keep the raw user answer in additional_kwargs.human_input_response.value (DeerMem reads that mapping via its read_human_input_response mirror), but redact_queued_messages rewrote only content, tool_calls, and original_user_content — the queued copy still carried the raw value into the memory queue and backend. The shared helper now rebuilds that mapping with value redacted (same batch redactor) while preserving the protocol fields (version/kind/source/ request_id/option_id/response_kind), merged with the existing additional_kwargs rewrite and rebuilt via model_copy so thread-state messages stay untouched. Regressions pin the redacted value, preserved protocol fields, the untouched original, and the flush-hook path. * chore(config): bump config_version to 47; generate token_secret on upgrade Review round 13 on #5577: existing deployments can legitimately sit at config_version 46 with pii_redaction.enabled: true and no token_secret; the mandatory-secret validator would fail their startup while config.example.yaml still said 46, so the upgrade checker considered them current and emitted no migration warning. - Bump config_version to 47 so the outdated-config warning fires for v46. - Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is enabled and token_secret is missing/blank, generate a random deployment- scoped secret (secrets.token_urlsafe) and persist it, so the upgraded deployment starts under the mandatory validation. - Regressions: a v46 enabled config is flagged outdated against the real example; the upgrade run generates a >= 16-char secret and the result passes AppConfig validation; a disabled config gets no secret. - Also switch the two v41 script tests from a hardcoded "bash" to SCRIPT_BASH/skipif, matching the documented Windows constraint and the neighboring tests (hardcoded bash resolves to the WSL stub there). * chore(helm): sync chart config_version to 47 Review round 14 on #5577: check_config_version.sh fails whenever the chart's embedded config_version lags config.example.yaml, and the chart workflow is triggered by config.example.yaml — so the v47 bump left this head failing that check. Update the values.yaml config block and the README example to 47; scripts/check_config_version.sh now passes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yang rui
|
1da284bb3a
|
feat(prompts): add operator overlays for core system prompts (#5794)
* feat(prompts): add operator overlays for core system prompts * test(prompts): isolate SOUL forwarding test from operator config * fix(prompts): wrap complete subagent system message with overlay * fix(subagents): serialize prompt overlays in durable batches --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Grapette.L
|
f250d11e73
|
fix(memory): reject a Honcho base_url that can never resolve (#5607)
* fix(memory): reject a Honcho base_url that can never resolve
HonchoConfig only did str(...).rstrip('/') on backend_config.base_url, so a
scheme-less or non-http value was accepted at Gateway startup and then made
every memory read return empty and every write drop silently, with only a
per-operation WARNING. httpx turns 'localhost:8000' into the bogus scheme
'localhost:'. The same check already exists in the mem0 and OpenViking
backends, and honcho/README.md plus HonchoMemoryManager.from_config both
promise a bad URL fails fast at startup.
* fix(memory): cover the scheme-case and host-less variants of the guard
- drive the api_key-over-plain-HTTP check off urlsplit(...).scheme: urlsplit
lowercases the scheme while startswith("http://") does not, so
"HTTP://internal:8000" reached httpx over plain HTTP with the key set and
allow_insecure_http still False
- read hostname instead of netloc in the base_url guard, so an authority
without a host ("http://", "http://:8000") is rejected at config time like
the scheme-less shape this PR already removes
- test direct HonchoConfig(base_url=...) construction, the path the guard's
placement in __post_init__ exists to protect
|
||
|
Grapette.L
|
c77e8744fa
|
fix(memory): keep a zero confidence from scoring as the default (#5586)
or 0.5 cannot distinguish an explicit 0.0 from an unset field, so a discounted fact got the default confidence weight in both the FTS5 document and the substring fallback, while the same row indexed through FTS5Retrieval.index_fact kept its zero weight. |
||
|
Keswyk Cheung
|
731e8c6eee
|
feat(memory): bulk-load sample for registered users (#4066)
* feat(memory): bulk-load sample for registered users * fix(memory): harden all-users sample loader --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
tomyan007
|
45411cee69
|
feat(memory): add cognitiveStyle for collaboration preferences (#3182)
* feat(memory): add cognitiveStyle for collaboration preferences Extend cross-session memory with user.cognitiveStyle and cognitive facts, backward-compatible normalization, prompt injection as Thinking Style, and settings UI. Add design notes and unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(memory): normalize legacy memory imports on frontend Add normalizeMemoryPayload aligned with backend normalize_memory_data, use it for Settings import and API reads, and add unit/regression tests. Document schema-evolution steps and sync FACT_EXTRACTION_PROMPT cognitive category. * fix(memory): harden legacy memory normalization * fix(memory): keep import envelope validation strict * fix(memory): keep unknown fields through frontend import normalization normalizeMemoryPayload rebuilt the memory document from a section whitelist, so every key it did not name was dropped. Because readMemoryResponse routes all memory API responses through it, the loss showed up on read, export, and import alike: revision vanished, and any future backend field would silently break lossless export/import. The Gateway models are extra="allow" and deermem normalizes additively, which left the frontend as the only lossy hop in the chain. Spread before overriding at each nesting level so normalization only validates and defaults contract-owned fields, leaving the envelope check as strict as before. Covered by a unit test on the strict import path, two backend tests for extension preservation and caller immutability, and an E2E round-trip that asserts on the captured import request body. * docs(memory): state the normalization invariant that actually holds The docstring cited `display` as a backend-driven field alongside `revision`, but #4546 was closed unmerged and no such field exists in the contract. It also promised lossless export/import, which the Gateway's `extra="ignore"` models preclude for any undeclared field. Replace both with the invariant the code is really defending: the frontend must not be narrower than the Gateway contract, using the declared top-level `revision` as the concrete example. * fix(memory): preserve legacy migration and reject lossy imports * fix(memory): align legacy fact normalization defaults * test(memory): use authenticated app for import regression --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
陈志谦
|
1b2898c367
|
fix(authz): cover create/run/memory/agent routes with permission checks; scope USER.md per user (#4989)
* fix(authz): cover create/run/memory/agent routes with permission checks; scope USER.md per user
Route permissions: the AuthorizationProvider model only applied to routes
carrying @require_permission, so POST /api/threads, /api/threads/search,
POST /api/runs/stream and /api/runs/wait (runs:create), every /api/memory
route and the custom-agent routes ran with authentication only — a
provider configured to deny threads:write/runs:create/... could not
enforce those decisions. Add the missing decorators (threads:write,
threads:read, runs:create, memory:read/write, agents:read/write) and
register the new permission names in authz.Permissions. No owner checks
are added: per-user data scoping stays in the repository layer.
User profile: GET/PUT /api/user-profile read and wrote a single global
{base_dir}/USER.md, so in a multi-user deployment any authenticated user
could overwrite the prompt context injected for everyone else (and read
it). Scope the file to the caller's bucket
({base_dir}/users/{user_id}/USER.md) like user-scoped skills; no other
consumer of the old global path exists in the tree (verified by grep and
the updated tests). test_put_user_profile asserted the wrong effective
user under the autouse conftest user fixture; fixed to test-user-autouse.
* fix(authz): bind positional args in require_permission; migrate legacy USER.md
Review follow-up on #4989 (willem-bd):
- require_permission's wrapper only looked for request in keyword
arguments, so direct positional calls like
create_thread(body, request) made the injected keyword stub collide
with the positional request (TypeError: multiple values). The wrapper
now binds the wrapped signature via inspect.signature().bind() and
honors a positionally-passed request (and thread_id) instead of
assuming kwargs; the test-stub injection only fires when request is
absent everywhere. The two positional callers in
tests/test_threads_router.py pass again (8/8 channel tests).
- migrate_user_isolation.py now claims the legacy global USER.md for
--user-id (default 'default') like the other unowned legacy
artifacts; without it, upgrading installs with an existing profile
would read content: null and later strand the old file beside the
new per-user one. Conflict handling mirrors migrate_memory (rename to
USER.legacy.md).
* style(scripts): keep migrate_user_isolation help within the line budget
* test(authz): give internal-request stubs realistic auth fields
The create_thread permission wrapper added in this PR authenticates
the direct calls in the internal-owner tests; their SimpleNamespace
requests had state.user but no auth_source and no cookies, so
get_current_user_from_request fell through to request.cookies.get and
raised AttributeError (verified introduced by this branch: both tests
pass on the base commit).
The stubs now carry cookies={} and
state.auth_source=AUTH_SOURCE_INTERNAL, which is exactly what
AuthMiddleware stamps on real internal requests, so state.user is
honored without the JWT path. All 80 tests in the file pass.
* fix(pat): keep memory/agent permissions out of PAT scopes by design
The route permissions exist (they guard the memory/agent routers) but PATs
govern the thread/run lifecycle only: _PAT_ROUTE_RULES default-denies those
routers for PAT callers regardless of scopes. The alignment invariant
becomes a subset check plus a pinned exclusion, and pat.py documents that
opening these scopes is a product decision requiring three synchronized
changes.
* test/docs: cover migrate_user_profile; drop the unclaimed USER.md injection wording
Five-scenario test class mirroring the sibling migration steps (move,
conflict-rename, noop, both under dry-run). The GET/PUT descriptions no
longer say USER.md is 'injected into agents' — nothing at this head consumes
it for prompts; the routes are storage/retrieval only. The memory AGENTS.md
migration pointer now enumerates skills/ and the global USER.md.
* docs(paths): align the USER.md layout comment with the injection disclaimer
* test(authz): align effective permission contracts
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Grapette.L
|
479d2f10c8
|
fix(memory): report malformed backend_config values by key name (#5555)
* fix(memory): report non-mapping Honcho backend_config values as ValueError
failure_policy, workspace_overrides and user_peer_overrides were read with a
falsy-only `or {}` fallback, so a truthy non-mapping (a bare string, a YAML
list) reached .get/.items and escaped as a bare AttributeError from inside
backend construction. Route all three through one _mapping helper that keeps
falsy values meaning "unset" and names the offending key as a ValueError, the
posture the mem0 and OpenViking backends already take.
* fix(memory): name the Honcho numeric knobs that cannot be cast
timeout_seconds / connect_timeout_seconds / message_char_limit /
max_injection_chars still reached float() / int() with a YAML null or a
mapping, so the operator got a TypeError naming neither the key nor the
config file. Narrow all four through one _number helper that keeps falsy
values meaning "unset" the way the sibling api_key / storage_path scalars
already do, and reports a value that cannot be cast as a ValueError on the
key. Numeric strings keep parsing, since that is what float/int accept.
* fix(memory): report non-numeric backend_config knobs by name in mem0 and OpenViking
mem0 casts top_k, score_threshold, max_injection_chars and timeout_seconds, and
OpenViking casts timeout_seconds, max_seen_message_ids, retrieval.top_k and
retrieval.max_injection_chars, straight through int()/float(). A knob written
without a value in YAML therefore escapes as "TypeError: int() argument must be
a string..." from inside backend construction, naming neither which knob nor
which file is wrong, and a non-numeric value escapes as the equally anonymous
"could not convert string to float".
Both now resolve numeric knobs through a helper that keeps a value-less key at
its default, the way the same dicts already treat failure_policy and
allow_insecure_http, and turns an uncastable value into a ValueError that names
the knob. OpenViking's score_threshold keeps None as a meaningful value rather
than a default to fall back to.
|
||
|
Kris
|
208ba7cc65
|
feat(memory): opt-in tolerant MarkdownMemoryStorage (#3124) (#5545)
* feat(memory): add opt-in tolerant MarkdownMemoryStorage (fixes #3124) User-memory summary load is now tolerant of corrupt/partially written files: a Markdown summary (fenced `memory-json` block, best-effort structured fallback) or JSON is accepted, and an unrecoverable file recovers to an empty memory instead of raising MemoryStorageCorruption and taking down the agent. On-disk JSON format and the JSON UI are unchanged, so enabling `memory.storage_class: markdown` is fully opt-in and cannot break existing deployments. Co-authored-by: WorkBuddy <noreply@workbuddy.ai> * fix(markdown-memory): address review - schema-safe loader, greedy fence, quarantine, CI tests - drop lossy structured Markdown fallback (invalid-shape crash on load/save) - greedy fence match: ``` inside remembered strings round-trips losslessly - quarantine unreadable summary before returning None (no silent erase) - remove dead _render_memory_markdown (deferred to write-path change) - move tests to backend/tests/ so CI runs them; update storage_class docs * fix(markdown-memory): address review - schema-safe loader, greedy fence, quarantine, CI tests - drop lossy structured Markdown fallback (invalid-shape crash on load/save) - greedy fence match: ``` inside remembered strings round-trips losslessly - quarantine unreadable summary before returning None (no silent erase) - remove dead _render_memory_markdown (deferred to write-path change) - move tests to backend/tests/ so CI runs them; update storage_class docs * fix(markdown-memory): address review - schema-safe loader, greedy fence, quarantine, CI tests - drop lossy structured Markdown fallback (invalid-shape crash on load/save) - greedy fence match: ``` inside remembered strings round-trips losslessly - quarantine unreadable summary before returning None (no silent erase) - remove dead _render_memory_markdown (deferred to write-path change) - move tests to backend/tests/ so CI runs them; update storage_class docs * fix(markdown-memory): address review - schema-safe loader, greedy fence, quarantine, CI tests - drop lossy structured Markdown fallback (invalid-shape crash on load/save) - greedy fence match: ``` inside remembered strings round-trips losslessly - quarantine unreadable summary before returning None (no silent erase) - remove dead _render_memory_markdown (deferred to write-path change) - move tests to backend/tests/ so CI runs them; update storage_class docs * fix(markdown-memory): address review - schema-safe loader, greedy fence, quarantine, CI tests - drop lossy structured Markdown fallback (invalid-shape crash on load/save) - greedy fence match: ``` inside remembered strings round-trips losslessly - quarantine unreadable summary before returning None (no silent erase) - remove dead _render_memory_markdown (deferred to write-path change) - move tests to backend/tests/ so CI runs them; update storage_class docs * fix(memory): correct tolerant Markdown parsing and regression tests --------- Co-authored-by: WorkBuddy <noreply@workbuddy.ai> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
wd_pan
|
cc27730348
|
feat(memory): add opt-in relevance-aware retrieval ranking (#5251)
* feat(memory): add opt-in relevance-aware retrieval ranking Add a deterministic, network-free lexical relevance strategy for DeerMem (issue #4495): memory_search ranks every fact in scope by idf-weighted token overlap combined with confidence, with optional greedy-MMR diversity against near-duplicate facts; prompt injection ranks facts against the current-turn query threaded from DynamicContextMiddleware through the new optional `query` keyword on MemoryManager.get_context/aget_context. Defaults preserve the legacy confidence-only behavior exactly; no prompt, storage-format, or vector/embedding-dependency changes. Refs #4495 Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): bound relevance retrieval and apply review feedback Bound tokenization and index shared stems, preserve mixed CJK tokens, warm jieba, and align missing confidence with legacy injection. Cache MMR token sets and stop selection at result or injection budgets. Document retrieval-adapter precedence and add regression coverage. Refs #4495. Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): preserve backend compatibility and normalize relevance Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): omit absent query hints and share injection IDF Signed-off-by: pwd11 <fvdsrc@163.com> * test(memory): retain timeout mock until injection worker exits Signed-off-by: pwd11 <fvdsrc@163.com> * docs(agents): drop root guidance compaction Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): validate token prefixes and preserve upload queries --------- Signed-off-by: pwd11 <fvdsrc@163.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Beautyl0ve
|
80f13935c2
|
feat(agents): allow custom agents to disable memory (#5167)
* feat(agents): allow custom agents to disable memory Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(agents): honor memory opt-out during compaction Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(runtime): preserve agent binding across state rewrites * fix(client): apply named-agent memory policy * fix(agents): address memory policy review feedback Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> * fix(agents): address remaining memory opt-out reviews Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> --------- Signed-off-by: Beautyl0ve <74452755+Beautyl0ve@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
wd_pan
|
5d86ce345c
|
feat(memory): add deterministic near-duplicate fact gate (#5254)
* feat(memory): add deterministic near-duplicate fact gate Add an opt-in write-side gate for DeerMem (issue #5252): a proposed NEW fact whose bounded token-Jaccard similarity to an existing fact in the same user/agent scope AND category reaches fact_dedup_similarity_threshold merges into that fact instead of being appended — the existing id/content/createdAt are kept, confidence is raised to the maximum, and the source is refreshed, with one facts_merged_dedup metric increment recording the merge. Defaults preserve the legacy behavior exactly; targeted updates by fact id and the exact-content key check are untouched. Companion write-side step to the read-side relevance/diversity work in #5251. Refs #5252 Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): preserve corrections during fact deduplication Signed-off-by: pwd11 <fvdsrc@163.com> --------- Signed-off-by: pwd11 <fvdsrc@163.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hao Zhe
|
ec274bdedb
|
fix(memory): enforce backend read failure policy (#4726)
* fix(memory): enforce backend read failure policy * fix(memory): harden failure policy handling * fix(memory): narrow strict read handling * fix(memory): keep timeout handling off saturated executor * fix(memory): preserve legacy fail-closed timeouts |
||
|
luo jiyin
|
e5977320a0
|
feat(sandbox): surface structured mount upload result on E2B sandbox (#4884)
* feat(e2b-sandbox): make mount upload deadline configurable Replace the hardcoded 120-second mount upload deadline with a configurable `mount_upload_deadline_seconds` key read from SandboxConfig (extra=allow). The value is validated: zero and negative inputs are clamped to 1 second. Omitting the key preserves the existing 120-second default. This addresses the follow-up from PR #4842 review: operators with large mounts or slow networks can now size the deadline to their deployment without changing code. * fix(e2b-sandbox): address review feedback on configurable deadline - Remove import-time default capture from _mount_deadline_reason() and _MountUploadBudget.deadline_seconds to prevent silent drift. - Add warning log when mount_upload_deadline_seconds is clamped to 1 (was silent before). - Update AGENTS.md E2B Mount Uploads section: deadline is now configurable, not fixed 120. - Add mount_upload_deadline_seconds to YAML examples in provider docstring and __init__.py. - Add config-path test that exercises SandboxConfig -> _load_config -> _apply_mounts end-to-end. * feat(e2b-sandbox): surface structured mount upload result on sandbox Introduce MountUploadResult dataclass and attach it to E2BSandbox.mount_upload_result after creation. This makes mount truncation observable in code without re-parsing Gateway logs. _apply_mounts() now returns MountUploadResult with truncated, reason, and upload totals. _create_sandbox() captures the result, stores it on the sandbox instance, and records it in a provider-level map so the result survives warm-pool reclaim and reconnect. MountUploadResult.truncated is True only when the upload pass was stopped early by a resource limit (deadline, file count cap, or byte budget). Individual mount failures (missing host path, SDK errors) are logged but do NOT set truncated. Tests cover: success totals, deadline truncation, file-count truncation, byte-budget truncation, non-limit failure not reported as truncation, missing host path not reported as truncation, create→sandbox wiring, and create→release→warm-pool→acquire result preservation. * fix(e2b-sandbox-provider): fix _mount_results lifecycle leak and review findings - Add _forget_mount_result() helper and call it at all terminal sandbox paths: _reuse_in_process_sandbox dead-evict, _reclaim_warm_pool_sandbox reconnect/dead/bootstrap/ownership/shutdown failure branches, _forget_local_sandbox, _kill_and_close. Prevents unbounded dict growth over a long-running Gateway process. - Make MountUploadResult @dataclass(frozen=True) to prevent silent mutation of the shared reference between provider map and sandbox attribute. - Move _mount_results insert under self._lock in _create_sandbox to match the read discipline in _register_connected_sandbox. - Guard _resolve_mount_upload_deadline against None (YAML explicit null) to avoid int(None) TypeError. - Add 5 regression tests covering each bypass path and the frozen invariant. * fix(e2b-sandbox-provider): add _forget_mount_result to _evict_oldest_warm branches Add _forget_mount_result() calls to all four terminal exit paths in the E2B _evict_oldest_warm override (reconnect failure, already-gone, kill failure, kill success). The peer-owned path already cleans up via _forget_local_sandbox. Add test_evict_oldest_warm_cleans_mount_result to pin the kill-success branch. * docs: reduce agent guidance size --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
SPEC
|
822c7bca4b
|
fix(memory): cancel buffered extraction when agent is deleted or cleared (#5123)
* fix(memory): cancel buffered extraction when agent is deleted or cleared * fix(memory): cancel buffered work before agent delete Address review: cancel before/after delete to close the rmtree race, scope user_id=None cancels to the legacy root only, and import memory helpers at module scope. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): close remaining cancel races from review Post-clear cancel, legacy-only all_agents scope, always cancel even when memory is disabled, and fold cancel+delete into one offloaded thread. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * docs(memory): align cancel_by_agent None-scope with legacy root Document that user_id=None cancels only the legacy no-user bucket, matching clear/storage semantics, not the whole process-local queue. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * test(memory): fix cancel_by_agent docstring regression assertion Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): address final cancel review nits Type the delete helper with AgentStore, replace docstring pinning with a kwargs mapping test, and document scoped cancel + residual window in AGENTS.md. Signed-off-by: SPEC <zt1y17@soton.ac.uk> * fix(memory): resolve agent store inside delete worker thread get_agent_store() does blocking config/FS work; keep it off the event loop so test_delete_agent_does_not_block_event_loop and backend-blocking-io CI pass. Signed-off-by: SPEC <zt1y17@soton.ac.uk> --------- Signed-off-by: SPEC <zt1y17@soton.ac.uk> |
||
|
Hyeonsang Cho
|
9146bfa03d
|
feature(gateway): issue request trace ids unconditionally (#5119)
* refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
betterkite
|
b552b5015c
|
fix(messages): drop legacy <uploaded_files> tag handling (#4826)
* fix(messages): drop legacy <uploaded_files> tag handling (#4212) PR #4174 unified upload-context injection on <current_uploads> (IM and web both flow through UploadsMiddleware), and #4632 documented the current path. This removes the remaining backward-compat parsing of the pre-#4174 <uploaded_files> tag, the final cleanup item tracked by the issue: - deermem: only <current_uploads> is stripped from human turns before memory persistence, and the upload-sentence scrubber drops the legacy tag alternative. - mem0: the mirrored message filter recognises only <current_uploads>. - InputSanitizationMiddleware: remove the legacy tag from the blocked-tag denylist (it existed only because deermem parsed the old tag). - frontend: stripUploadedFilesTag / stripInternalMarkers / parseUploadedFiles and the message-list fallback parse only <current_uploads>; demo thread fixtures are migrated to the current tag. Scope decision: a <uploaded_files> block in pre-#4174 history is now treated as ordinary user content (pinned by tests in both layers) instead of being silently dropped or stripped. * style: apply prettier formatting to stripUploadedFilesTag * fix(uploads): keep legacy <uploaded_files> stripping for display/export only Addresses review feedback on #4826: removing the legacy tag from the frontend display layer made pre-#4174 threads render raw <uploaded_files> XML (with server-side upload paths) in chat, copy data, and JSON exports. The backend cleanup stands — memory pipelines and the sanitization denylist treat only <current_uploads> as an internal marker. The frontend keeps the legacy spelling in its display/export-only utilities (stripUploadedFilesTag / INTERNAL_MARKER_TAGS / parseUploadedFiles and the message-list fallback) so old history renders cleanly without leaking internal paths, while the memory/sanitization scope-decision tests remain unchanged. Frontend tests now pin both spellings: <current_uploads> and legacy <uploaded_files> are stripped from copy data, markdown leak-stripping, and JSON exports. * docs(ui): record accepted display-spoof tradeoff for legacy upload tag Review note (willem-bd): since <uploaded_files> is off the sanitization denylist, a live user can type the legacy spelling and fabricate file chips / hide their own message text in display. Display-only and self-inflicted with no backend semantics, so it is accepted for now; documented at both the message-list fallback and stripUploadedFilesTag. Age-gating the legacy spelling remains a possible follow-up. --------- Co-authored-by: betterkite <313258397+betterkite@users.noreply.github.com> |
||
|
starslittle
|
f0276c9f5a
|
fix(memory): validate Honcho timeout and character limits (#4783)
* fix(memory): validate Honcho timeout and character limits * fix(memory): enforce HonchoConfig invariants |