e5977320a0
* feat(e2b-sandbox): make mount upload deadline configurable Replace the hardcoded 120-second mount upload deadline with a configurable `mount_upload_deadline_seconds` key read from SandboxConfig (extra=allow). The value is validated: zero and negative inputs are clamped to 1 second. Omitting the key preserves the existing 120-second default. This addresses the follow-up from PR #4842 review: operators with large mounts or slow networks can now size the deadline to their deployment without changing code. * fix(e2b-sandbox): address review feedback on configurable deadline - Remove import-time default capture from _mount_deadline_reason() and _MountUploadBudget.deadline_seconds to prevent silent drift. - Add warning log when mount_upload_deadline_seconds is clamped to 1 (was silent before). - Update AGENTS.md E2B Mount Uploads section: deadline is now configurable, not fixed 120. - Add mount_upload_deadline_seconds to YAML examples in provider docstring and __init__.py. - Add config-path test that exercises SandboxConfig -> _load_config -> _apply_mounts end-to-end. * feat(e2b-sandbox): surface structured mount upload result on sandbox Introduce MountUploadResult dataclass and attach it to E2BSandbox.mount_upload_result after creation. This makes mount truncation observable in code without re-parsing Gateway logs. _apply_mounts() now returns MountUploadResult with truncated, reason, and upload totals. _create_sandbox() captures the result, stores it on the sandbox instance, and records it in a provider-level map so the result survives warm-pool reclaim and reconnect. MountUploadResult.truncated is True only when the upload pass was stopped early by a resource limit (deadline, file count cap, or byte budget). Individual mount failures (missing host path, SDK errors) are logged but do NOT set truncated. Tests cover: success totals, deadline truncation, file-count truncation, byte-budget truncation, non-limit failure not reported as truncation, missing host path not reported as truncation, create→sandbox wiring, and create→release→warm-pool→acquire result preservation. * fix(e2b-sandbox-provider): fix _mount_results lifecycle leak and review findings - Add _forget_mount_result() helper and call it at all terminal sandbox paths: _reuse_in_process_sandbox dead-evict, _reclaim_warm_pool_sandbox reconnect/dead/bootstrap/ownership/shutdown failure branches, _forget_local_sandbox, _kill_and_close. Prevents unbounded dict growth over a long-running Gateway process. - Make MountUploadResult @dataclass(frozen=True) to prevent silent mutation of the shared reference between provider map and sandbox attribute. - Move _mount_results insert under self._lock in _create_sandbox to match the read discipline in _register_connected_sandbox. - Guard _resolve_mount_upload_deadline against None (YAML explicit null) to avoid int(None) TypeError. - Add 5 regression tests covering each bypass path and the frozen invariant. * fix(e2b-sandbox-provider): add _forget_mount_result to _evict_oldest_warm branches Add _forget_mount_result() calls to all four terminal exit paths in the E2B _evict_oldest_warm override (reconnect failure, already-gone, kill failure, kill success). The peer-owned path already cleans up via _forget_local_sandbox. Add test_evict_oldest_warm_cleans_mount_result to pin the kill-success branch. * docs: reduce agent guidance size --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
12 KiB
Memory System
This directory owns memory capture, storage, retrieval, prompt injection, and model-driven memory tools.
Main components
manager.pydefines the backend-neutralMemoryManagercontract.agents/middlewares/memory_middleware.pyqueues filtered conversations for passive capture.summarization_hook.pyconnects memory work to the summarization lifecycle.tools.pyprovidesmemory_search,memory_add,memory_update, andmemory_delete.backends/deermem/contains the default local backend.backends/mem0/,backends/openviking/, andbackends/honcho/contain optional adapters.
cancel_by_agent cancels only pending debounce contexts in one user scope.
user_id=None selects only the legacy no-user root.
agent_name=None selects all agent buckets in that user scope.
It does not interrupt a context after _process_queue removes it from _items.
Broader cancellation must iterate known user scopes.
Focused updater tests live in backend/tests/test_memory_updater.py.
Backend-specific tests use backend/tests/test_<backend>_memory_backend.py.
Identity and isolation
Resolve users with resolve_runtime_user_id(runtime) in middleware and tools.
This keeps Gateway and standalone LangGraph runs in the same user scope.
Server-owned langgraph_auth_user_id takes precedence over ordinary client identity.
Lead-agent construction normalizes it with make_safe_user_id.
Memory, custom agents, user skills, skill policy, and prompt assembly reuse that identity.
Gateway removes client-supplied langgraph_auth_user and langgraph_auth_user_id before graph construction.
Gateway memory routes use _resolve_memory_user_id(request).
Trusted IM requests can act for the connection owner.
Other requests use get_effective_user_id().
Only AuthMiddleware can authorize the internal owner header.
No-auth mode uses DEFAULT_USER_ID, which is "default".
An absolute storage_path opts out of the default per-user root.
DeerMem uses this layout:
{base_dir}/users/{user_id}/memory.json
{base_dir}/users/{user_id}/agents/{agent_name}/facts/{sha256-prefix}/{fact-id}.md
memory.json stores only shared summaries, revision data, and timestamps.
It never stores facts or a fact index.
Each Markdown file stores one fact with YAML front matter.
Custom agent files share the per-user agent directory. The legacy shared agent layout is read-only fallback data.
DeerMem maps a missing agent name to __default__.
That name is reserved and cannot identify a custom agent.
Public agent names use lowercase canonical form.
Operating modes
memory.mode: middleware is the default passive mode.
MemoryMiddleware queues filtered user and final assistant messages.
It captures user_id when it enqueues work.
This identity survives the background timer boundary.
memory.mode: tool registers the four memory tools.
The model chooses when to search or change facts.
Tool mode still uses MemoryMiddleware for passive writes on supported remote backends.
Middleware injection includes shared summaries and the selected agent's facts.
Tool-mode injection includes only shared summaries.
Tool mode leaves agent facts behind memory_search.
memory.injection_enabled: false disables the complete injected block.
DeerMem storage contract
FileMemoryStorage owns canonical storage and the retrieval adapter.
Do not reach into its private adapter state from higher layers.
The repository supports fact CRUD, summary updates, migration, search, and index lifecycle operations.
Targeted writes change only the selected Markdown files.
Whole-document load and save remain compatibility operations.
apply_changes() returns complete: false with fact deltas.
It never labels a partial cache as a complete memory document.
Public callers reload only when their response contract requires a complete document.
Writes use a user lock, shared revision, fact revisions, and a recovery journal. Point operations can rebase only when all original fact preconditions still hold. Snapshot operations must reload and recompute after a manifest conflict. Use the typed conflict classes instead of matching exception text.
The weak lock cache must not retain inactive user scopes.
Cache validation uses the manifest metadata and persisted revision.
Out-of-band Markdown edits require reload().
POSIX atomic replacement must sync the parent directory.
DeerMem converts storage conflicts to the public MemoryManager error types.
The Gateway maps conflicts to HTTP 409.
The Gateway maps storage corruption to a stable HTTP 500 response.
Migration
A normal default-manager read migrates legacy facts into __default__.
It adopts an old lead-agent bucket only when no custom-agent config exists.
Unexpected files stop migration and remain on disk.
The v1-to-v2 migration is one-way during application operation.
Operators must stop DeerFlow and snapshot the storage root before migration.
Every destructive migration first writes a verified {manifest_filename}.v1.bak file.
Missing or mismatched backups abort migration without changing v1 data.
Delete legacy agent JSON only after safe summary adoption or equality checks.
Summary conflicts keep the source file and return an error.
Run the proactive migration from backend/:
PYTHONPATH=. python scripts/migrate_memory_markdown.py --all-users --dry-run
Remove --dry-run to migrate.
Use repeated --user-id options for exact source identities.
Use --storage-path for a non-default DeerMem root.
The command is idempotent and continues after per-user failures.
It returns a nonzero status when any user fails.
The older isolation migration remains available:
PYTHONPATH=. python scripts/migrate_user_isolation.py --dry-run
Retrieval
retrieval_adapter owns indexing and retrieval.
DeerMem selects persistent SQLite FTS5 by default.
An empty value selects the substring fallback.
SQLite index data lives below .retrieval/ and remains rebuildable.
Chinese tokenization uses jieba only with the memory-zh extra.
Malformed facts are logged and skipped during rebuild.
A fatal rebuild failure keeps lazy retry active.
A corrupt persistent database is deleted and recreated once.
Storage sends adapter updates after it releases durable locks. Adapter failures mark the scope dirty. Search then uses canonical substring matching until rebuild succeeds.
Gateway startup schedules DeerMem.warm_retrieval() without delaying readiness.
The first search can rebuild its exact scope.
Shutdown waits one second for retrieval warm-up.
It reserves the full configured timeout for canonical memory flush.
The Gateway closes the derived SQLite connection after that flush.
Extraction safety
Extraction labels proposals with scope, durability, and authority.
Automatic writes accept only user-scoped, durable, descriptive facts.
Summary prose must be user-scoped and descriptive.
Missing labels reject that item without stopping unrelated updates.
Contradiction removals include id, scope, reason, and optional replacementFactIndex.
Task-scoped and project-scoped removals fail closed.
A paired removal requires its replacement to pass every write gate.
Tool-mode CRUD does not use the extraction gate.
Custom prompt directories must include the same classification fields. Old templates cause extraction writes to fail closed. The rejection counter and high-rejection warning expose this condition.
Capacity and review
All automatic, manual, tool, and import paths use deermem/core/eviction.py.
confidence is the default capacity policy.
hybrid-v1 is opt-in and uses confidence, confirmation freshness, and access heat.
Shadow mode records disagreement while enforcing confidence-only selection.
Only deterministic message processing can confirm a fact.
The updater's factsToReinforce output supplies only the fact binding.
The deterministic gate matches a human message in the last six filtered batch messages.
It does not require a separate signal-to-fact match.
Search increments access heat only for facts it returns.
Prompt injection and get_context() do not increment access heat.
Usage and audit sidecars live below the agent .metadata/ directory.
They must not change canonical Markdown timestamps or revisions.
Write audits only after canonical persistence succeeds.
User delete and clear operations must remove matching sidecar data.
Staleness review reuses the regular updater call. It can keep, remove, or extend eligible aged facts. Protected categories and non-aged facts cannot become removal targets. Apply the per-cycle removal cap after candidate validation. Do not extend a fact proposed for removal, even when the cap keeps that fact. Extension bounds must prevent date overflow.
Consolidation also reuses the regular updater call. Source facts must exist and cannot overlap across groups. Enforce the source-count and confidence limits at apply time. Use the newest source creation time for the merged fact. Use the earliest source review deadline for its next review.
Remote backends
OpenViking uses the maintained langchain-openviking package.
Keep it in middleware mode.
One API key is bound to one configured DeerFlow owner.
Reject another owner before remote access.
DeerFlow owns capture timing, the recall query, and the transcript cursor.
The package owns transport, message conversion, batching, and Session commits.
One DeerFlow thread maps to one stable OpenViking Session.
Store bounded hash-only cursors below {storage_path}/openviking/sessions/.
Async OpenViking entry points must offload synchronous SDK and file operations.
Shutdown must drain active work before closing the recorder client.
Pass an empty extra_headers mapping to prevent configuration-added transport headers.
Do not add embedded OpenViking imports, root-key access, or trusted identity headers.
Honcho is a remote HTTP adapter for user-model memory.
It creates one workspace per resolved user_id.
A missing user fails closed to no memory.
Its async methods offload synchronous HTTP work with asyncio.to_thread.
The default read failure policy logs and returns no results.
failure_policy.read: fail_closed rethrows recall failures.
Honcho configuration rejects non-finite or non-positive timeouts. It also rejects non-positive character budgets during construction.
Run identity and token counting
Each run hashes its effective hidden memory block.
The run records one context:memory event with content_sha256.
The full memory text stays in checkpoint state.
Only current DynamicContextMiddleware output can establish first-run memory identity.
Checkpoint reuse requires the block to exist before the run.
Gateway input handling removes forged dynamic-context markers.
prompt.py::_count_tokens controls the injection budget.
Default tiktoken mode loads and caches its encoding lazily.
A failed load uses character estimation for a 600-second cooldown.
Concurrent callers use character estimation while one load is active.
Set memory.token_counting: char to prevent network access.
Configuration
The schema lives in deerflow/config/memory_config.py.
Do not duplicate its complete field list here.
Keep these cross-component constraints in sync:
- The shutdown flush budget is between 1 and 300 seconds.
- The pod grace period must include retrieval wait, flush time, and shutdown margin.
retrieval_adapterselects FTS5, a custom factory, or the empty fallback.- Eviction weights must total
1.0. watermark_max_keys: 0makes the conversation watermark cache unbounded.- A dropped watermark can re-extract one batch on the next turn.