github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-11-01 08:00:40 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/backend/packages/harness/deerflow/storage/AGENTS.md
zeng-bohan 82ac1612e3
feat(storage): add a content-addressed blob store contract (issue #4189 item 2) (#5361) 
* feat(storage): add a content-addressed blob store contract (#4189 item 2)

Addresses the review on the previous head:

* the retention/dedup claim in the doc now states the deletion rule (liveness
  from the durable references) instead of promising refcount-free deletion, and
  the sidecar records the first writer as advisory provenance only;
* one validate_blob_kind() owns the kind grammar, so the message a producer sees
  is the pattern that is enforced;
* adds the missing storage/AGENTS.md (registry entry included) that both
  docstrings already pointed at;
* drops the dead _import_backend_module() by routing the folder scan through it;
* exports BlobNotConfiguredError from the package;
* renames _KIND_MAX_BYTES to _MAX_BLOB_BYTES (it caps one put, not a kind);
* delete failures raise the neutral BlobStoreError instead of BlobReadError.

Rebased onto current main so the guidance-budget test matches upstream.

Signed-off-by: zeng-bohan <zengbh1@gmail.com>

* fix(storage): restore the config module and backend packages dropped by the squash

The force-push to e72b435 silently lost four items the surviving code
imports, breaking `import deerflow.storage` and the test suite at
collection:

- deerflow/config/blob_storage_config.py (BlobStorageConfig +
  get/set_blob_storage_config + load_blob_storage_config_from_dict,
  mirroring memory_config's host-shared schema and hot-reload pattern)
- app_config.py wiring: the blob_storage AppConfig field, the import,
  and the _apply_singleton_configs -> load_blob_storage_config_from_dict
  call (memory stayed wired; blob did not)
- storage/backends/__init__.py (drop-in contract docstring)
- storage/backends/local_fs/__init__.py (LocalFsBlobStore re-export +
  STORE_CLASS for _scan_backends)

Verified: pytest tests/test_blob_store.py tests/test_app_config_reload.py
tests/test_app_config_name_indexes.py -> 67 passed; ruff format/check
clean on all four files.

* fix(storage): name the failing digest check; document the 64 MiB per-put cap

- get_bytes: split the corruption message into size mismatch (truncation /
  partial write) vs digest mismatch (same-size bit rot, reporting what the
  content hashes to vs what the ref names) — the combined message reported
  'got 1024 bytes, expected 1024' on same-size corruption and sent the
  debugger in the wrong direction.
- docs/blob-storage.md + storage/AGENTS.md: state the _MAX_BLOB_BYTES cap
  (64 MiB) and that it is backend-level defense-in-depth rather than part
  of the BlobStore contract, so the externalized-tool-results follow-up
  sees the limit before the first production put fails.

* test(storage): pin both get_bytes corruption branches separately

The pre-split test corrupted the blob with different-size bytes, so only
the size-mismatch branch ever ran and neither new message was asserted.
Now: different-size corruption asserts 'size mismatch', and a same-size
reversal (bit rot) asserts 'digest mismatch' — the branch the split was
actually about.

* docs(config): document blob_storage in config.example.yaml

Add the blob_storage section (enabled/backend/backend_config) mirroring the
memory section's shape, matching the Configuration section of
backend/docs/blob-storage.md, and bump config_version 46 -> 47 per the
example-file convention for schema additions. Default state is unchanged:
enabled: false, purely opt-in.

* fix(helm): align chart config version with example

* fix(storage): refresh blob store on config changes

---------

Signed-off-by: zeng-bohan <zengbh1@gmail.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
2026-09-24 09:35:03 +08:00

4.6 KiB
Raw Permalink Blame History

Blob Store (packages/harness/deerflow/storage/)

What it owns: a content-addressed store for bytes that must resolve on every instance, not just the one that wrote them (issue #4189, item 2). The address is the SHA-256 of the content, so writes are idempotent, deduplication is free, and a reader can verify what it got back. Contract: storage/contract.py (BlobRef, BlobStore, the BlobStoreError family, validate_blob_kind). Operator-facing behaviour and the producer migration plan: docs/blob-storage.md. Factory: storage/manager.py.

Disabled by default: blob_storage.enabled is false, and no producer has been migrated, so an untouched deployment behaves exactly as before. Producers call get_blob_store_if_enabled() and keep their existing local-path code for the None case; only a caller that genuinely requires the store uses get_blob_store(), which raises BlobNotConfiguredError when it is disabled. Both accessors are exported from this package — prefer from deerflow.storage import ... over reaching into submodules. The factory compares the effective backend and backend config before reusing its cached store, so hot-reloaded blob_storage edits take effect on new accesses. Replaced stores stay open for callers that already hold them and are closed by reset_blob_store(); changing a root requires preserving existing blob refs and coordinating deployment instances.

Layout (local_fs, the default)::

<root>/<kind>/<sha256[:2]>/<sha256>          the bytes
<root>/<kind>/<sha256[:2]>/<sha256>.json     sidecar metadata

The sidecar holds what the bytes cannot say (content_type, writer_thread_id, created_at). It is not a reference count and reads must not depend on it: losing it cannot make content unreadable.

Adding a backend — the memory backends' drop-in contract, folder name == backend name == blob_storage.backend value:

  1. Create storage/backends/<name>/ with an __init__.py that exposes STORE_CLASS = <YourStore>; the factory discovers backends by scanning that folder, so no registry edit is needed.
  2. Subclass BlobStore and implement from_config, put_bytes, get_bytes; exists / delete / close have working defaults (delete raises so a backend that cannot delete says so). Verify the digest on read — a store that silently returns wrong bytes is indistinguishable from a corrupt checkpoint.
  3. Keep the portability rule: a backend talks to the host through the method arguments and its backend_config dict only. The single deerflow import a backend folder needs is the contract line importing BlobStore, so a third-party backend can live outside this tree and be selected with a dotted path (pkg.mod.Class or pkg.mod:Class).
  4. Reject an unusable config by raising; a store that silently starts on a different root or backend strands previously written content.
  5. Add the backend's semantics tests to backend/tests/test_blob_store.py; the suite runs every backend-agnostic scenario against a temporary root.

Durability rules that reviewers have enforced, worth keeping intact:

  • Validate the kind through validate_blob_kind instead of re-implementing the grammar; it is also a path segment in local_fs, so traversal and separator surprises are ruled out once, at the contract level.
  • Publish atomically and tolerate a concurrent writer: two gateway instances may put the same content at the same time (see LocalFsBlobStore._publish, which handles the Windows os.replace/WinError 5 case by recognising that the destination already holds this content).
  • Fail fast on an unresolvable backend: _resolve_store_class raises rather than substituting a default, because blobs are persistent state.
  • Reject oversized puts early: local_fs enforces _MAX_BLOB_BYTES (64 MiB) per put as a backend-level defense-in-depth limit — not part of the BlobStore contract, so a different backend sets its own policy, and the externalized-tool-results follow-up must stream or split payloads before put_bytes.

Garbage collection is not this module's job. A blob row may be referenced by several threads once identical content dedupes, so a sweep must establish liveness from the durable references (the checkpoint rows that name the blob) and never from the sidecar's writer_thread_id. The exact rule, and how it composes with the checkpoint retention contract (#5255), is stated in docs/blob-storage.md.

Entry points: packages/harness/deerflow/storage/contract.py, storage/manager.py, storage/backends/local_fs/local_fs_store.py, config/blob_storage_config.py; tests in backend/tests/test_blob_store.py.