mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 13:40:37 +00:00
82ac1612e3
* feat(storage): add a content-addressed blob store contract (#4189 item 2) Addresses the review on the previous head: * the retention/dedup claim in the doc now states the deletion rule (liveness from the durable references) instead of promising refcount-free deletion, and the sidecar records the first writer as advisory provenance only; * one validate_blob_kind() owns the kind grammar, so the message a producer sees is the pattern that is enforced; * adds the missing storage/AGENTS.md (registry entry included) that both docstrings already pointed at; * drops the dead _import_backend_module() by routing the folder scan through it; * exports BlobNotConfiguredError from the package; * renames _KIND_MAX_BYTES to _MAX_BLOB_BYTES (it caps one put, not a kind); * delete failures raise the neutral BlobStoreError instead of BlobReadError. Rebased onto current main so the guidance-budget test matches upstream. Signed-off-by: zeng-bohan <zengbh1@gmail.com> * fix(storage): restore the config module and backend packages dropped by the squash The force-push to e72b435 silently lost four items the surviving code imports, breaking `import deerflow.storage` and the test suite at collection: - deerflow/config/blob_storage_config.py (BlobStorageConfig + get/set_blob_storage_config + load_blob_storage_config_from_dict, mirroring memory_config's host-shared schema and hot-reload pattern) - app_config.py wiring: the blob_storage AppConfig field, the import, and the _apply_singleton_configs -> load_blob_storage_config_from_dict call (memory stayed wired; blob did not) - storage/backends/__init__.py (drop-in contract docstring) - storage/backends/local_fs/__init__.py (LocalFsBlobStore re-export + STORE_CLASS for _scan_backends) Verified: pytest tests/test_blob_store.py tests/test_app_config_reload.py tests/test_app_config_name_indexes.py -> 67 passed; ruff format/check clean on all four files. * fix(storage): name the failing digest check; document the 64 MiB per-put cap - get_bytes: split the corruption message into size mismatch (truncation / partial write) vs digest mismatch (same-size bit rot, reporting what the content hashes to vs what the ref names) — the combined message reported 'got 1024 bytes, expected 1024' on same-size corruption and sent the debugger in the wrong direction. - docs/blob-storage.md + storage/AGENTS.md: state the _MAX_BLOB_BYTES cap (64 MiB) and that it is backend-level defense-in-depth rather than part of the BlobStore contract, so the externalized-tool-results follow-up sees the limit before the first production put fails. * test(storage): pin both get_bytes corruption branches separately The pre-split test corrupted the blob with different-size bytes, so only the size-mismatch branch ever ran and neither new message was asserted. Now: different-size corruption asserts 'size mismatch', and a same-size reversal (bit rot) asserts 'digest mismatch' — the branch the split was actually about. * docs(config): document blob_storage in config.example.yaml Add the blob_storage section (enabled/backend/backend_config) mirroring the memory section's shape, matching the Configuration section of backend/docs/blob-storage.md, and bump config_version 46 -> 47 per the example-file convention for schema additions. Default state is unchanged: enabled: false, purely opt-in. * fix(helm): align chart config version with example * fix(storage): refresh blob store on config changes --------- Signed-off-by: zeng-bohan <zengbh1@gmail.com> Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
35 lines
885 B
Python
35 lines
885 B
Python
"""Content-addressed blob store (issue #4189, item 2).
|
|
|
|
See ``contract.py`` for the interface and ``AGENTS.md`` for how to add a
|
|
backend. Nothing in deer-flow writes here until ``blob_storage.enabled`` is
|
|
True and a producer has been migrated to the store.
|
|
"""
|
|
|
|
from deerflow.storage.contract import (
|
|
BlobNotConfiguredError,
|
|
BlobNotFoundError,
|
|
BlobReadError,
|
|
BlobRef,
|
|
BlobStore,
|
|
BlobStoreError,
|
|
BlobWriteError,
|
|
is_valid_blob_kind,
|
|
validate_blob_kind,
|
|
)
|
|
from deerflow.storage.manager import get_blob_store, get_blob_store_if_enabled, reset_blob_store
|
|
|
|
__all__ = [
|
|
"BlobNotFoundError",
|
|
"BlobNotConfiguredError",
|
|
"BlobReadError",
|
|
"BlobRef",
|
|
"BlobStore",
|
|
"BlobStoreError",
|
|
"BlobWriteError",
|
|
"get_blob_store",
|
|
"get_blob_store_if_enabled",
|
|
"is_valid_blob_kind",
|
|
"reset_blob_store",
|
|
"validate_blob_kind",
|
|
]
|