mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-20 11:20:35 +00:00
* docs: turn README into an index and move reference detail to docs/guide The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md. * docs: send user docs to topic guides and budget the root README The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003). * chore: remove unreferenced screenshots and finished plan notes pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them. * docs: route always-loaded AGENTS.md files to scoped guides * docs(agents): cut scoped guides to scope, verify, traps, decisions Each AGENTS.md keeps four kinds of content: the scope of the directory, the verification commands, traps a local test run does not show, and decisions the code cannot state. Values, field lists, function inventories, test names, issue numbers and history are deleted, because the code, the tests and git state them. Search and fetch providers share one community guide. Four guides with no remaining content are removed. The size check now allows 6 KiB per file (hard limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB). * style: format frontend guide table and guidance test imports --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2.9 KiB
2.9 KiB
Blob store (deerflow/storage/)
A content-addressed store for bytes that every instance must be able to read, not only the instance that wrote them. The address is the SHA-256 of the content. Operator behavior, the producer migration plan and the garbage-collection rule: blob storage.
Verify
cd backend
python -m pytest tests/test_blob_store.py -q
make test-blocking-io
A new backend adds its tests to tests/test_blob_store.py, which runs the
backend-independent scenarios against a temporary root.
Using the store
- The store is disabled by default. A producer calls
get_blob_store_if_enabled()and keeps its local-path code forNone. Only a caller that cannot work without the store callsget_blob_store(). - A store failure is a failed operation, not a silent downgrade. A producer that could not write does not claim that the content is readable on another instance. A reader tries the blob first and keeps the local path as a fallback.
- Validate a checkpointed ref before you read or write with it. Its kind, digest, size and content type must agree with the metadata that was recorded separately.
- A fallback read through an HTTP route stays inside the authorized thread and the exact virtual path, and verifies the bytes before it serves them.
- Changing the root needs a plan. Existing refs must stay readable, and every instance must change together.
Backends
- The folder name, the backend name and the
blob_storage.backendvalue are the same string. The factory finds a backend by scanningstorage/backends/, throughSTORE_CLASSin its__init__.py. - A backend imports one thing from
deerflow: theBlobStorecontract. It talks to the host through method arguments and itsbackend_configonly, so a third-party backend can live outside this tree. - Verify the digest on read. A store that returns wrong bytes cannot be told from a corrupt checkpoint.
- Reject an unusable config, and fail on an unresolvable backend. A store that starts on a different root or a default backend strands the content that was already written.
- Validate
kindthroughvalidate_blob_kind. The kind is a path segment inlocal_fs, so the traversal check lives in one place. - Publish atomically, and accept a concurrent writer. Two Gateways can put the same content at the same time.
- The size limit belongs to the backend, not to the contract. A producer of
large payloads streams or splits them before
put_bytes.
Sidecar and garbage collection
- The sidecar is metadata only. It is not a reference count, and a read must not depend on it.
- This module does not collect garbage. Identical content is shared by several
threads, so liveness comes from the checkpoint rows that name the blob, never
from the sidecar's
writer_thread_id.