Xinmin Zeng 2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00

2.9 KiB

Blob store (deerflow/storage/)

A content-addressed store for bytes that every instance must be able to read, not only the instance that wrote them. The address is the SHA-256 of the content. Operator behavior, the producer migration plan and the garbage-collection rule: blob storage.

Verify

cd backend
python -m pytest tests/test_blob_store.py -q
make test-blocking-io

A new backend adds its tests to tests/test_blob_store.py, which runs the backend-independent scenarios against a temporary root.

Using the store

  • The store is disabled by default. A producer calls get_blob_store_if_enabled() and keeps its local-path code for None. Only a caller that cannot work without the store calls get_blob_store().
  • A store failure is a failed operation, not a silent downgrade. A producer that could not write does not claim that the content is readable on another instance. A reader tries the blob first and keeps the local path as a fallback.
  • Validate a checkpointed ref before you read or write with it. Its kind, digest, size and content type must agree with the metadata that was recorded separately.
  • A fallback read through an HTTP route stays inside the authorized thread and the exact virtual path, and verifies the bytes before it serves them.
  • Changing the root needs a plan. Existing refs must stay readable, and every instance must change together.

Backends

  • The folder name, the backend name and the blob_storage.backend value are the same string. The factory finds a backend by scanning storage/backends/, through STORE_CLASS in its __init__.py.
  • A backend imports one thing from deerflow: the BlobStore contract. It talks to the host through method arguments and its backend_config only, so a third-party backend can live outside this tree.
  • Verify the digest on read. A store that returns wrong bytes cannot be told from a corrupt checkpoint.
  • Reject an unusable config, and fail on an unresolvable backend. A store that starts on a different root or a default backend strands the content that was already written.
  • Validate kind through validate_blob_kind. The kind is a path segment in local_fs, so the traversal check lives in one place.
  • Publish atomically, and accept a concurrent writer. Two Gateways can put the same content at the same time.
  • The size limit belongs to the backend, not to the contract. A producer of large payloads streams or splits them before put_bytes.

Sidecar and garbage collection

  • The sidecar is metadata only. It is not a reference count, and a read must not depend on it.
  • This module does not collect garbage. Identical content is shared by several threads, so liveness comes from the checkpoint rows that name the blob, never from the sidecar's writer_thread_id.