9 Commits

Author SHA1 Message Date
laoyyu
1cd1a8ac63
fix(storage): reject trailing newlines in blob kinds (#8287)
* fix(storage): anchor the blob-kind pattern at end of string, not before a newline

In Python, "$" also matches just before a trailing newline, so the blob
kind pattern accepted "tool\n" and 64 chars + "\n" (65 bytes) even
though the contract grammar is 1-64 chars and the kind is used as a
filesystem path segment by the local_fs backend. Anchor with "\Z"
instead.

* fix(storage): report the enforced blob-kind pattern

---------

Co-authored-by: laoyyu <laoyyu@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-17 15:59:09 +08:00
Xinmin Zeng
2a97ec3e58
docs: turn README into an index and move reference detail to docs/guide (#8113)
* docs: turn README into an index and move reference detail to docs/guide

The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.

* docs: send user docs to topic guides and budget the root README

The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).

* chore: remove unreferenced screenshots and finished plan notes

pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.

* docs: route always-loaded AGENTS.md files to scoped guides

* docs(agents): cut scoped guides to scope, verify, traps, decisions

Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.

Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).

* style: format frontend guide table and guidance test imports

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-11-11 13:26:17 +08:00
Willem Jiang
03afd05917
fix(artifacts): authorize shared image reads and offload MIME lookup (#7604)
* fix(artifacts): authorize shared image reads and offload MIME lookup

* fix(artifacts): skip disabled blob lookups and warn on failures
2026-10-27 16:59:19 +08:00
mika
e2e8ba7aec
fix(artifacts): resolve viewed images from shared blobs (#6329)
Co-authored-by: mikamikasuki <211269698+mikamikasuki@users.noreply.github.com>
2026-10-27 14:21:42 +08:00
lakers-abing
f23a411325
fix(storage): resolve single-module colon backend imports (#7017)
* fix(storage): resolve single-module colon backend imports

* docs(storage): describe both backend import separators

---------

Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
2026-10-20 09:35:56 +08:00
Zheng Haoran
a8250eaa4b
fix(storage): tolerate blob sidecar allocation failures (#6538)
* test(storage): cover sidecar allocation failures

* fix(storage): tolerate sidecar allocation failures

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-10-09 18:22:06 +08:00
PeaceMaker-best
fcee85b822
feat(storage): persist viewed images as shared blobs (#6160)
Store validated image bytes in the configured content-addressed blob backend and resolve checkpointed refs before local compatibility paths.

Refs: #4189

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-10-05 19:06:16 +08:00
PeaceMaker-best
d62d9e786c
feat(storage): restore tool outputs from shared blobs (#6159)
Checkpoint durable refs for host-externalized tool results and materialize verified bytes before model calls on another Gateway.

Refs: #4189

Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
2026-10-02 19:33:25 +08:00
zeng-bohan
82ac1612e3
feat(storage): add a content-addressed blob store contract (issue #4189 item 2) (#5361)
* feat(storage): add a content-addressed blob store contract (#4189 item 2)

Addresses the review on the previous head:

* the retention/dedup claim in the doc now states the deletion rule (liveness
  from the durable references) instead of promising refcount-free deletion, and
  the sidecar records the first writer as advisory provenance only;
* one validate_blob_kind() owns the kind grammar, so the message a producer sees
  is the pattern that is enforced;
* adds the missing storage/AGENTS.md (registry entry included) that both
  docstrings already pointed at;
* drops the dead _import_backend_module() by routing the folder scan through it;
* exports BlobNotConfiguredError from the package;
* renames _KIND_MAX_BYTES to _MAX_BLOB_BYTES (it caps one put, not a kind);
* delete failures raise the neutral BlobStoreError instead of BlobReadError.

Rebased onto current main so the guidance-budget test matches upstream.

Signed-off-by: zeng-bohan <zengbh1@gmail.com>

* fix(storage): restore the config module and backend packages dropped by the squash

The force-push to e72b435 silently lost four items the surviving code
imports, breaking `import deerflow.storage` and the test suite at
collection:

- deerflow/config/blob_storage_config.py (BlobStorageConfig +
  get/set_blob_storage_config + load_blob_storage_config_from_dict,
  mirroring memory_config's host-shared schema and hot-reload pattern)
- app_config.py wiring: the blob_storage AppConfig field, the import,
  and the _apply_singleton_configs -> load_blob_storage_config_from_dict
  call (memory stayed wired; blob did not)
- storage/backends/__init__.py (drop-in contract docstring)
- storage/backends/local_fs/__init__.py (LocalFsBlobStore re-export +
  STORE_CLASS for _scan_backends)

Verified: pytest tests/test_blob_store.py tests/test_app_config_reload.py
tests/test_app_config_name_indexes.py -> 67 passed; ruff format/check
clean on all four files.

* fix(storage): name the failing digest check; document the 64 MiB per-put cap

- get_bytes: split the corruption message into size mismatch (truncation /
  partial write) vs digest mismatch (same-size bit rot, reporting what the
  content hashes to vs what the ref names) — the combined message reported
  'got 1024 bytes, expected 1024' on same-size corruption and sent the
  debugger in the wrong direction.
- docs/blob-storage.md + storage/AGENTS.md: state the _MAX_BLOB_BYTES cap
  (64 MiB) and that it is backend-level defense-in-depth rather than part
  of the BlobStore contract, so the externalized-tool-results follow-up
  sees the limit before the first production put fails.

* test(storage): pin both get_bytes corruption branches separately

The pre-split test corrupted the blob with different-size bytes, so only
the size-mismatch branch ever ran and neither new message was asserted.
Now: different-size corruption asserts 'size mismatch', and a same-size
reversal (bit rot) asserts 'digest mismatch' — the branch the split was
actually about.

* docs(config): document blob_storage in config.example.yaml

Add the blob_storage section (enabled/backend/backend_config) mirroring the
memory section's shape, matching the Configuration section of
backend/docs/blob-storage.md, and bump config_version 46 -> 47 per the
example-file convention for schema additions. Default state is unchanged:
enabled: false, purely opt-in.

* fix(helm): align chart config version with example

* fix(storage): refresh blob store on config changes

---------

Signed-off-by: zeng-bohan <zengbh1@gmail.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: zeng-bohan <310902929+zeng-bohan@users.noreply.github.com>
2026-09-24 09:35:03 +08:00