mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-20 11:20:35 +00:00
* docs: turn README into an index and move reference detail to docs/guide The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md. * docs: send user docs to topic guides and budget the root README The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003). * chore: remove unreferenced screenshots and finished plan notes pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them. * docs: route always-loaded AGENTS.md files to scoped guides * docs(agents): cut scoped guides to scope, verify, traps, decisions Each AGENTS.md keeps four kinds of content: the scope of the directory, the verification commands, traps a local test run does not show, and decisions the code cannot state. Values, field lists, function inventories, test names, issue numbers and history are deleted, because the code, the tests and git state them. Search and fetch providers share one community guide. Four guides with no remaining content are removed. The size check now allows 6 KiB per file (hard limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB). * style: format frontend guide table and guidance test imports --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
71 lines
3.5 KiB
Markdown
71 lines
3.5 KiB
Markdown
# Shared utilities (`deerflow/utils/`)
|
|
|
|
Small helpers that several subsystems share: atomic file publication, advisory
|
|
file locks, off-loop assembly, document conversion and outlines, message text
|
|
extraction, host path portability, retry hints, port allocation and the proxy
|
|
policy for internal HTTP calls. Before you write a local variant of one of these,
|
|
use the helper here; each exists because separate copies drifted apart.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
cd backend
|
|
python -m pytest tests -q -k "atomic or file_lock or host_path or conversion or outline or llm_text or retry_after or port_allocator or http_policy"
|
|
make test-blocking-io
|
|
```
|
|
|
|
## Files on disk
|
|
|
|
- **Publish bytes with `atomic_write.publish_bytes_atomically`.** A reader then
|
|
sees the old file or the new one, never a partial write. On failure, or when you
|
|
discard an owned temporary, call `cleanup_failed_atomic_write`.
|
|
- **Do not read the process umask by changing it.** The runtime is threaded, and
|
|
another thread would create a file with the temporary mask.
|
|
- **Retry a file lock only on contention**, through
|
|
`file_lock.retry_until_locked`. Other errors propagate at once. Each caller
|
|
keeps its own deadline, and clocks and sleeps are injected, so tests do not
|
|
patch the shared `time` module.
|
|
- **Reject Windows device names on every platform** for a path the host will
|
|
create. A workspace written on Linux is opened on Windows later. Compare the
|
|
stem only, and leave the supplied file name unchanged.
|
|
- **`extended_length_path()` is a spelling, not a check.** It does not replace
|
|
symlink resolution or confinement, and it is not used for persisted relative
|
|
paths or Docker mount sources.
|
|
|
|
## Off the event loop
|
|
|
|
- **Build models, tools and agents through `assembly_io.run_assembly`.** Assembly
|
|
can read credential files and wait on MCP discovery. The dedicated executor
|
|
keeps that work off the event loop and off the default thread pool, and it
|
|
preserves the request context. A new async entry point that assembles an agent
|
|
or a model uses it too.
|
|
- **Cancellation during construction must not start inference.**
|
|
- **Resolve the agent factory before admission**, so that an import error cannot
|
|
admit a run.
|
|
- **Document conversion is always offloaded, and its worker is drained through
|
|
cancellation.** A small compressed Office file can take seconds, and the caller
|
|
deletes the source file right after it returns.
|
|
|
|
## Text and content
|
|
|
|
- **Message content can be `None`, a string or a list of blocks.** Use the shared
|
|
extraction helpers, which ignore blocks without string text.
|
|
- **Strip only well-formed reasoning tags.** A word that merely starts with
|
|
`think` is ordinary text.
|
|
- **Outlines and previews are read as `utf-8-sig`**, so a leading BOM does not
|
|
become part of the first heading.
|
|
- **`text_detection` decides which MIME types are active content** for artifact
|
|
and project-document downloads. Keep platform aliases mapped to their standard
|
|
type, and keep the generic `+xml` rule.
|
|
|
|
## Network
|
|
|
|
- **`http_policy.control_plane_trust_env` decides whether a client inherits the
|
|
proxy environment.** Loopback, private, container-internal and cluster-local
|
|
hosts bypass the proxy; public hosts keep it. The module has no third-party
|
|
imports, so the Gateway and the channels import it without loading the sandbox
|
|
stack. Keep it that way.
|
|
- **Bound every provider retry hint with `retry_after`.** An unbounded
|
|
`Retry-After` value would park a run.
|
|
- **`PortAllocator` never probes a port above 65535.** It reports exhaustion.
|