mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-08 20:10:34 +00:00
fe70cfc5ae
* feat(examples): add a standalone text classification plugin A deployment-installed package contributes one model tool, classify_texts, and a status action through the full-stack plugin contract. The tool labels a list of texts against caller-supplied categories with a deployment-configured backend: Jev (one choice question per item over a shared batch state, checked per item) or an OpenAI-compatible chat endpoint (a JSON label list validated in code). Items are packed into requests under a size limit and sent with bounded concurrency; every failure is explicit per call or per item; credentials come only from deployment-named environment variables; item text never enters instructions, question keys, results or errors, and the served model name is kept to a short identifier. Each batch runs under an asyncio timeout bounded by the call deadline, so the whole call stays inside the host's 30-second limit, and ids and category names are bounded in bytes so 300 results fit the host's 64 KiB output limit. * docs(examples): add an opt-in live smoke test for the text classification plugin The script loads the package through the real extension path and classifies 20 synthetic English and Chinese sentences plus a blank and an over-long item with the configured backend, printing labels and statuses only. Against real Jev it agreed on all 20 labels in two requests and 1.6 seconds, with the served model reported as jev-1.13.0. * fix(examples): measure request size as sent and cover transport failures httpx serializes json= bodies with the json.dumps default, escaping every non-ASCII character as six bytes, while the size estimators measured raw UTF-8, so CJK batches could reach about twice the 256 KiB limit. The estimators now measure the same encoding; a test with CJK categories and texts checks the bytes the transport actually received. The network status gets a test through connect and protocol errors, and the deadline test's wall-clock slack is widened for loaded runners. * fix(examples): match classifier request wire encoding * fix(examples): bound classification output keys by JSON size --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
78 lines
4.4 KiB
Markdown
78 lines
4.4 KiB
Markdown
# Backend Tests
|
|
|
|
Backend tests must preserve the runtime invariants they exercise without changing production execution topology.
|
|
|
|
## Shared sandbox search contracts
|
|
|
|
`test_sandbox_search_contract.py` runs shared `ls`/`glob`/`grep` scenarios through
|
|
Local, AIO, E2B, BoxLite, Tenki and OpenSandbox adapters. POSIX shell transports
|
|
execute production commands against temporary files; AIO file RPCs use an
|
|
independent filesystem-backed double. Do not substitute precomputed final search
|
|
results or import another test module's private fixtures. Provisioning, live SDK
|
|
compatibility, permissions enforced above Sandbox, and session lifecycle remain
|
|
in their existing suites. Windows skips this POSIX execution tier explicitly.
|
|
|
|
Compare result names/content, errors and completeness, allowing documented root
|
|
entries and directory suffixes to differ. First-stage cases use ordinary roots,
|
|
positive caps and a plain literal pattern: ignored-root fixes (#5667), E2B literal
|
|
metacharacters (#5627), and exactly-full local caps (#5491) have separate owners.
|
|
Add those shared scenarios after their fixes land; do not encode known bugs as
|
|
expected successful behavior or hide them with permanent xfails.
|
|
|
|
## External system-role admission
|
|
|
|
`poc_external_system_message_injection.py --help` is the opt-in live reproduction
|
|
and post-fix verifier; never run it automatically against an existing user's chat.
|
|
`test_poc_external_system_message_injection.py` tests that CLI offline.
|
|
`test_external_system_message_boundary.py` records model inputs with a fake model:
|
|
the same regression must fail on unfixed admission and pass after rejection,
|
|
without production-provider logging or interpreting model obedience as proof.
|
|
|
|
## Scope-isolation benchmark
|
|
|
|
`test_bench_deermem_scope_isolation.py` exercises production admission and storage.
|
|
Check persisted facts and user/history summaries for semantic safety, but only
|
|
agent-local facts for routing. Failed updates must not become sealed observations;
|
|
report and resume must reject failed or old-schema rows. Stub live models so these
|
|
tests never need credentials or network access.
|
|
|
|
## MCP claim fencing
|
|
|
|
`test_mcp_task_repository.py` covers same-worker reclaim during an in-flight
|
|
release, poll/cancel snapshot, or notification completion. Use explicit events
|
|
to pause the old operation at the persistence boundary, reclaim via the real
|
|
repository, then verify the entire new row remains unchanged. Reclaiming before
|
|
the old operation starts does not catch SQLite SELECT/ORM-flush races. Keep the
|
|
old completion timestamp within its original lease so expiry cannot mask a
|
|
missing token fence; always drain paused tasks and restore session patches.
|
|
|
|
## Executor starvation tests
|
|
|
|
`test_executor_starvation.py` covers the deterministic starvation semantics from RFC #4560:
|
|
|
|
- default-executor saturation and queueing;
|
|
- cancellation of an awaiter while an already-started synchronous worker continues;
|
|
- isolation between the asyncio default executor and DeerFlow's dedicated file-I/O executor.
|
|
|
|
Use explicit synchronization such as `threading.Event` rather than sleep-based timing thresholds for worker lifecycle assertions. Every test must release blocked workers and restore any process-global monkeypatches so teardown cannot leak threads or state into later tests.
|
|
|
|
Stress/soak testing, AnyIO worker instrumentation, Uvicorn multi-process behavior, and broad production executor redesign are separate concerns and should not be folded into these deterministic regressions.
|
|
|
|
## Managed DeepSeek compatibility
|
|
|
|
`test_managed_deepseek.py` exercises real SDK request serialization and SSE parsing
|
|
with an HTTP double; do not replace the provider classes with successful stubs.
|
|
`test_managed_deepseek_live.py` uses the same production probe/model configuration
|
|
against DeepSeek only with `DEER_FLOW_RUN_LIVE_TESTS=1` and
|
|
`DEEPSEEK_TEST_API_KEY`, never in CI. Keep credentials and provider payloads out of
|
|
committed evidence. A passing connectivity probe does not establish full agent
|
|
compatibility; distinguish protocol assertions from observed live behavior.
|
|
|
|
## Classification request packing
|
|
|
|
Pin the Jev example's outbound byte budget against captured HTTP request bodies.
|
|
Cover UTF-8 text and JSON embedded in chat messages, including escaping-heavy
|
|
inputs that fit the host's separate input limit. Use offline transports.
|
|
Check the host's JSON output bound with the maximum item count and keys that
|
|
expand when escaped; preserving raw UTF-8 limits alone does not cover it.
|