github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-10-27 13:40:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/examples/deerflow-extension-jev-classify/README.md
Zeren Wang d700ad9edf
feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client (#5906) 
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client

- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
  configuration: one connection-precedence rule, credential fingerprint, wire-size
  counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
  DeerMem through one injected judge hook (both off by default, `enforce` gated by
  the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
  `config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
  the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
  per-stratum gates, censoring and the no-network baseline.

Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
  upload-event scrub that precedes persistence, only removals/reinforcements of ids
  that existed, and persisted near-duplicate merges (matched by id, or by identity
  for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
  `typesafe` block changes (including a credential rotation); publication is
  serialized and revalidates the config generation, so a build started before an
  `enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
  the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
  and correction.

* fix(typesafe): resolve one credential per layer and refuse unusable connection values

Review follow-ups on the shared TypeSafe client (#5906):

- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
  shared `typesafe:` block: the first layer that sets either credential setting
  decides it, and inside one layer the literal key wins (the standalone
  semantics). A consumer whose endpoint does not accept the block's key no longer
  sends it on every call, and the missing-credential message keeps naming the
  variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
  fragment, embedded credentials, whitespace or control characters, and a
  malformed port fails at construction instead of on every request (httpx raises
  `UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
  swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
  a secret mounted from a file, a non-printable character — is refused when the
  connection resolves. It used to construct fine and then put the whole
  `Bearer <key>` value into h11's `LocalProtocolError` message, which the
  guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
  so that message can no longer carry the credential through the cause chain; the
  exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
  path: `max_attempts: true` was silently read as `1`. An integer stays a usable
  float, so `timeout: 5` keeps working.

Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.

* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence

Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.

Typesafe client (the reply is treated as hostile input too):

- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
  `Content-Encoding`, and read through a bounded `iter_bytes` that stops past
  `MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
  decoded in full and, on the gate path, allowed the tool call unexamined; it is
  now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
  escaped the module's taxonomy, so the memory consumers lost the round's record
  instead of recording a fallback.

Memory judge:

- A request-level failure is recorded as `request_failed`, never `no_verdict`
  (`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
  endpoint was indistinguishable from a provider with no usable answer). The
  adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
  and log; `no_verdict` stays for a question-level failure, and an ineligible side
  keeps its own reason even when the round's request failed.

Shadow evaluation (the enablement gate reads the evidence, not one number):

- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
  meet the 1% target; a sample that only meets the point estimate is
  `INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
  the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
  treated as absent rather than scored: `-3` read as "nothing lost", adding a
  phantom saving to the numerator and the denominator.

Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.

Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.

* fix(memory-eval): pair the baseline comparison on the same records

Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).

Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).

- `Evidence` carries the paired numbers (`baseline_paired_skips`,
  `baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
  heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
  when the annotation covers only a subset of the scored skips (the message carries
  the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
  paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
  `backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.

Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.

* fix(typesafe): bound the sync body read by the deadline

`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.

`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.

* fix(memory): read the YAML off spelling, and veto a skip over the whole batch

Two review findings on the memory judging path.

`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.

The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.

Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
2026-09-27 20:28:05 +08:00

186 lines
9.3 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Text-list classification — independent plugin example
An opt-in Python plugin that gives the agent a `classify_texts` tool: label a
list of short texts with one of the categories supplied in the call. The
backend is chosen by the deployment: **Jev** (TypeSafe System One, one typed
`choice` question per item) or an **OpenAI-compatible chat endpoint** (one JSON
label list per batch). It uses extension API 0.2.2 and the full-stack plugin
catalog, with no host code changes and no browser code. The measurements behind
the design are in [RFC #5653](https://github.com/bytedance/deer-flow/issues/5653).
## Install and configure
From `backend/` in a compatible DeerFlow deployment:
```sh
uv run deerflow extensions install ../examples/deerflow-extension-jev-classify --yes
export TYPESAFE_API_KEY='your-deployment-secret'
```
Set the registered entry in the deployment configuration:
```yaml
plugins:
- use: deerflow_extension_jev_classify:install
enabled: true
config:
enabled: true
backend: jev
```
For a chat-model backend instead:
```yaml
plugins:
- use: deerflow_extension_jev_classify:install
enabled: true
config:
enabled: true
backend: llm
llm_url: https://api.deepseek.com/chat/completions
llm_model: deepseek-flash
```
and export `CLASSIFY_LLM_API_KEY` (or the name set in `llm_api_key_env`).
Provide the environment variable to the actual Gateway process (including its
container, if applicable). Restart Gateway after installing or changing deployment
configuration; refresh Capability Center to see **Text classification**. The outer
switch loads the package; `config.enabled` enables the tool and catalog entry.
It is disabled by default. The plugin catalog is read-only; there is no online
key editor and no browser page. The authenticated `status` backend action reports
enabled/backend/configured/limits without exposing credentials.
**Deployment opt-in sends the item texts, the category names and descriptions,
and the instruction to the configured backend** (`https://api.typesafe.ai/v1/systemone`
for Jev, or the configured `llm_url`). This is a separate API from the main chat
model; a free main model does not make the classifier free. Keys are read only
from the named environment variables, never from plugin settings or messages.
## How the agent uses it
The tool takes a text list, not a file. For a CSV, the agent reads the file with
its normal file tool, sends the rows as items with the row number as `id`, and
writes the returned labels back itself:
```json
{
"items": [{"id": "2", "text": "My card was charged twice"}, {"id": "3", "text": "The webhook returns 500"}],
"categories": [{"name": "billing", "description": "Payments, invoices and refunds."}, {"name": "technical", "description": "Product bugs and integration failures."}],
"instruction": "When a text mentions both, prefer the customer's main request."
}
```
The result lists one entry per item, in input order:
```json
{"backend": "jev", "model": "jev-1.13.0", "results": [{"id": "2", "label": "billing", "status": "ok", "error": ""}, {"id": "3", "label": "technical", "status": "ok", "error": ""}], "counts": {"ok": 2}, "requests": 1}
```
`status` is `ok`, `empty_text`, `too_long`, `error` or `not_processed`; `error`
carries a code such as `http_401`, `timeout`, `network`, `invalid_response`,
`internal`, `stopped` (after an authentication or rate-limit failure) or
`deadline`. A request the plugin cannot serve at all returns
`{"error": {"code": ..., "message": ...}}` with `backend_not_configured`,
`invalid_items`, `invalid_categories` or `too_many_items`.
## Behavior and limits
- Items are grouped up to `batch_size` per request (default 10) and each
request is kept under 256 KiB, so large category descriptions or long texts
simply produce more requests. Outbound sizes use httpx's compact UTF-8 JSON,
including the extra escaping of JSON embedded in chat messages.
Requests run with bounded concurrency (default 4). Caller ids are reconciled
by position; they never become wire ids,
question keys or prompt text.
- Jev sends one `choice` question per item over the shared batch state, so the
classifier sees the batch context, and each item's answer is checked on its
own: a missing, mistyped or out-of-set answer fails only that item. The chat
backend asks for a JSON object `{"labels": [{"id", "label"}]}` with
`response_format: json_object` and temperature 0 (the endpoint must accept
both), then validates the id set and every label in code; a bad list fails
its batch. A label that differs from a category name only by case or
surrounding whitespace is mapped to the name when that mapping is unambiguous.
- At most `max_items` per call (default 200, hard limit 300), 2 to 32 categories,
texts up to `max_text_chars` (default 2000). Ids and category names are at
most 64 bytes in both UTF-8 and JSON-encoded output, so 300 results fit the
host's 64 KiB output bound. Blank and over-long texts are reported, not sent.
The host measures the call input as JSON with non-ASCII escaped and rejects
more than 256 KiB before
the plugin runs, which is roughly 200,000 ASCII or 40,000 CJK characters per
call; the tool description tells the agent to split longer lists.
- The whole call is bounded by `deadline_seconds` (default 25, under the host's
30-second tool limit). Every batch runs under an asyncio timeout equal to the
smaller of `timeout_seconds` and the remaining budget, so a slow or trickling
upstream cannot push the call past the deadline; batches that would start
with under half a second left are `not_processed`.
- There are no automatic retries and no payload logs. Cancellation propagates
and closes the client. An HTTP 401, 402, 403 or 429 stops further batches.
A plugin bug fails its batch with `internal` and a type-name-only log line,
never the whole call.
- **Not a production contract, and not sharing the host's transport.** This example
implements its own HTTP path, including its own `choice`-question shape, size
packing and `deadline_seconds` bound, and it does not retry a failed batch,
unlike the host's shared TypeSafe client (`packages/harness/deerflow/typesafe/`,
used by `deerflow.guardrails.typesafe`), which owns retry/backoff, one error
taxonomy and the two-layer response contract. The extension is a separately
distributed package and must not import host internals, so sharing that transport
needs a publishable client package — option A of the shared Jev/TypeSafe client
design, deferred to a later review.
- Item text is data: the default instruction says so, and nothing from the
items or responses is copied into error messages. The only backend-supplied
text in a result is the served model name, kept to a short identifier.
Optional deployment fields (all validated; unknown fields rejected):
| Field | Default | Meaning |
| --- | --- | --- |
| `backend` | `jev` | `jev` or `llm` |
| `batch_size` | `10` | Items per request, 1–20 |
| `concurrency` | `4` | Concurrent requests, 1–8 |
| `max_items` | `200` | Items per tool call, at most 300 |
| `max_text_chars` | `2000` | Longer texts are reported as `too_long` |
| `timeout_seconds` | `10` | Per-batch bound and per-phase socket timeout, at most 25 |
| `deadline_seconds` | `25` | Whole-call budget, at most 28 |
| `instruction` | built-in | Default instruction prepended to any caller instruction |
| `jev_model` | `jev-latest` | Pin a version when controlled revisions are required |
| `jev_base_url` | `https://api.typesafe.ai` | HTTPS, or HTTP on localhost only; no query, fragment or credentials |
| `jev_api_key_env` | `TYPESAFE_API_KEY` | Environment variable name, never the key itself |
| `llm_url` | none | Full chat-completions URL (same rules as above); required for `backend: llm` |
| `llm_model` | none | Model name for `backend: llm` |
| `llm_api_key_env` | `CLASSIFY_LLM_API_KEY` | Environment variable name, never the key itself |
The chat backend holds its own key because the extension API has no host model
capability yet. Once the host model invocation proposed in
[#5679](https://github.com/bytedance/deer-flow/issues/5679) exists, that backend
is the place to switch to it; the Jev backend stays plugin-owned by design.
## Verify
From `backend/`:
```sh
uv run pytest tests/test_jev_classify_extension.py
```
Coverage includes registration and the disabled state, the Jev request shape and
label mapping, batching, size-based packing, bounded concurrency, per-item and
per-batch failures, label canonicalisation, HTTP, authentication and rate-limit
failures, transport timeouts, the enforced call deadline, cancellation, a backend
bug, the chat backend's output contract and prompt escaping, missing credentials,
invalid requests, the output bound at the largest allowed call, served model
names, real tool-node dispatch, the status action and rejected configuration.
An opt-in smoke test classifies 20 synthetic English and Chinese sentences plus a
blank and an over-long item through the real plugin path against the configured
backend. It costs money, prints labels and statuses only, and never prints the key:
```sh
export TYPESAFE_API_KEY='your-deployment-secret'
uv run python ../examples/deerflow-extension-jev-classify/scripts/verify_live.py
```
Add `--backend llm --llm-url ... --llm-model ... --api-key-env NAME` for a chat
endpoint. A label that differs from the obvious one is printed with the expected
value; that is a prompt to look, not a test failure.
Reference in New Issue View Git Blame Copy Permalink