mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
577c39643b
* feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one `noul` question per probed tool call and denies calls whose Jev risk probability reaches `threshold`. It reuses the existing GuardrailProvider slot: no middleware, config-schema or dependency changes. Semantics that are load-bearing: - state is strict JSON (no `default=`, `allow_nan=False`, string keys only); unserializable arguments, lone surrogates among them, and argument text above `max_state_chars` are denied locally with zero network requests, never truncated and sent. - the async path cancels in-flight requests through `asyncio.timeout`; the sync path checks the deadline after the response headers arrive, around the body read and before accepting the result, and drops late results. No total return-time bound is promised on the sync path. - clients are created per evaluation inside a context manager, and the injected `transport_factory` is called once per network-needing evaluation: httpx closes an injected transport with its client, so a shared instance would be dead on the second call. - denial reason messages carry probability, threshold, served model version and state digest, so the threshold comparison can be replayed from the run journal; local denials record no probability or model version, and allow decisions stay unpersisted (the audit path is unchanged). - provider errors carry no response body or arguments: malformed responses report the field and the offending type only, so a response echoing tool arguments cannot reach middleware logs or the evaluation report. The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py` plus its labeled case set) is the threshold/tools evidence that must precede enabling interception. It reports four mutually exclusive populations (uncached network, cache hits, unprobed tools, local refusals), separates model classification from local refusals and fail-closed errors, measures latency from the caller entering `aevaluate` through cleanup, discloses censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as a coverage gap rather than a pass. Real responses are never written to disk. Cache-pass samples are classified from `decision.metadata['cached']` and counted once, warming calls are retained as an auxiliary group with their failures named in the report and JSON, and an optional connection-probe diagnostic can never discard completed evaluation results. Tests: the provider suite over MockTransport plus the middleware integration, a blocking-IO anchor that drives `aevaluate` against a real loopback HTTP server with a meta-check proving the sync path on the loop trips the gate, and harness cases covering the reporting defects. Every fix above was verified to fail its new tests when reverted. Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off, deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC migration, lone-surrogate local refusals and the no-response-body property of error messages) and config.example.yaml gains a smoke-scope example. Config: the new example block bumps `config_version` 46 -> 47, with the Helm chart's embedded version and README example kept in step as `scripts/check_config_version.sh` (chart.yaml CI) requires. * feat(guardrails): whitelist tool calls inside the TypeSafe risk gate Static allow rules now live in the provider instead of waiting on a composite provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means "not enforced", `[]` refuses every tool, and a list refuses every tool outside it locally (`typesafe.tool_not_whitelisted`) before state construction, caching and networking, independently of `fail_closed`. Whitelisted tools are still probed and still denied when their risk probability reaches `threshold`, and the whitelist is checked before the probe scope so an unlisted tool cannot inherit an allow from being unprobed. Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the evaluation script reports whitelist refusals as their own population, and `GUARDRAILS.md` / `config.example.yaml` document the allow-half migration. * fix(guardrails): address TypeSafe risk gate review findings - Status errors report only the numeric status. The HTTP status line is server-controlled like the body, and this message reaches middleware logs and the evaluation report, so a hostile endpoint could echo the state (raw tool arguments) through it. - Expired cache entries are removed with `pop(key, None)`. Sync evaluation runs on executor threads when a turn produces parallel tool calls, so a second caller can remove the same expired entry; a bare `del` raised KeyError, which the middleware reports as a provider error and, under fail_closed, turns into a spurious denial. - The evaluation harness states its acceptance gates inline (no missed dangerous call, system false-block <= 5%, uncached network p95 <= 1s) instead of citing a design spec that is not in the repository, and reports the whitelist-refusal population alongside the other four. - `--tools` / `--whitelist` strip whitespace and drop empty entries, so `--tools "bash, write_file"` no longer probes only bash while treating `" write_file"` as an unknown tool name. - The case set's `padding` note says top-level argument fields only, matching what the runner implements. Each fix was verified by running its new test against the pre-fix code. * refactor(guardrails): name the TypeSafe permission list allowed_tools Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration copies it verbatim instead of translating `allowed_tools` into `whitelist`. The companion names follow: reason code `typesafe.tool_not_allowed`, metadata `tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools` flag. Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool, tools outside the list are refused locally before any state, request or cache entry, and listed tools still face the risk gate. * fix(guardrails): record only a bounded served model token; fail the eval exit code Two findings from the re-review at d3f0b072. `model` is response content, and `_model_decision` interpolated it verbatim into the reason message -- which reaches the denied ToolMessage, the run journal, middleware logs and the evaluation report on disk, and which the cache retains for replay. A malformed or hostile endpoint can put an echoed tool argument, injected text or a multi-megabyte string there. The token is now recorded as itself only when it matches a conservative shape; anything else is recorded as `unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance record without echoing the value. Rejecting the response instead would turn an odd version string into a fail-closed denial of every call the endpoint answers. The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a value `json.loads` happily produces -- cannot raise where this must not fail. `scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an operator scripting `... && enable` got a green light on a failing evaluation. `main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after the JSON report is written, and prints the failed gate names. Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an echoed argument, a space, a lone surrogate and an oversized value, plus the cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both directions). Both fail against the pre-fix source. docs/GUARDRAILS.md audit boundary updated to match. 113 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py. * fix(evals): fail the risk-gate run when a labeled population is uncovered Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero` is true over an empty risky population. `--tools read_file` excludes all 18 risky cases in the bundled set, so the run printed every gate as PASS and exited 0 after evaluating no risky operation at all. `_score` now gates coverage itself: `risky_coverage_present` and `safe_coverage_present` fail when the probe scope leaves that labeled population with nothing evaluated, so a scope that cannot substantiate the verdict no longer reports one. The safe side had the same vacuity -- `system_false_block_within_target` passes trivially with zero safe cases. Verified end to end against a loopback mock endpoint with the bundled case set: `--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the other four gates still PASS, where dcd38cc3 exits 0 on the same command. Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and `test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3), `test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the exit-code test now runs with both populations present. 116 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py.