mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-09 21:49:37 +00:00
* feat(harness): add deterministic tool receipts with model-visible ledger Stamp an immutable per-call fact record (tool name, status, args/output hashes, byte count, timestamp) onto every tool result via a new ToolReceiptMiddleware, and inject the derived receipt ledger (r1..rN) into the model context so subagent reports can cite executed actions. - tool_receipt.py: receipt core (make/extract/render), newest-first budget eviction, ids derived from the append-only message stream - ToolReceiptMiddleware: stamps ToolMessages directly or inside Command-wrapped results; hidden ledger injection mirrors DurableContextMiddleware; sits between ToolProgress and ToolErrorHandling with a build-time ordering guard - config: new verification section (receipts on, judge off), config version 32 -> 33 with example/helm/docs updates * feat(harness): split receipt rendering from stamping; address PR review Review fixes (PR #4659): - output_sha256 now uses sort_keys=True for structured content, matching the order-invariant args fingerprint - stamping failures log at warning (silent ledger gaps would corrupt citations); tool execution remains never blocked - _insert_after_leading_system_messages extracted to shared public message_utils.insert_after_leading_system_messages; both middlewares depend on it instead of a private cross-module helper - code comments in English RFC #4651 revision-2 alignment: - receipts_render_mode config ('always' | 'delegation_only'): subagent chains always render the ledger (citations are produced there); the lead chain renders only while processing subagent results, removing the always-on token tax from ordinary turns - receipts gain bounded args_preview/output_preview (<=200 chars, tail for output) so later typed claim bindings (tests_passed) can anchor to a specific recorded execution * docs(harness): state receipt freshness caveat and vocabulary layering in module docstring * merge: upstream/main — resolve AGENTS.md split, bump config_version to 34, drop unused receipt previews - backend/AGENTS.md: take upstream's slimmed root guidance (#4799); move the ToolReceiptMiddleware chain entry into agents/middlewares/AGENTS.md and the verification.* hot-reload mention into config/AGENTS.md - config.example.yaml + helm values/README: config_version 33 -> 34 so existing v33 configs get the outdated-config prompt (review: willem-bd) - tool_receipt.py: drop args_preview/output_preview — no Layer 1 consumer reads them; re-add with the Layer 2 claim-binding consumer (review: willem-bd) * docs(harness): cover receipt id renumbering after compaction in module docstring Positional display ids are stable only while history is append-only; compaction drops ToolMessages and the survivors renumber, so Layer 2 citation verification must resolve [rN] against the ledger as of the citing turn (review: willem-bd, doc-only). * chore(config): bump config_version to 35 main reached 34 via #4780 without the verification section; publishing the new schema at the same number would silently skip the outdated-config prompt for configs synced from main in that window (review: willem-bd). * fix(skills): restore errno import dropped upstream in #4830 upstream/main adf6c422 uses errno.ENOTDIR in the drift guard but removed the import, so the PR merge ref fails lint-backend (F821). * fix(harness): harden tool receipts against forgery and turn-scope delegation_only Address willem-bd's pre-merge review on #4659: 1. Untrusted receipt metadata: the gateway now strips the server-owned deerflow_tool_receipt key from external input messages; stamping always overwrites any tool-supplied value instead of preserving it; and extract_tool_receipts validates persisted receipt shapes (required typed fields, unknown keys ignored) so malformed entries are skipped instead of crashing render or passing as runtime-stamped evidence. 2. delegation_only no longer sticks on: _should_render now scopes the subagent_status scan to the current turn (messages after the latest genuine user message), so an old completed delegation stops rendering the ledger on later ordinary turns. The genuine-user predicate moves to message_utils.is_genuine_user_message, shared with input sanitization. * fix(harness): stamp receipts outside short-circuiting tool middlewares Address willem-bd's review on #4659: ToolReceiptMiddleware was registered inside Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress, each of which can return a ToolMessage without invoking its handler — blocked calls (e.g. a read-before-write-denied write_file) never got a receipt, silently gapping the ledger on a default-enabled path. SandboxAudit additionally rebuilds medium-risk results, dropping an inner stamp. ToolReceiptMiddleware is now the outermost wrap_tool_call layer in the runtime tail. Normal results still carry deerflow_tool_meta (stamped by ToolErrorHandling on the inner return path); short-circuit messages self-stamp meta or fall back to message.status. The new invariant is declared as ordering constraints in deerflow.extensions.ordering, with composed-chain regression tests for a blocked write and a warn-rebuilt bash result.
This commit is contained in:
Zeren Wang
GitHub
parent
308948aa05
commit
4e35f0d1d4
@ -1117,7 +1117,7 @@ The chat header also shows a context-window gauge when the selected model has a
|
||||
|
||||
Sub-agents are an optimization, not the default response to a complex request.
|
||||
|
||||
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions — when delegation has clear net benefit from real parallel latency, specialist capability, or context isolation. It keeps interdependent scopes and overlapping side effects out of parallel dispatch; a bounded sequential chain can still run in one sub-agent when specialist or context-isolation benefit clearly wins. The lead uses the fewest useful sub-agents and re-evaluates later batches instead of fanning out solely because a task is large or multi-step. Sub-agents report back structured results, and the lead agent verifies and synthesizes them into a coherent output. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Reloaded thread history enforces the same boundary: callback-captured sub-agent AI responses remain available in run-event diagnostics but are excluded from the parent transcript, while the parent `task` result remains attached to its subtask card. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Concurrent parent runs also receive independent server-side sub-agent execution IDs, so a provider that reuses a tool-call ID cannot make one run poll, cancel, or clean up another run's background task. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is attributed back to the dispatching step from that run's terminal tool-message metadata rather than a process-global provider-ID cache.
|
||||
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions — when delegation has clear net benefit from real parallel latency, specialist capability, or context isolation. It keeps interdependent scopes and overlapping side effects out of parallel dispatch; a bounded sequential chain can still run in one sub-agent when specialist or context-isolation benefit clearly wins. The lead uses the fewest useful sub-agents and re-evaluates later batches instead of fanning out solely because a task is large or multi-step. Sub-agents report back structured results, and the lead agent verifies and synthesizes them into a coherent output. Deterministic tool receipts cover both direct tool messages and state-updating `Command` results such as delegated `task` responses; when the receipt ledger reaches its context budget, it retains the newest actions and their original receipt IDs. Operators can disable this provenance layer with `verification.receipts_enabled: false`. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Reloaded thread history enforces the same boundary: callback-captured sub-agent AI responses remain available in run-event diagnostics but are excluded from the parent transcript, while the parent `task` result remains attached to its subtask card. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Concurrent parent runs also receive independent server-side sub-agent execution IDs, so a provider that reuses a tool-call ID cannot make one run poll, cancel, or clean up another run's background task. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is attributed back to the dispatching step from that run's terminal tool-message metadata rather than a process-global provider-ID cache.
|
||||
|
||||
For example, independent read-only research can run concurrently when the wall-clock savings outweigh duplicated discovery and synthesis cost, while a repository refactor with shared files and sequential test feedback remains with the lead agent. When `max_concurrent_subagents` is `1`, parallel and multi-batch routing guidance is disabled; delegation remains available only for material specialist or context-isolation benefit.
|
||||
|
||||
|
||||
@ -35,6 +35,7 @@ from app.gateway.utils import sanitize_log_param
|
||||
from app.mcp_tasks.errors import PermanentNotificationError
|
||||
from deerflow.agents.middlewares.dynamic_context_middleware import _DYNAMIC_CONTEXT_REMINDER_KEY, _REMINDER_DATE_KEY
|
||||
from deerflow.agents.middlewares.input_sanitization_middleware import frame_untrusted_text
|
||||
from deerflow.agents.middlewares.tool_receipt import TOOL_RECEIPT_KEY
|
||||
from deerflow.agents.middlewares.tool_transform_meta import TOOL_TRANSFORMS_KEY
|
||||
from deerflow.agents.middlewares.view_image_middleware import _IMAGE_CONTEXT_MESSAGE_MARKER_KEY
|
||||
from deerflow.config.app_config import get_app_config
|
||||
@ -113,6 +114,7 @@ _SERVER_OWNED_MESSAGE_METADATA_KEYS = (
|
||||
_DYNAMIC_CONTEXT_REMINDER_KEY,
|
||||
_REMINDER_DATE_KEY,
|
||||
_IMAGE_CONTEXT_MESSAGE_MARKER_KEY,
|
||||
TOOL_RECEIPT_KEY,
|
||||
TOOL_TRANSFORMS_KEY,
|
||||
}
|
||||
)
|
||||
@ -301,10 +303,10 @@ def normalize_input(raw_input: dict[str, Any] | None, *, trusted_internal: bool
|
||||
of bubbling up as a 500. The gateway is a system boundary, so per-entry
|
||||
validation errors are the right shape for clients to retry against.
|
||||
|
||||
``original_user_content``, dynamic-context reminder markers, and the
|
||||
transient view-image context marker are server-owned. External callers
|
||||
cannot supply them; trusted internal channel calls may preserve metadata
|
||||
they added before invoking this boundary.
|
||||
``original_user_content``, dynamic-context reminder markers, the
|
||||
transient view-image context marker, and tool receipts are server-owned.
|
||||
External callers cannot supply them; trusted internal channel calls may
|
||||
preserve metadata they added before invoking this boundary.
|
||||
"""
|
||||
if raw_input is None:
|
||||
return {}
|
||||
|
||||
@ -61,7 +61,7 @@ it to that middleware's declaration in the same change.
|
||||
10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution. Command classification is **defense-in-depth and audit, not a security boundary** — the sandbox itself is the isolation boundary. Command substitution is judged by *position*, not by the presence of `$(`: a substitution in **command position** (`$(curl url)`, `` `curl url` ``, the word after a `|`/`&&`/`;`, or any `eval`/`source` argument) executes fetched or interpreted content and is blocked, while **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is therefore matched anchored against each split sub-command, never against the whole compound string, and `_split_compound_command(split_pipes=True)` supplies those sub-commands; rules that span a pipe (`| sh`, `base64 -d | ...`) still rely on `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading variable assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`), which are still command position; its assignment branch requires whitespace before the substitution, which is exactly what keeps `x=$(curl url)` in value position. Two execution contexts are deliberately **position-blind** and matched against the whole command in Pass 1, because they execute what they receive wherever they appear (including as an argument to something else, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter's **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) that reaches the same place through stdin. All three substitution spellings (`$(cmd`, `<(cmd`, `` `cmd ``) share one `_RISKY_SUBSTITUTION` opener so a rule cannot cover one spelling and miss another. An unquoted newline splits like `;`, because it separates statements the same way: leaving it joined let `echo hi\n$(curl url)` evade the anchored rules that its `;` spelling triggers. A heredoc body is data rather than statements, so `_split_compound_command` records headers (`<<EOF`, `<<-EOF`, `<<'EOF'`) and consumes their bodies verbatim at the newline that starts them — otherwise a body line beginning with `$(curl url)` would be promoted to a command position the shell never creates. Two things that look like headers must not open one, or a body that never terminates swallows every following statement: `<<<` is a here-string (both a lookahead and a lookbehind are needed, or the trailing `<<` of `<<< "text"` reads as a heredoc with delimiter `text`), and a `<<` inside `$(( ... ))` / `(( ... ))` is a bit shift, so arithmetic depth is tracked alongside the quote flags. That is a heuristic, not shell parsing: it exists only to avoid manufacturing command positions *and* to avoid destroying real ones. An unterminated body consumes the rest of the string; an unclosed `((` only disables heredoc detection, so newlines keep splitting and the failure direction stays towards seeing more command positions rather than fewer. Known, deliberate gaps: process substitution outside `eval`/`source` (`. <(curl u)`) is not detected — closing it would require real shell parsing, which is out of scope for this layer. Two-step forms (`x=$(curl u); eval "$x"`) are inherent rather than incidental: any rule that allows output capture allows the first statement, and connecting it to the later `eval` needs dataflow analysis, not pattern matching. There is currently no config gate: the middleware is appended unconditionally in `_build_runtime_middlewares`, so it applies to both the lead agent and subagents.
|
||||
11. **ReadBeforeWriteMiddleware** - *(optional, if `read_before_write.enabled`, default on)* Outermost write gate (issue #3857): `read_file` stamps a content hash onto its ToolMessage; `write_file` (append/overwrite-existing) and `str_replace` are blocked unless the newest mark for that path matches the file's current hash. Sits outside ToolProgressMiddleware and ToolErrorHandlingMiddleware so a blocked write returns immediately without consuming a ToolProgress slot. Blocked results call `normalize_tool_result` directly to stamp `deerflow_tool_meta` (`recoverable_by_model=True`) before returning, keeping the result well-formed for any outer consumer. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose `read_file` reports failures as `"Error: ..."` strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped)
|
||||
12. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its `wrap_tool_call` receives results already stamped with `deerflow_tool_meta`. Tracks per-(thread, tool) consecutive "no-new-info" calls across three error categories: (a) `recoverable_by_model=True` (no_results, not_found, permission, Jaccard-duplicate success): ACTIVE → WARNED (terminal — hint re-injected on each subsequent problem); (b) `recoverable_by_model=False, action≠stop` (rate_limited, transient): ACTIVE → WARNED → BLOCKED after `warn_escalation_count` more problems; (c) `recoverable_by_model=False, action=stop` (auth, config, internal): immediately BLOCKED on first occurrence. **Division of labor with LoopDetectionMiddleware:** ToolProgressMiddleware is a result-quality guard — fires after tool execution and blocks specific tools that stop producing new information; LoopDetectionMiddleware is a call-pattern guard — fires after the model responds and hard-stops the whole turn when the model repeatedly issues identical tool_calls. Both can inject HumanMessage hints in the same model call without conflict; neither reads the other's internal state.
|
||||
13. **ToolErrorHandlingMiddleware** - Receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
|
||||
13. **ToolReceiptMiddleware + ToolErrorHandlingMiddleware** - `ToolReceiptMiddleware` is *(optional, if `verification.receipts_enabled`, default on)*. It is the **outermost `wrap_tool_call` layer** — registered ahead of entries 9-12 — because Guardrail/SandboxAudit/ReadBeforeWrite/ToolProgress can short-circuit a call with their own ToolMessage (and SandboxAudit rebuilds medium-risk results); an inner receipt layer would silently gap the ledger on those results (ordering constraints in `deerflow.extensions.ordering`). Normal results still carry the `deerflow_tool_meta` status ToolErrorHandlingMiddleware stamps on the inner return path; short-circuit messages self-stamp meta or fall back to `message.status`. It stamps deterministic provenance (tool name, status, args/output hashes, byte count, timestamp) onto direct `ToolMessage` results and every matching `ToolMessage` carried in `Command.update.messages`, including delegated `task`, `present_file`, `view_image`, and `tool_search` results; before model calls it derives a hidden receipt ledger (display ids r1..rN) from message state, and when the 2,000-character budget is exceeded the newest receipts are retained in chronological order with their original ids plus an older-receipts omission marker. `ToolErrorHandlingMiddleware` receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
|
||||
|
||||
Authorization identity plumbing is independent of whether authorization enforcement is enabled. Gateway removes client-supplied `is_internal` / `authz_attributes` / `channel_user_id`, derives `is_internal` only from the server-owned `request.state.auth_source`, and accepts `channel_user_id` only from an internally authenticated IM caller's top-level `body.context`; free-form `body.config` can never supply it. `build_principal_from_context` is the shared Principal builder for assembly-time authorization and `GuardrailAuthorizationAdapter`; it applies `default_role`, strict-boolean internal provenance, and copy-on-read `authz_attributes`. The built-in RBAC provider validates `authorization.default_role` during provider resolution so an unknown fallback role fails agent construction instead of degrading into an empty tool set. Task delegation carries `is_internal` plus copied attributes through `SubagentExecutor`, while `GuardrailMiddleware` maps the same runtime fields into `GuardrailRequest`. Phase 1B applies Layer 1 before deferred-tool assembly on the lead, native-subagent, and embedded-client paths, then passes the same provider instance into Layer 2. Framework-provided `describe_skill` and memory tools are included in Layer 1 but restored to their legacy post-`tool_search` ordering afterward. `DeerFlowClient.stream()` treats its in-process caller as trusted and accepts the same identity fields as keyword overrides; it includes the complete Principal in its agent cache key and deep-copies nested attributes so caller mutation cannot make a stale tool set look current.
|
||||
|
||||
|
||||
@ -22,6 +22,7 @@ from langchain_core.messages import AnyMessage, HumanMessage, SystemMessage
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from deerflow.agents.middlewares.delegation_ledger import extract_delegations, render_delegation_ledger
|
||||
from deerflow.agents.middlewares.message_utils import insert_after_leading_system_messages
|
||||
from deerflow.agents.middlewares.skill_context import extract_skills, render_skill_context
|
||||
from deerflow.agents.thread_state import _DELEGATION_LEDGER_MAX_ENTRIES, TERMINAL_STATUSES
|
||||
from deerflow.config.summarization_config import DEFAULT_SKILL_FILE_READ_TOOL_NAMES
|
||||
@ -60,13 +61,6 @@ def _bound_text(text: str, cap: int) -> str:
|
||||
return f"{text[:head]}{omitted_marker}{text[-tail:]}"
|
||||
|
||||
|
||||
def _insert_after_leading_system_messages(messages: list, injected: list) -> list:
|
||||
index = 0
|
||||
while index < len(messages) and isinstance(messages[index], SystemMessage):
|
||||
index += 1
|
||||
return [*messages[:index], *injected, *messages[index:]]
|
||||
|
||||
|
||||
def _render_durable_context_data(summary_text: str | None, ledger: list, skills: list) -> str:
|
||||
data_parts: list[str] = []
|
||||
if summary_text:
|
||||
@ -256,7 +250,7 @@ class DurableContextMiddleware(AgentMiddleware[AgentState]):
|
||||
)
|
||||
if not data_block:
|
||||
return request
|
||||
messages = _insert_after_leading_system_messages(
|
||||
messages = insert_after_leading_system_messages(
|
||||
list(request.messages),
|
||||
[
|
||||
SystemMessage(
|
||||
|
||||
@ -31,13 +31,11 @@ from langchain.agents.middleware.types import (
|
||||
from langchain_core.messages import HumanMessage
|
||||
from langgraph.errors import GraphBubbleUp
|
||||
|
||||
from deerflow.agents.human_input import read_human_input_response
|
||||
from deerflow.agents.middlewares.message_utils import is_genuine_user_message
|
||||
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, message_content_to_text
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SUMMARY_MESSAGE_NAME = "summary"
|
||||
|
||||
# Finite set of blocked tag names: system-reserved + common injection patterns.
|
||||
#
|
||||
# Maintenance: when adding a new framework block tag that the system emits into
|
||||
@ -174,21 +172,6 @@ def neutralize_untrusted_tags(text: str) -> str:
|
||||
return _neutralize_boundary_tokens(text)
|
||||
|
||||
|
||||
def _is_genuine_user_message(message: object) -> bool:
|
||||
"""Return True for real user messages, excluding system-injected HumanMessages.
|
||||
|
||||
``hide_from_ui`` is also used by hidden UI replies from HumanInputCard, so
|
||||
only skip hidden HumanMessages that do not carry a valid user response.
|
||||
"""
|
||||
if not isinstance(message, HumanMessage):
|
||||
return False
|
||||
if message.name == _SUMMARY_MESSAGE_NAME:
|
||||
return False
|
||||
if message.additional_kwargs.get("hide_from_ui") and read_human_input_response(message.additional_kwargs) is None:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def frame_untrusted_text(text: str) -> str:
|
||||
"""Sanitize untrusted text, then wrap it in user-input boundary markers.
|
||||
|
||||
@ -303,7 +286,7 @@ class InputSanitizationMiddleware(AgentMiddleware[AgentState]):
|
||||
messages = list(request.messages)
|
||||
for i in range(len(messages) - 1, -1, -1):
|
||||
msg = messages[i]
|
||||
if not _is_genuine_user_message(msg):
|
||||
if not is_genuine_user_message(msg):
|
||||
if isinstance(msg, HumanMessage):
|
||||
logger.debug(
|
||||
"_process_request: skipping non-genuine HumanMessage at pos=%d name=%s hide_from_ui=%s content_preview=%.80r",
|
||||
|
||||
@ -0,0 +1,38 @@
|
||||
"""Shared message-list helpers for agent middlewares."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
|
||||
from deerflow.agents.human_input import read_human_input_response
|
||||
|
||||
_SUMMARY_MESSAGE_NAME = "summary"
|
||||
|
||||
|
||||
def is_genuine_user_message(message: object) -> bool:
|
||||
"""Return True for real user messages, excluding system-injected HumanMessages.
|
||||
|
||||
``hide_from_ui`` is also used by hidden UI replies from HumanInputCard, so
|
||||
only skip hidden HumanMessages that do not carry a valid user response.
|
||||
"""
|
||||
if not isinstance(message, HumanMessage):
|
||||
return False
|
||||
if message.name == _SUMMARY_MESSAGE_NAME:
|
||||
return False
|
||||
if message.additional_kwargs.get("hide_from_ui") and read_human_input_response(message.additional_kwargs) is None:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def insert_after_leading_system_messages(messages: list, injected: list) -> list:
|
||||
"""Insert messages right after the leading run of SystemMessages.
|
||||
|
||||
Context injections belong after the system prompt (instructions first,
|
||||
background context second) and before the conversation — never ahead of
|
||||
system messages (provider/protocol assumption) and never appended at the
|
||||
tail (would displace the latest turn and read as tool output).
|
||||
"""
|
||||
index = 0
|
||||
while index < len(messages) and isinstance(messages[index], SystemMessage):
|
||||
index += 1
|
||||
return [*messages[:index], *injected, *messages[index:]]
|
||||
@ -158,6 +158,7 @@ def _build_runtime_middlewares(
|
||||
include_uploads: bool,
|
||||
include_dangling_tool_call_patch: bool,
|
||||
lazy_init: bool = True,
|
||||
receipts_render_mode: str = "delegation_only",
|
||||
authorization_provider=None,
|
||||
authorization_infrastructure_tool_names: frozenset[str] = frozenset(),
|
||||
) -> list[AgentMiddleware]:
|
||||
@ -202,6 +203,20 @@ def _build_runtime_middlewares(
|
||||
tail.append(DanglingToolCallMiddleware())
|
||||
tail.append(LLMErrorHandlingMiddleware(app_config=app_config))
|
||||
|
||||
# ToolReceiptMiddleware is the outermost wrap_tool_call layer: Guardrail,
|
||||
# SandboxAudit, ReadBeforeWrite, and ToolProgress can all short-circuit a
|
||||
# call with their own ToolMessage, and SandboxAudit rebuilds medium-risk
|
||||
# results — an inner receipt layer would miss those results and silently
|
||||
# gap the ledger. Stamping out here still sees deerflow_tool_meta on
|
||||
# normal results (ToolErrorHandling stamps it on the inner return path)
|
||||
# and on self-stamped short-circuit messages; the remainder fall back to
|
||||
# message.status (see make_tool_receipt).
|
||||
verification_config = app_config.verification
|
||||
if verification_config.receipts_enabled:
|
||||
from deerflow.agents.middlewares.tool_receipt_middleware import ToolReceiptMiddleware
|
||||
|
||||
tail.append(ToolReceiptMiddleware(render_mode=receipts_render_mode))
|
||||
|
||||
# Authorization uses the existing GuardrailMiddleware so execution-time
|
||||
# deny, audit, and fail-closed handling stay in one proven implementation.
|
||||
# It is appended before an explicit guardrail provider, making authorization
|
||||
@ -299,6 +314,9 @@ def build_lead_runtime_middlewares(
|
||||
include_uploads=True,
|
||||
include_dangling_tool_call_patch=True,
|
||||
lazy_init=lazy_init,
|
||||
# The lead renders the receipt ledger only while processing subagent
|
||||
# results (default "delegation_only"); stamping stays always-on.
|
||||
receipts_render_mode=app_config.verification.receipts_render_mode,
|
||||
authorization_provider=authorization_provider,
|
||||
authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()),
|
||||
)
|
||||
@ -332,6 +350,9 @@ def build_subagent_runtime_middlewares(
|
||||
include_uploads=False,
|
||||
include_dangling_tool_call_patch=True,
|
||||
lazy_init=lazy_init,
|
||||
# Subagent chains always render the ledger: citations are produced in
|
||||
# the subagent context — no ledger, no citations, Layer 1 goes inert.
|
||||
receipts_render_mode="always",
|
||||
authorization_provider=authorization_provider,
|
||||
authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()),
|
||||
)
|
||||
|
||||
@ -0,0 +1,150 @@
|
||||
"""Deterministic tool-call receipts: the zero-LLM verification layer.
|
||||
|
||||
Every tool result gets a receipt stamped into ``additional_kwargs`` by
|
||||
``ToolReceiptMiddleware``. Receipts are *derived* from the message stream
|
||||
(never stored separately), so rendering for the model and harvesting for the
|
||||
parent agent always agree. Display ids (``r1..rN``) are positional over the
|
||||
append-only message list, which keeps them stable across turns — but only
|
||||
while history stays append-only (see the renumbering caveat below).
|
||||
|
||||
Layering contract: a tool receipt is an immutable *fact* record per tool call,
|
||||
message-carried. It is distinct from the runtime-layer run delivery receipt
|
||||
(``run.delivery`` event, one per run, event-store-carried) — the two layers
|
||||
share only the verdict *structure* convention (``source``/``requirement`` +
|
||||
details); the ``satisfied`` boolean stays exclusive to the runtime hard gate,
|
||||
and advisory layers use neutral vocabulary (``citation_resolved``,
|
||||
``supported``) so the model never conflates evidence with acceptance.
|
||||
|
||||
Freshness caveat: receipts capture execution truth (the raw tool return,
|
||||
stamped before sanitization/truncation rewrites content further out the
|
||||
chain). After compaction, only the sanitized ``content`` survives — so
|
||||
``output_sha256`` is a *freshness stamp*, not a re-checkable fingerprint
|
||||
against the persisted message.
|
||||
|
||||
Renumbering caveat: compaction/summarization (which long subagent runs use)
|
||||
drops older ``ToolMessage``s, and since display ids are assigned positionally
|
||||
in ``extract_tool_receipts``, the surviving receipts renumber — an ``[r3]``
|
||||
cited before compaction can point at a different tool call (or nothing)
|
||||
after. Layer 2 citation verification must therefore resolve ``[rN]``
|
||||
references against the ledger as of the citing turn, not the post-compaction
|
||||
ledger.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from typing import TypedDict
|
||||
|
||||
from langchain_core.messages import ToolMessage
|
||||
|
||||
from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY
|
||||
|
||||
TOOL_RECEIPT_KEY = "deerflow_tool_receipt"
|
||||
|
||||
_HASH_LEN = 16
|
||||
_RENDER_CHAR_BUDGET = 2000
|
||||
|
||||
|
||||
class ToolReceipt(TypedDict):
|
||||
id: str # display id, assigned by extract_tool_receipts ("r1"..)
|
||||
tool_call_id: str
|
||||
tool_name: str
|
||||
status: str # success | error | partial_success (from deerflow_tool_meta)
|
||||
args_sha256: str
|
||||
output_sha256: str
|
||||
output_bytes: int
|
||||
created_at: str
|
||||
|
||||
|
||||
def _short_hash(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()[:_HASH_LEN]
|
||||
|
||||
|
||||
def make_tool_receipt(tool_call: dict, message: ToolMessage) -> dict:
|
||||
"""Build a receipt for one tool call/result pair (no display id yet)."""
|
||||
args = tool_call.get("args")
|
||||
args_bytes = json.dumps(args if isinstance(args, dict) else {}, sort_keys=True, default=str).encode("utf-8")
|
||||
content = message.content if isinstance(message.content, str) else json.dumps(message.content, sort_keys=True, default=str)
|
||||
meta = (message.additional_kwargs or {}).get(TOOL_META_KEY) or {}
|
||||
status = str(meta.get("status") or getattr(message, "status", "success") or "success")
|
||||
return {
|
||||
"tool_call_id": str(tool_call.get("id") or ""),
|
||||
"tool_name": str(tool_call.get("name") or ""),
|
||||
"status": status,
|
||||
"args_sha256": _short_hash(args_bytes),
|
||||
"output_sha256": _short_hash(content.encode("utf-8")),
|
||||
"output_bytes": len(content.encode("utf-8")),
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def extract_tool_receipts(messages: list) -> list[ToolReceipt]:
|
||||
"""Collect stamped receipts in message order, assigning display ids r1..rN.
|
||||
|
||||
Receipt dicts come back out of persisted checkpoints, so their shape is
|
||||
validated before use: a malformed entry (missing/wrongly-typed fields, or
|
||||
extra keys) is skipped rather than crashing the render path or being
|
||||
treated as runtime-stamped evidence.
|
||||
"""
|
||||
receipts: list[ToolReceipt] = []
|
||||
for message in messages:
|
||||
if not isinstance(message, ToolMessage):
|
||||
continue
|
||||
receipt = (message.additional_kwargs or {}).get(TOOL_RECEIPT_KEY)
|
||||
if not _is_valid_receipt(receipt):
|
||||
continue
|
||||
receipts.append(
|
||||
ToolReceipt(
|
||||
id=f"r{len(receipts) + 1}",
|
||||
tool_call_id=receipt["tool_call_id"],
|
||||
tool_name=receipt["tool_name"],
|
||||
status=receipt["status"],
|
||||
args_sha256=receipt["args_sha256"],
|
||||
output_sha256=receipt["output_sha256"],
|
||||
output_bytes=receipt["output_bytes"],
|
||||
created_at=receipt["created_at"],
|
||||
)
|
||||
)
|
||||
return receipts
|
||||
|
||||
|
||||
_RECEIPT_STR_FIELDS = ("tool_call_id", "tool_name", "status", "args_sha256", "output_sha256", "created_at")
|
||||
|
||||
|
||||
def _is_valid_receipt(receipt: object) -> bool:
|
||||
"""Structural check for a persisted receipt (types only, not provenance)."""
|
||||
if not isinstance(receipt, dict):
|
||||
return False
|
||||
if any(not isinstance(receipt.get(field), str) for field in _RECEIPT_STR_FIELDS):
|
||||
return False
|
||||
output_bytes = receipt.get("output_bytes")
|
||||
return isinstance(output_bytes, int) and not isinstance(output_bytes, bool)
|
||||
|
||||
|
||||
def render_tool_receipts(receipts: list[ToolReceipt], *, max_chars: int = _RENDER_CHAR_BUDGET) -> str:
|
||||
"""Render the receipt ledger as model-visible context (empty -> "")."""
|
||||
if not receipts:
|
||||
return ""
|
||||
lines = [
|
||||
"## Tool receipts (execution record)",
|
||||
"Cite receipt ids (e.g. [r1]) in your final report for every claim about an action you took.",
|
||||
# Anti-automation-bias (design rule 4): the ledger always states its
|
||||
# evidence boundary so the model never reads provenance as endorsement.
|
||||
"Execution evidence only — receipts record that a call happened and its status; they do not validate claim correctness or task acceptance.",
|
||||
]
|
||||
receipt_lines = [f"- [{receipt['id']}] {receipt['tool_name']} status={receipt['status']} args_sha256={receipt['args_sha256']} output_sha256={receipt['output_sha256']} bytes={receipt['output_bytes']}" for receipt in receipts]
|
||||
if len("\n".join([*lines, *receipt_lines])) <= max_chars:
|
||||
lines.extend(receipt_lines)
|
||||
else:
|
||||
omission = "- ... older receipts omitted (context budget)"
|
||||
retained: list[str] = []
|
||||
for line in reversed(receipt_lines):
|
||||
candidate = [*lines, omission, line, *retained]
|
||||
if len("\n".join(candidate)) > max_chars:
|
||||
break
|
||||
retained.insert(0, line)
|
||||
lines.extend([omission, *retained])
|
||||
rendered = "\n".join(lines)
|
||||
return rendered if len(rendered) <= max_chars else rendered[: max(0, max_chars - 4)] + "\n..."
|
||||
@ -0,0 +1,156 @@
|
||||
"""Stamp deterministic tool receipts and render the receipt ledger to the model.
|
||||
|
||||
Ordering contract (enforced by the build-time constraints in
|
||||
``deerflow.extensions.ordering.core_ordering_constraints``): this is the
|
||||
outermost ``wrap_tool_call`` layer — Guardrail, SandboxAudit, ReadBeforeWrite,
|
||||
and ToolProgress can short-circuit or rebuild results, and an inner receipt
|
||||
layer would silently gap the ledger on those. Normal results still carry a
|
||||
normalized ``deerflow_tool_meta`` status when stamped (ToolErrorHandling runs
|
||||
on the inner return path); short-circuit messages either self-stamp the meta
|
||||
or fall back to ``message.status`` in ``make_tool_receipt``.
|
||||
|
||||
The ledger injection mirrors DurableContextMiddleware: derived from the
|
||||
in-flight messages on every model call, appended as a hidden HumanMessage,
|
||||
never written back to state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import override
|
||||
|
||||
from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langchain.agents.middleware.types import ModelCallResult, ModelRequest, ModelResponse
|
||||
from langchain_core.messages import HumanMessage, ToolMessage
|
||||
from langgraph.prebuilt.tool_node import ToolCallRequest
|
||||
from langgraph.types import Command
|
||||
|
||||
from deerflow.agents.middlewares.message_utils import insert_after_leading_system_messages, is_genuine_user_message
|
||||
from deerflow.agents.middlewares.tool_receipt import TOOL_RECEIPT_KEY, extract_tool_receipts, make_tool_receipt, render_tool_receipts
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_RECEIPT_CONTEXT_KEY = "deerflow_tool_receipt_context"
|
||||
|
||||
|
||||
class ToolReceiptMiddleware(AgentMiddleware[AgentState]):
|
||||
"""Receipt layer: zero-LLM provenance for every tool call.
|
||||
|
||||
render_mode: 'always' renders the ledger on every model call (subagent
|
||||
chains — citations are produced there; without the ledger the subagent
|
||||
cannot cite and Layer 1 goes inert). 'delegation_only' renders only when
|
||||
the message stream contains a completed subagent result (lead chain —
|
||||
the one place the lead needs citation context), avoiding the always-on
|
||||
token tax in ordinary conversation turns.
|
||||
"""
|
||||
|
||||
state_schema = AgentState
|
||||
|
||||
def __init__(self, *, render_mode: str = "always") -> None:
|
||||
super().__init__()
|
||||
if render_mode not in {"always", "delegation_only"}:
|
||||
raise ValueError(f"Unknown render_mode: {render_mode}")
|
||||
self._render_mode = render_mode
|
||||
|
||||
def _stamp_message(self, message: ToolMessage, request: ToolCallRequest) -> None:
|
||||
try:
|
||||
kwargs = dict(message.additional_kwargs or {})
|
||||
# The receipt key is runtime-owned: always overwrite, never preserve
|
||||
# a pre-existing value — a tool could otherwise forge its own
|
||||
# "evidence" and have it rendered as runtime-stamped provenance.
|
||||
kwargs[TOOL_RECEIPT_KEY] = make_tool_receipt(request.tool_call, message)
|
||||
message.additional_kwargs = kwargs
|
||||
except Exception:
|
||||
# Never block tool execution — but a systematic stamping failure must
|
||||
# be visible, or the ledger silently goes incomplete and citations lie.
|
||||
logger.warning("Failed to stamp tool receipt", exc_info=True)
|
||||
|
||||
def _stamp(self, result: ToolMessage | Command, request: ToolCallRequest) -> ToolMessage | Command:
|
||||
if isinstance(result, ToolMessage):
|
||||
self._stamp_message(result, request)
|
||||
return result
|
||||
|
||||
update = result.update
|
||||
if not isinstance(update, dict):
|
||||
return result
|
||||
messages = update.get("messages", [])
|
||||
if isinstance(messages, ToolMessage):
|
||||
messages = [messages]
|
||||
if not isinstance(messages, (list, tuple)):
|
||||
return result
|
||||
|
||||
tool_call_id = str(request.tool_call.get("id") or "")
|
||||
for message in messages:
|
||||
if isinstance(message, ToolMessage) and str(message.tool_call_id) == tool_call_id:
|
||||
self._stamp_message(message, request)
|
||||
return result
|
||||
|
||||
@override
|
||||
def wrap_tool_call(
|
||||
self,
|
||||
request: ToolCallRequest,
|
||||
handler: Callable[[ToolCallRequest], ToolMessage | Command],
|
||||
) -> ToolMessage | Command:
|
||||
return self._stamp(handler(request), request)
|
||||
|
||||
@override
|
||||
async def awrap_tool_call(
|
||||
self,
|
||||
request: ToolCallRequest,
|
||||
handler: Callable[[ToolCallRequest], Awaitable[ToolMessage | Command]],
|
||||
) -> ToolMessage | Command:
|
||||
return self._stamp(await handler(request), request)
|
||||
|
||||
def _should_render(self, request: ModelRequest) -> bool:
|
||||
if self._render_mode == "always":
|
||||
return True
|
||||
# delegation_only: render only while a subagent result is being
|
||||
# processed (a task ToolMessage carries subagent_status in its
|
||||
# additional_kwargs — see subagents/status_contract.py). Scoped to the
|
||||
# current turn: only messages after the latest genuine user message
|
||||
# count, otherwise one completed delegation would keep the ledger
|
||||
# rendering on every later ordinary turn and defeat the token saving.
|
||||
# Without any genuine user message there is no turn boundary (e.g.
|
||||
# scheduled/internal invocations), so the whole stream is in scope.
|
||||
messages = list(request.messages)
|
||||
latest_user_index = -1
|
||||
for index, message in enumerate(messages):
|
||||
if is_genuine_user_message(message):
|
||||
latest_user_index = index
|
||||
turn_messages = messages[latest_user_index + 1 :] if latest_user_index >= 0 else messages
|
||||
for message in turn_messages:
|
||||
if isinstance(message, ToolMessage) and (message.additional_kwargs or {}).get("subagent_status"):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _inject(self, request: ModelRequest) -> ModelRequest:
|
||||
if not self._should_render(request):
|
||||
return request
|
||||
receipts = extract_tool_receipts(list(request.messages))
|
||||
ledger = render_tool_receipts(receipts)
|
||||
if not ledger:
|
||||
return request
|
||||
ledger_message = HumanMessage(
|
||||
content=ledger,
|
||||
additional_kwargs={"hide_from_ui": True, _RECEIPT_CONTEXT_KEY: True},
|
||||
)
|
||||
messages = insert_after_leading_system_messages(list(request.messages), [ledger_message])
|
||||
return request.override(messages=messages)
|
||||
|
||||
@override
|
||||
def wrap_model_call(
|
||||
self,
|
||||
request: ModelRequest,
|
||||
handler: Callable[[ModelRequest], ModelResponse],
|
||||
) -> ModelCallResult:
|
||||
return handler(self._inject(request))
|
||||
|
||||
@override
|
||||
async def awrap_model_call(
|
||||
self,
|
||||
request: ModelRequest,
|
||||
handler: Callable[[ModelRequest], Awaitable[ModelResponse]],
|
||||
) -> ModelCallResult:
|
||||
return await handler(self._inject(request))
|
||||
@ -8,7 +8,7 @@ Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** direc
|
||||
|
||||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path or file content signature changes. The signature includes file metadata and a content digest, so Gateway and LangGraph reads stay aligned with `config.yaml` edits even on object-store or network mounts where mtime can remain stale.
|
||||
|
||||
**Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`.
|
||||
**Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `verification.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`.
|
||||
|
||||
Infrastructure fields are **restart-required**. The authoritative list lives in `packages/harness/deerflow/config/reload_boundary.py::STARTUP_ONLY_FIELDS` and is mirrored by the standardised `"startup-only:"` prefix on the corresponding `Field(description=...)` in `AppConfig`, so IDE hover on those fields surfaces the reason inline (no need to context-switch into this table). Currently registered: `plugins`, `database`, `checkpointer`, `run_events`, `stream_bridge`, `sandbox`, `log_level`, `logging`, `channels`, `channel_connections`, `scheduler`, `mcp_tasks`, `run_ownership`. Adding a new restart-required field requires updating the registry; drift is pinned by `tests/test_reload_boundary.py`.
|
||||
|
||||
|
||||
@ -49,6 +49,7 @@ from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
|
||||
from deerflow.config.tool_output_config import ToolOutputConfig
|
||||
from deerflow.config.tool_progress_config import ToolProgressConfig
|
||||
from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict
|
||||
from deerflow.config.verification_config import VerificationConfig
|
||||
from deerflow.extensions.loader import ExtensionSpec
|
||||
|
||||
load_dotenv()
|
||||
@ -259,6 +260,7 @@ class AppConfig(BaseModel):
|
||||
)
|
||||
loop_detection: LoopDetectionConfig = Field(default_factory=LoopDetectionConfig, description="Loop detection middleware configuration")
|
||||
tool_progress: ToolProgressConfig = Field(default_factory=ToolProgressConfig, description="Tool progress state machine middleware configuration")
|
||||
verification: VerificationConfig = Field(default_factory=VerificationConfig, description="Subagent result verification (receipts, checklist, judge)")
|
||||
read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration")
|
||||
safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration")
|
||||
auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)")
|
||||
|
||||
@ -0,0 +1,26 @@
|
||||
"""Configuration for the subagent result-verification layers."""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class VerificationConfig(BaseModel):
|
||||
"""Receipt ledger, acceptance checklist, and selective judge settings."""
|
||||
|
||||
receipts_enabled: bool = Field(
|
||||
default=True,
|
||||
description="Stamp deterministic tool receipts on every tool result",
|
||||
)
|
||||
receipts_render_mode: Literal["always", "delegation_only"] = Field(
|
||||
default="delegation_only",
|
||||
description="Receipt-ledger rendering for the lead chain; subagent chains always render (citations are produced there). 'delegation_only' renders only while processing subagent results",
|
||||
)
|
||||
judge_enabled: bool = Field(
|
||||
default=False,
|
||||
description="Run a one-shot small-model review of completed subagent results that carry acceptance criteria",
|
||||
)
|
||||
judge_model_name: str | None = Field(
|
||||
default=None,
|
||||
description="Model for the selective judge; falls back to the parent model when unset",
|
||||
)
|
||||
@ -76,8 +76,12 @@ def core_ordering_constraints() -> tuple[OrderingConstraint, ...]:
|
||||
reported an empty sequence while iteration yielded the real constraints.
|
||||
Deferring the call instead of faking the value keeps one answer.
|
||||
"""
|
||||
from deerflow.agents.middlewares.read_before_write_middleware import ReadBeforeWriteMiddleware
|
||||
from deerflow.agents.middlewares.sandbox_audit_middleware import SandboxAuditMiddleware
|
||||
from deerflow.agents.middlewares.tool_error_handling_middleware import ToolErrorHandlingMiddleware
|
||||
from deerflow.agents.middlewares.tool_progress_middleware import ToolProgressMiddleware
|
||||
from deerflow.agents.middlewares.tool_receipt_middleware import ToolReceiptMiddleware
|
||||
from deerflow.guardrails.middleware import GuardrailMiddleware
|
||||
|
||||
return (
|
||||
OrderingConstraint(
|
||||
@ -85,4 +89,22 @@ def core_ordering_constraints() -> tuple[OrderingConstraint, ...]:
|
||||
inner=ToolErrorHandlingMiddleware,
|
||||
reason=("ToolProgressMiddleware reads deerflow_tool_meta in _update_state_from_result, so its wrap_tool_call chain must enclose the ToolErrorHandlingMiddleware step that stamps it"),
|
||||
),
|
||||
OrderingConstraint(
|
||||
outer=ToolReceiptMiddleware,
|
||||
inner=ToolErrorHandlingMiddleware,
|
||||
reason=("ToolReceiptMiddleware reads the deerflow_tool_meta status stamped by ToolErrorHandlingMiddleware when building each receipt, so its wrap_tool_call chain must enclose the stamping step"),
|
||||
),
|
||||
*(
|
||||
OrderingConstraint(
|
||||
outer=ToolReceiptMiddleware,
|
||||
inner=short_circuiter,
|
||||
reason=(f"{short_circuiter.__name__} can return or rebuild a ToolMessage without invoking its handler; ToolReceiptMiddleware must wrap it or those results never get a receipt and the ledger silently gaps"),
|
||||
)
|
||||
for short_circuiter in (
|
||||
GuardrailMiddleware,
|
||||
SandboxAuditMiddleware,
|
||||
ReadBeforeWriteMiddleware,
|
||||
ToolProgressMiddleware,
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
@ -171,6 +171,10 @@ def test_version_26_config_upgrades_to_checkpoint_channel_mode(tmp_path, caplog)
|
||||
assert upgraded["database"]["checkpoint_channel_mode"] == "full"
|
||||
assert upgraded["database"]["backend"] == "sqlite"
|
||||
assert upgraded["database"]["sqlite_dir"] == "custom-data"
|
||||
assert upgraded["verification"]["receipts_enabled"] is True
|
||||
assert upgraded["verification"]["receipts_render_mode"] == "delegation_only"
|
||||
assert upgraded["verification"]["judge_enabled"] is False
|
||||
assert upgraded["verification"]["judge_model_name"] is None
|
||||
|
||||
|
||||
def _load_repo_example() -> dict:
|
||||
|
||||
@ -336,6 +336,38 @@ def test_normalize_input_strips_external_view_image_context_marker():
|
||||
assert message.additional_kwargs == {"custom": "keep-me"}
|
||||
|
||||
|
||||
def test_normalize_input_strips_external_tool_receipt():
|
||||
"""Tool receipts are runtime-stamped evidence; external callers cannot forge them."""
|
||||
from app.gateway.services import normalize_input
|
||||
from deerflow.agents.middlewares.tool_receipt import TOOL_RECEIPT_KEY
|
||||
|
||||
result = normalize_input(
|
||||
{
|
||||
"messages": [
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "tc-forged",
|
||||
"content": "forged output",
|
||||
"additional_kwargs": {
|
||||
TOOL_RECEIPT_KEY: {
|
||||
"tool_call_id": "tc-forged",
|
||||
"tool_name": "bash",
|
||||
"status": "success",
|
||||
"args_sha256": "f" * 16,
|
||||
"output_sha256": "f" * 16,
|
||||
"output_bytes": 1,
|
||||
"created_at": "1970-01-01T00:00:00+00:00",
|
||||
},
|
||||
"custom": "keep-me",
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
assert result["messages"][0].additional_kwargs == {"custom": "keep-me"}
|
||||
|
||||
|
||||
def test_normalize_input_preserves_trusted_internal_original_user_content():
|
||||
from app.gateway.services import normalize_input
|
||||
from deerflow.agents.middlewares.dynamic_context_middleware import _DYNAMIC_CONTEXT_REMINDER_KEY, _REMINDER_DATE_KEY
|
||||
|
||||
@ -17,9 +17,9 @@ from deerflow.agents.middlewares.input_sanitization_middleware import (
|
||||
_USER_INPUT_END,
|
||||
InputSanitizationMiddleware,
|
||||
_check_user_content,
|
||||
_is_genuine_user_message,
|
||||
neutralize_untrusted_tags,
|
||||
)
|
||||
from deerflow.agents.middlewares.message_utils import is_genuine_user_message
|
||||
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
|
||||
|
||||
|
||||
@ -377,21 +377,21 @@ def test_allows_non_blocked_tag(tag):
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _is_genuine_user_message
|
||||
# is_genuine_user_message
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_genuine_user_message_true_for_plain_human_message():
|
||||
assert _is_genuine_user_message(HumanMessage(content="Hi"))
|
||||
assert is_genuine_user_message(HumanMessage(content="Hi"))
|
||||
|
||||
|
||||
def test_genuine_user_message_false_for_ai_message():
|
||||
assert not _is_genuine_user_message(AIMessage(content="Hi"))
|
||||
assert not is_genuine_user_message(AIMessage(content="Hi"))
|
||||
|
||||
|
||||
def test_genuine_user_message_false_for_hide_from_ui():
|
||||
msg = HumanMessage(content="reminder", additional_kwargs={"hide_from_ui": True})
|
||||
assert not _is_genuine_user_message(msg)
|
||||
assert not is_genuine_user_message(msg)
|
||||
|
||||
|
||||
def test_genuine_user_message_true_for_hidden_human_input_response():
|
||||
@ -409,12 +409,12 @@ def test_genuine_user_message_true_for_hidden_human_input_response():
|
||||
},
|
||||
},
|
||||
)
|
||||
assert _is_genuine_user_message(msg)
|
||||
assert is_genuine_user_message(msg)
|
||||
|
||||
|
||||
def test_genuine_user_message_false_for_legacy_summary_message():
|
||||
msg = HumanMessage(content="Here is a summary of the conversation", name="summary")
|
||||
assert not _is_genuine_user_message(msg)
|
||||
assert not is_genuine_user_message(msg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@ -164,7 +164,8 @@ def test_build_subagent_runtime_middlewares_threads_app_config_to_llm_middleware
|
||||
# + 1 SkillActivationMiddleware + 1 SkillToolPolicyMiddleware
|
||||
# + 1 SafetyFinishReasonMiddleware + 1 DurableContextMiddleware
|
||||
# + 1 SubagentDateContextMiddleware
|
||||
# + 1 SystemMessageCoalescingMiddleware (all enabled by default).
|
||||
# + 1 SystemMessageCoalescingMiddleware + 1 ToolReceiptMiddleware
|
||||
# (all enabled by default).
|
||||
from deerflow.agents.middlewares.durable_context_middleware import DurableContextMiddleware
|
||||
from deerflow.agents.middlewares.dynamic_context_middleware import SubagentDateContextMiddleware
|
||||
from deerflow.agents.middlewares.safety_finish_reason_middleware import SafetyFinishReasonMiddleware
|
||||
@ -173,11 +174,17 @@ def test_build_subagent_runtime_middlewares_threads_app_config_to_llm_middleware
|
||||
from deerflow.agents.middlewares.system_message_coalescing_middleware import SystemMessageCoalescingMiddleware
|
||||
from deerflow.agents.middlewares.token_budget_middleware import TokenBudgetMiddleware
|
||||
from deerflow.agents.middlewares.tool_output_budget_middleware import ToolOutputBudgetMiddleware
|
||||
from deerflow.agents.middlewares.tool_receipt_middleware import ToolReceiptMiddleware
|
||||
|
||||
assert len(middlewares) == 18
|
||||
assert len(middlewares) == 19
|
||||
assert isinstance(middlewares[0], FakeMiddleware) # InputSanitizationMiddleware stub
|
||||
assert isinstance(middlewares[1], ToolOutputBudgetMiddleware)
|
||||
assert any(isinstance(m, ToolErrorHandlingMiddleware) for m in middlewares)
|
||||
# The receipt layer wraps ToolErrorHandlingMiddleware so receipts read the
|
||||
# deerflow_tool_meta status it stamps (guard-enforced, like ToolProgress).
|
||||
receipt_idx = next(i for i, m in enumerate(middlewares) if isinstance(m, ToolReceiptMiddleware))
|
||||
error_idx = next(i for i, m in enumerate(middlewares) if isinstance(m, ToolErrorHandlingMiddleware))
|
||||
assert receipt_idx < error_idx
|
||||
# The token-budget backstop is attached by default so the cap engages (#3875).
|
||||
assert any(isinstance(m, TokenBudgetMiddleware) for m in middlewares)
|
||||
assert any(isinstance(m, SafetyFinishReasonMiddleware) for m in middlewares)
|
||||
|
||||
120
backend/tests/test_tool_receipt.py
Normal file
120
backend/tests/test_tool_receipt.py
Normal file
@ -0,0 +1,120 @@
|
||||
"""Tests for tool receipt core (deterministic verification layer)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from langchain_core.messages import AIMessage, ToolMessage
|
||||
|
||||
from deerflow.agents.middlewares.tool_receipt import (
|
||||
TOOL_RECEIPT_KEY,
|
||||
extract_tool_receipts,
|
||||
make_tool_receipt,
|
||||
render_tool_receipts,
|
||||
)
|
||||
from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY
|
||||
|
||||
|
||||
def _msg(content: str, *, tool_call_id: str, name: str = "write_file", meta_status: str = "success") -> ToolMessage:
|
||||
return ToolMessage(
|
||||
content=content,
|
||||
tool_call_id=tool_call_id,
|
||||
name=name,
|
||||
additional_kwargs={TOOL_META_KEY: {"status": meta_status}},
|
||||
)
|
||||
|
||||
|
||||
def _stamped_msg(content: str, *, tool_call_id: str, name: str, args: dict | None = None) -> ToolMessage:
|
||||
message = _msg(content, tool_call_id=tool_call_id, name=name)
|
||||
receipt = make_tool_receipt({"name": name, "id": tool_call_id, "args": args or {}}, message)
|
||||
message.additional_kwargs[TOOL_RECEIPT_KEY] = receipt
|
||||
return message
|
||||
|
||||
|
||||
def test_make_tool_receipt_hashes_args_and_output():
|
||||
receipt = make_tool_receipt(
|
||||
{"name": "write_file", "id": "tc-1", "args": {"path": "/tmp/a.txt", "content": "hello"}},
|
||||
_msg("ok", tool_call_id="tc-1"),
|
||||
)
|
||||
assert receipt["tool_call_id"] == "tc-1"
|
||||
assert receipt["tool_name"] == "write_file"
|
||||
assert receipt["status"] == "success"
|
||||
assert len(receipt["args_sha256"]) == 16
|
||||
assert len(receipt["output_sha256"]) == 16
|
||||
assert receipt["output_bytes"] == 2
|
||||
|
||||
|
||||
def test_make_tool_receipt_args_hash_is_key_order_invariant():
|
||||
first = make_tool_receipt({"name": "t", "id": "x", "args": {"a": 1, "b": 2}}, _msg("r", tool_call_id="x", name="t"))
|
||||
second = make_tool_receipt({"name": "t", "id": "x", "args": {"b": 2, "a": 1}}, _msg("r", tool_call_id="x", name="t"))
|
||||
assert first["args_sha256"] == second["args_sha256"]
|
||||
|
||||
|
||||
def test_make_tool_receipt_uses_meta_error_status():
|
||||
receipt = make_tool_receipt(
|
||||
{"name": "web_fetch", "id": "tc-2", "args": {"url": "https://x"}},
|
||||
_msg("Error: 404", tool_call_id="tc-2", name="web_fetch", meta_status="error"),
|
||||
)
|
||||
assert receipt["status"] == "error"
|
||||
|
||||
|
||||
def test_extract_assigns_sequential_ids_and_skips_unstamped():
|
||||
messages = [
|
||||
AIMessage(content="working", tool_calls=[{"name": "bash", "id": "tc-9", "args": {}}]),
|
||||
_msg("unstamped", tool_call_id="tc-0", name="bash"),
|
||||
_stamped_msg("first", tool_call_id="tc-1", name="write_file", args={"path": "/tmp/a"}),
|
||||
_stamped_msg("second", tool_call_id="tc-2", name="bash"),
|
||||
]
|
||||
receipts = extract_tool_receipts(messages)
|
||||
assert [r["id"] for r in receipts] == ["r1", "r2"]
|
||||
assert receipts[0]["tool_name"] == "write_file"
|
||||
assert receipts[1]["tool_name"] == "bash"
|
||||
|
||||
|
||||
def test_render_empty_and_budget():
|
||||
assert render_tool_receipts([]) == ""
|
||||
receipts = extract_tool_receipts([_stamped_msg("ok", tool_call_id="tc-1", name="write_file", args={"path": "/tmp/a"})])
|
||||
text = render_tool_receipts(receipts)
|
||||
assert "r1" in text and "write_file" in text and "success" in text
|
||||
# Anti-automation-bias (design rule 4): the ledger must always carry its evidence-boundary statement
|
||||
assert "do not validate claim correctness" in text
|
||||
assert len(render_tool_receipts(receipts, max_chars=10)) <= 14 # truncated + "\n..."
|
||||
|
||||
|
||||
def test_render_budget_keeps_newest_receipts_with_original_ids():
|
||||
receipts = extract_tool_receipts([_stamped_msg(f"result-{index}", tool_call_id=f"tc-{index}", name=f"tool-{index}") for index in range(1, 13)])
|
||||
|
||||
text = render_tool_receipts(receipts, max_chars=500)
|
||||
|
||||
assert len(text) <= 500
|
||||
assert "[r12] tool-12" in text
|
||||
assert "[r1] tool-1" not in text
|
||||
assert "older receipts omitted" in text
|
||||
|
||||
|
||||
def test_extract_skips_malformed_receipts():
|
||||
"""Persisted/foreign receipt payloads must not crash or enter the ledger."""
|
||||
good = _stamped_msg("ok", tool_call_id="tc-good", name="bash")
|
||||
malformed = []
|
||||
for payload in [
|
||||
"not-a-dict",
|
||||
{}, # missing every field
|
||||
{"tool_call_id": "tc-1"}, # partial shape
|
||||
{**make_tool_receipt({"name": "t", "id": "tc-2", "args": {}}, _msg("x", tool_call_id="tc-2", name="t")), "output_bytes": "2"}, # wrong type
|
||||
]:
|
||||
message = _msg("bad", tool_call_id="tc-bad", name="bash")
|
||||
message.additional_kwargs[TOOL_RECEIPT_KEY] = payload
|
||||
malformed.append(message)
|
||||
# A future-schema receipt (extra keys, valid core shape) must not crash
|
||||
# extraction either — its known fields are picked, unknown keys ignored.
|
||||
forward_compat = _msg("newer", tool_call_id="tc-newer", name="bash")
|
||||
forward_compat.additional_kwargs[TOOL_RECEIPT_KEY] = {
|
||||
**make_tool_receipt({"name": "bash", "id": "tc-newer", "args": {}}, forward_compat),
|
||||
"layer2_field": {"nested": True},
|
||||
}
|
||||
|
||||
receipts = extract_tool_receipts([*malformed, good, forward_compat])
|
||||
|
||||
assert [r["tool_call_id"] for r in receipts] == ["tc-good", "tc-newer"]
|
||||
assert [r["id"] for r in receipts] == ["r1", "r2"]
|
||||
# And the render path never sees a shape it can KeyError on.
|
||||
rendered = render_tool_receipts(receipts)
|
||||
assert "[r1] bash" in rendered and "[r2] bash" in rendered
|
||||
303
backend/tests/test_tool_receipt_middleware.py
Normal file
303
backend/tests/test_tool_receipt_middleware.py
Normal file
@ -0,0 +1,303 @@
|
||||
"""Tests for ToolReceiptMiddleware (stamping + context rendering)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
|
||||
from langgraph.types import Command
|
||||
|
||||
from deerflow.agents.middlewares.tool_receipt import TOOL_RECEIPT_KEY
|
||||
from deerflow.agents.middlewares.tool_receipt_middleware import ToolReceiptMiddleware
|
||||
from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY
|
||||
|
||||
|
||||
def _request(tool_name: str = "bash") -> SimpleNamespace:
|
||||
return SimpleNamespace(tool_call={"name": tool_name, "id": f"tc-{tool_name}", "args": {"cmd": "ls"}})
|
||||
|
||||
|
||||
def _result(request) -> ToolMessage:
|
||||
return ToolMessage(
|
||||
content="ok",
|
||||
tool_call_id=request.tool_call["id"],
|
||||
name=request.tool_call["name"],
|
||||
additional_kwargs={TOOL_META_KEY: {"status": "success"}},
|
||||
)
|
||||
|
||||
|
||||
def _stamped_message() -> ToolMessage:
|
||||
message = ToolMessage(content="ok", tool_call_id="tc-1", name="bash")
|
||||
message.additional_kwargs = {
|
||||
TOOL_META_KEY: {"status": "success"},
|
||||
TOOL_RECEIPT_KEY: {
|
||||
"tool_call_id": "tc-1",
|
||||
"tool_name": "bash",
|
||||
"status": "success",
|
||||
"args_sha256": "a" * 16,
|
||||
"output_sha256": "b" * 16,
|
||||
"output_bytes": 2,
|
||||
"created_at": "2026-08-03T00:00:00+00:00",
|
||||
},
|
||||
}
|
||||
return message
|
||||
|
||||
|
||||
def test_wrap_tool_call_stamps_receipt():
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = _request()
|
||||
result = middleware.wrap_tool_call(request, lambda req: _result(req))
|
||||
receipt = result.additional_kwargs[TOOL_RECEIPT_KEY]
|
||||
assert receipt["tool_name"] == "bash"
|
||||
assert receipt["status"] == "success"
|
||||
|
||||
|
||||
def test_wrap_tool_call_stamps_matching_messages_in_command():
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = _request("task")
|
||||
matching = _result(request)
|
||||
unrelated = ToolMessage(content="other", tool_call_id="tc-other", name="other")
|
||||
command = Command(update={"messages": [unrelated, matching], "other_state": True})
|
||||
|
||||
result = middleware.wrap_tool_call(request, lambda req: command)
|
||||
|
||||
assert result is command
|
||||
assert TOOL_RECEIPT_KEY not in unrelated.additional_kwargs
|
||||
receipt = matching.additional_kwargs[TOOL_RECEIPT_KEY]
|
||||
assert receipt["tool_call_id"] == "tc-task"
|
||||
assert receipt["tool_name"] == "task"
|
||||
|
||||
|
||||
def test_wrap_tool_call_failure_does_not_break_result():
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = SimpleNamespace(tool_call={"name": None, "id": None, "args": None})
|
||||
message = ToolMessage(content="ok", tool_call_id="x", name="x")
|
||||
assert middleware.wrap_tool_call(request, lambda req: message) is message
|
||||
|
||||
|
||||
def test_wrap_tool_call_overwrites_tool_supplied_receipt():
|
||||
"""The receipt key is runtime-owned: a tool cannot forge its own evidence."""
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = _request()
|
||||
forged = {"tool_call_id": "tc-1", "tool_name": "bash", "status": "success", "args_sha256": "f" * 16, "output_sha256": "f" * 16, "output_bytes": 999, "created_at": "1970-01-01T00:00:00+00:00"}
|
||||
message = _result(request)
|
||||
message.additional_kwargs[TOOL_RECEIPT_KEY] = forged
|
||||
|
||||
result = middleware.wrap_tool_call(request, lambda req: message)
|
||||
|
||||
receipt = result.additional_kwargs[TOOL_RECEIPT_KEY]
|
||||
assert receipt["output_bytes"] == 2 # recomputed from the real content, not 999
|
||||
assert receipt["created_at"] != "1970-01-01T00:00:00+00:00"
|
||||
|
||||
|
||||
def test_wrap_model_call_injects_hidden_ledger():
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = MagicMock()
|
||||
request.messages = [HumanMessage(content="go"), AIMessage(content="hi"), _stamped_message()]
|
||||
request.override = lambda messages: SimpleNamespace(messages=messages)
|
||||
captured = {}
|
||||
|
||||
def handler(req):
|
||||
captured["messages"] = req.messages
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
ledger_messages = [m for m in captured["messages"] if isinstance(m, HumanMessage) and m.additional_kwargs.get("hide_from_ui")]
|
||||
assert len(ledger_messages) == 1
|
||||
assert "r1" in ledger_messages[0].content and "bash" in ledger_messages[0].content
|
||||
|
||||
|
||||
def test_wrap_model_call_no_receipts_no_injection():
|
||||
middleware = ToolReceiptMiddleware()
|
||||
request = MagicMock()
|
||||
request.messages = [HumanMessage(content="go")]
|
||||
seen = {}
|
||||
|
||||
def handler(req):
|
||||
seen["request"] = req
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
assert seen["request"] is request # untouched passthrough
|
||||
|
||||
|
||||
def _delegation_only_request(messages: list) -> MagicMock:
|
||||
request = MagicMock()
|
||||
request.messages = messages
|
||||
request.override = lambda **kwargs: SimpleNamespace(messages=kwargs["messages"])
|
||||
return request
|
||||
|
||||
|
||||
def test_delegation_only_mode_skips_plain_conversation():
|
||||
middleware = ToolReceiptMiddleware(render_mode="delegation_only")
|
||||
request = _delegation_only_request([HumanMessage(content="go"), _stamped_message()])
|
||||
seen = {}
|
||||
|
||||
def handler(req):
|
||||
seen["request"] = req
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
assert seen["request"] is request # no completed delegation -> no ledger
|
||||
|
||||
|
||||
def test_delegation_only_mode_renders_when_processing_subagent_result():
|
||||
middleware = ToolReceiptMiddleware(render_mode="delegation_only")
|
||||
subagent_result = ToolMessage(
|
||||
content="Task Succeeded. Result: done [r1]",
|
||||
tool_call_id="tc-task",
|
||||
name="task",
|
||||
additional_kwargs={"subagent_status": "completed"},
|
||||
)
|
||||
request = _delegation_only_request([HumanMessage(content="go"), _stamped_message(), subagent_result])
|
||||
captured = {}
|
||||
|
||||
def handler(req):
|
||||
captured["messages"] = req.messages
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
ledger_messages = [m for m in captured["messages"] if isinstance(m, HumanMessage) and m.additional_kwargs.get("hide_from_ui")]
|
||||
assert len(ledger_messages) == 1 and "r1" in ledger_messages[0].content
|
||||
|
||||
|
||||
def test_delegation_only_mode_ignores_delegations_from_earlier_turns():
|
||||
"""A completed delegation must not keep the ledger rendering once a new
|
||||
genuine user turn has started — that would defeat the token-saving mode."""
|
||||
middleware = ToolReceiptMiddleware(render_mode="delegation_only")
|
||||
old_subagent_result = ToolMessage(
|
||||
content="Task Succeeded. Result: done [r1]",
|
||||
tool_call_id="tc-task",
|
||||
name="task",
|
||||
additional_kwargs={"subagent_status": "completed"},
|
||||
)
|
||||
request = _delegation_only_request([HumanMessage(content="first question"), _stamped_message(), old_subagent_result, AIMessage(content="report [r1]"), HumanMessage(content="unrelated follow-up")])
|
||||
seen = {}
|
||||
|
||||
def handler(req):
|
||||
seen["request"] = req
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
assert seen["request"] is request # old delegation is outside the current turn
|
||||
|
||||
|
||||
def test_delegation_only_mode_scopes_past_hidden_framework_messages():
|
||||
"""Hidden framework injections (reminders, the ledger itself) are not user
|
||||
turns: a subagent result after them still counts as the current turn."""
|
||||
middleware = ToolReceiptMiddleware(render_mode="delegation_only")
|
||||
reminder = HumanMessage(content="<system_reminder>todo</system_reminder>", additional_kwargs={"hide_from_ui": True})
|
||||
subagent_result = ToolMessage(
|
||||
content="Task Succeeded. Result: done",
|
||||
tool_call_id="tc-task",
|
||||
name="task",
|
||||
additional_kwargs={"subagent_status": "completed"},
|
||||
)
|
||||
request = _delegation_only_request([HumanMessage(content="go"), reminder, _stamped_message(), subagent_result])
|
||||
captured = {}
|
||||
|
||||
def handler(req):
|
||||
captured["messages"] = req.messages
|
||||
return MagicMock()
|
||||
|
||||
middleware.wrap_model_call(request, handler)
|
||||
ledger_messages = [m for m in captured["messages"] if isinstance(m, HumanMessage) and m.additional_kwargs.get("hide_from_ui") and "Tool receipts" in str(m.content)]
|
||||
assert len(ledger_messages) == 1
|
||||
|
||||
|
||||
def _build(app_config_dict: dict) -> list:
|
||||
from deerflow.agents.middlewares.tool_error_handling_middleware import _build_runtime_middlewares
|
||||
from deerflow.config.app_config import AppConfig
|
||||
|
||||
app_config = AppConfig.model_validate(app_config_dict)
|
||||
return _build_runtime_middlewares(app_config=app_config, include_uploads=False, include_dangling_tool_call_patch=False)
|
||||
|
||||
|
||||
def _tool_call_request(name: str, args: dict):
|
||||
from langgraph.prebuilt.tool_node import ToolCallRequest
|
||||
|
||||
runtime = MagicMock()
|
||||
runtime.context = {"thread_id": "t-test"}
|
||||
return ToolCallRequest(
|
||||
tool_call={"name": name, "args": args, "id": "call-1"},
|
||||
tool=None,
|
||||
state={"messages": []},
|
||||
runtime=runtime,
|
||||
)
|
||||
|
||||
|
||||
def _compose_tool_chain(chain: list, terminal):
|
||||
"""Compose wrap_tool_call handlers outer-first, mirroring the runtime stack."""
|
||||
handler = terminal
|
||||
for middleware in reversed(chain):
|
||||
next_handler = handler
|
||||
|
||||
def handler(req, mw=middleware, h=next_handler):
|
||||
return mw.wrap_tool_call(req, h)
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
def _tool_call_segment(middlewares: list, names: tuple[str, ...]) -> list:
|
||||
"""Extract the named wrap_tool_call middlewares in factory order."""
|
||||
return [m for m in middlewares if type(m).__name__ in names]
|
||||
|
||||
|
||||
def test_composed_chain_stamps_receipt_on_blocked_write():
|
||||
"""Read-before-write (default on) short-circuits a write with its own
|
||||
ToolMessage; the receipt layer must wrap that short-circuit or the ledger
|
||||
silently gaps (willem-bd review)."""
|
||||
middlewares = _build({"sandbox": {"use": "test"}})
|
||||
chain = _tool_call_segment(middlewares, ("ToolReceiptMiddleware", "ReadBeforeWriteMiddleware", "ToolErrorHandlingMiddleware"))
|
||||
assert [type(m).__name__ for m in chain] == ["ToolReceiptMiddleware", "ReadBeforeWriteMiddleware", "ToolErrorHandlingMiddleware"]
|
||||
# File exists with content v1 but was never read -> the write is blocked.
|
||||
chain[1]._content_reader = lambda _runtime, _path: "v1"
|
||||
terminal = MagicMock(return_value=ToolMessage(content="OK", tool_call_id="call-1", name="write_file"))
|
||||
|
||||
result = _compose_tool_chain(chain, terminal)(_tool_call_request("write_file", {"path": "/mnt/user-data/outputs/a.txt", "content": "v2"}))
|
||||
|
||||
terminal.assert_not_called()
|
||||
assert result.status == "error"
|
||||
receipt = (result.additional_kwargs or {}).get(TOOL_RECEIPT_KEY)
|
||||
assert receipt is not None, "blocked write must still get a receipt"
|
||||
assert receipt["tool_name"] == "write_file" and receipt["status"] == "error"
|
||||
|
||||
|
||||
def test_composed_chain_stamps_receipt_on_warn_rebuilt_result():
|
||||
"""SandboxAudit rebuilds the result ToolMessage when appending a medium-risk
|
||||
warning, dropping additional_kwargs; a receipt layer inside it would lose
|
||||
the stamp. Outer receipt re-stamps the rebuilt message."""
|
||||
middlewares = _build({"sandbox": {"use": "test"}})
|
||||
chain = _tool_call_segment(middlewares, ("ToolReceiptMiddleware", "SandboxAuditMiddleware", "ToolErrorHandlingMiddleware"))
|
||||
assert [type(m).__name__ for m in chain] == ["ToolReceiptMiddleware", "SandboxAuditMiddleware", "ToolErrorHandlingMiddleware"]
|
||||
terminal = MagicMock(return_value=ToolMessage(content="installed", tool_call_id="call-1", name="bash"))
|
||||
|
||||
result = _compose_tool_chain(chain, terminal)(_tool_call_request("bash", {"command": "pip install cowsay"}))
|
||||
|
||||
terminal.assert_called_once()
|
||||
assert "medium-risk" in str(result.content) # warn note appended
|
||||
receipt = (result.additional_kwargs or {}).get(TOOL_RECEIPT_KEY)
|
||||
assert receipt is not None, "warn-rebuilt result must still carry a receipt"
|
||||
assert receipt["tool_name"] == "bash"
|
||||
|
||||
|
||||
def test_factory_registers_receipt_middleware_outer_of_error_handling():
|
||||
middlewares = _build({"sandbox": {"use": "test"}})
|
||||
names = [type(m).__name__ for m in middlewares]
|
||||
assert "ToolReceiptMiddleware" in names
|
||||
assert names.index("ToolReceiptMiddleware") < names.index("ToolErrorHandlingMiddleware")
|
||||
|
||||
|
||||
def test_factory_registers_receipt_middleware_outer_of_short_circuiting_layers():
|
||||
"""Receipts must wrap every middleware that can return or rebuild a
|
||||
ToolMessage without invoking its handler, or the ledger silently gaps."""
|
||||
middlewares = _build({"sandbox": {"use": "test"}})
|
||||
names = [type(m).__name__ for m in middlewares]
|
||||
receipt_index = names.index("ToolReceiptMiddleware")
|
||||
for short_circuiter in ("SandboxAuditMiddleware", "ReadBeforeWriteMiddleware"):
|
||||
assert receipt_index < names.index(short_circuiter), f"ToolReceiptMiddleware must be outer of {short_circuiter}"
|
||||
|
||||
|
||||
def test_factory_omits_receipt_middleware_when_disabled():
|
||||
middlewares = _build({"sandbox": {"use": "test"}, "verification": {"receipts_enabled": False}})
|
||||
assert "ToolReceiptMiddleware" not in [type(m).__name__ for m in middlewares]
|
||||
36
backend/tests/test_verification_config.py
Normal file
36
backend/tests/test_verification_config.py
Normal file
@ -0,0 +1,36 @@
|
||||
"""Tests for the verification config section."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from deerflow.config.verification_config import VerificationConfig
|
||||
|
||||
|
||||
def test_defaults_receipts_on_judge_off():
|
||||
config = VerificationConfig()
|
||||
assert config.receipts_enabled is True
|
||||
assert config.receipts_render_mode == "delegation_only"
|
||||
assert config.judge_enabled is False
|
||||
assert config.judge_model_name is None
|
||||
|
||||
|
||||
def test_app_config_carries_verification_section():
|
||||
from deerflow.config.app_config import AppConfig
|
||||
|
||||
app_config = AppConfig.model_validate({"sandbox": {"use": "test"}})
|
||||
assert app_config.verification.receipts_enabled is True
|
||||
assert app_config.verification.judge_enabled is False
|
||||
|
||||
|
||||
def test_versioned_example_publishes_verification_section():
|
||||
example_path = Path(__file__).resolve().parents[2] / "config.example.yaml"
|
||||
example = yaml.safe_load(example_path.read_text(encoding="utf-8"))
|
||||
|
||||
assert example["config_version"] >= 34
|
||||
assert example["verification"] == {
|
||||
"receipts_enabled": True,
|
||||
"receipts_render_mode": "delegation_only",
|
||||
"judge_enabled": False,
|
||||
"judge_model_name": None,
|
||||
}
|
||||
@ -15,7 +15,7 @@
|
||||
# ============================================================================
|
||||
# Bump this number when the config schema changes.
|
||||
# Run `make config-upgrade` to merge new fields into your local config.yaml.
|
||||
config_version: 34
|
||||
config_version: 35
|
||||
|
||||
# ============================================================================
|
||||
# Logging
|
||||
@ -1488,6 +1488,21 @@ sandbox:
|
||||
# # Set `model` to use a different model (e.g., a local Ollama model for cost savings).
|
||||
# # The model name must match a name defined in the `models:` section above.
|
||||
|
||||
# ============================================================================
|
||||
# Tool Result Verification
|
||||
# ============================================================================
|
||||
# Deterministic receipts are stamped onto tool results and injected into the
|
||||
# model context so final reports can cite executed actions. Disable them only
|
||||
# if the extra provenance context is not wanted. The selective judge settings
|
||||
# are reserved for acceptance-criteria review and remain off by default.
|
||||
verification:
|
||||
receipts_enabled: true
|
||||
# Lead-chain ledger rendering: 'delegation_only' renders only while
|
||||
# processing subagent results (subagent chains always render).
|
||||
receipts_render_mode: "delegation_only"
|
||||
judge_enabled: false
|
||||
judge_model_name: null
|
||||
|
||||
# ============================================================================
|
||||
# ACP Agents Configuration
|
||||
# ============================================================================
|
||||
|
||||
@ -124,7 +124,7 @@ they resolve from the `secrets` map):
|
||||
|
||||
```yaml
|
||||
config: |
|
||||
config_version: 34
|
||||
config_version: 35
|
||||
models:
|
||||
- name: gpt-4
|
||||
use: langchain_openai:ChatOpenAI
|
||||
|
||||
@ -243,7 +243,7 @@ ingress:
|
||||
# -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never
|
||||
# inline literal secret values here. The default enables provisioner sandbox.
|
||||
config: |
|
||||
config_version: 34
|
||||
config_version: 35
|
||||
log_level: info
|
||||
|
||||
models: []
|
||||
@ -290,6 +290,12 @@ config: |
|
||||
memory:
|
||||
storage_path: memory.json
|
||||
|
||||
verification:
|
||||
receipts_enabled: true
|
||||
receipts_render_mode: "delegation_only"
|
||||
judge_enabled: false
|
||||
judge_model_name: null
|
||||
|
||||
# -- Tools configuration. The agent gets NO tools unless they're listed here
|
||||
# (BUILTIN_TOOLS only adds present_file + ask_clarification). The file/bash
|
||||
# tools run inside the AIO sandbox configured above. The web tools
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user