fix(middleware): keep the bash exit marker last after budget rewrites (#6354)

* fix(middleware): keep the bash exit marker last after budget rewrites

* fix(middleware): keep exit marker at exact-fit fallback budget; harden test cleanup

* docs(changelog): reference #6354 in the middleware entry
This commit is contained in:
wgwfn3 2026-10-06 20:44:49 +08:00 committed by GitHub
parent 1523a94551
commit eeb2c4dbcc
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 300 additions and 9 deletions

View File

@ -623,6 +623,18 @@ This release closes that milestone with **439 merged pull requests**.
### Fixed
- **middleware:** Tool-output budgeting no longer hides a failed shell exit from
subagent evidence. A bash result between `externalize_min_chars` (12,000) and
the sandbox limit (20,000) was replaced by a preview ending in its `Access:`
footer, so the trailing `Exit Code: N` was no longer last,
`_bash_evidence_status` fell back to `deerflow_tool_meta` (`success`), and a
failed `pytest` whose output still said `12 passed` could satisfy a
`tests_passed` acceptance criterion. `ToolOutputBudgetMiddleware` now
re-appends the original trailing `Exit Code: N` / `Command exited with code N`
after the preview, and the storage-unavailable fallback reserves it from the
`fallback_max_chars` budget the way sandbox truncation does. Only `bash` and
`bash_tool` results are affected; the persisted full output is unchanged.
([#6354])
- **make:** `make clean` now says what it deletes and refuses to run under a live
Docker Gateway. `make help` described it as cleaning up "temporary files", but
it deletes `backend/.deer-flow`: the local database, users, threads, uploads,
@ -8828,3 +8840,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
[#6347]: https://github.com/bytedance/deer-flow/pull/6347
[#6350]: https://github.com/bytedance/deer-flow/pull/6350
[#6351]: https://github.com/bytedance/deer-flow/pull/6351
[#6354]: https://github.com/bytedance/deer-flow/pull/6354

View File

@ -558,6 +558,15 @@
### 修复
- **中间件:** 工具输出预算不再让子智能体证据丢失失败的 shell 退出码。长度介于
`externalize_min_chars`(12,000)与沙箱上限(20,000)之间的 bash 结果会被替换为
以 `Access:` 页脚结尾的预览,结尾的 `Exit Code: N` 不再位于最后,
`_bash_evidence_status` 退回到报告 `success` 的 `deerflow_tool_meta`,输出里仍写着
`12 passed` 的失败 `pytest` 可能满足 `tests_passed` 验收条件。现在
`ToolOutputBudgetMiddleware` 会在预览之后重新追加原始结尾的 `Exit Code: N` /
`Command exited with code N`;存储不可用时的回退截断也像沙箱截断一样,从
`fallback_max_chars` 预算中为它预留位置。仅影响 `bash` 与 `bash_tool` 的结果,
落盘的完整输出保持不变。([#6354])
- **make:** `make clean` 现在会说明它删除的内容,并拒绝在运行中的 Docker Gateway 下执行。`make help`
此前称其清理"临时文件",但它实际删除 `backend/.deer-flow`:本地数据库、用户、线程、上传、记忆和密钥。
两套 Docker 栈都把该目录挂载进 `deer-flow-gateway` 容器,而 `make stop` 不会停止它,因此数据可能在
@ -7309,3 +7318,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子
[#6344]: https://github.com/bytedance/deer-flow/pull/6344
[#6350]: https://github.com/bytedance/deer-flow/pull/6350
[#6351]: https://github.com/bytedance/deer-flow/pull/6351
[#6354]: https://github.com/bytedance/deer-flow/pull/6354

View File

@ -50,7 +50,7 @@ strict providers reject.
1. **InputSanitizationMiddleware** - First, so it is the outermost `wrap_model_call` wrapper; every inner middleware (including LLM retries) sees sanitized messages. `additional_kwargs.original_user_content` is server-owned provenance: Gateway strips caller-supplied values for non-internal run requests, trusted IM calls may carry the string they captured before adding transport/file context, and the middleware replaces any non-string value before wrapping. Uploads and sanitization retain first-writer-wins only for validated strings. Caller markers are marked `untrusted_input`, never stripped; scope is every turn.
**KnowledgeScopeMiddleware** follows input sanitization: it exposes only Gateway-admitted execution scope, removes scope/display data from model messages, and blocks `knowledge_search` when disabled without reading storage or RAGFlow.
2. **ToolOutputBudgetMiddleware** - Budgets model-bound text/JSON with configured limits and exemptions; externalizes to thread outputs (`tool_output.storage_subdir`, default `.tool-results`, `TOOL_RESULTS_DIRNAME`) with a synopsis + `read_file` reference. Preserve media in mixed JSON results. Blob refs restore across hosts; hash/write failures use inline fallback. Internal outputs bypass workspace/delivery scans. Publication and model-only write elision: [contract](TOOL_OUTPUT_PERSISTENCE.md). Elision uses shared `tool_call_args`; retain `keep_recent_writes` newest writes. Controls: `elide_superseded_writes`, `superseded_write_min_chars`.
2. **ToolOutputBudgetMiddleware** - Budgets model-bound text/JSON with configured limits and exemptions; externalizes to thread outputs (`tool_output.storage_subdir`, default `.tool-results`, `TOOL_RESULTS_DIRNAME`) with a synopsis + `read_file` reference. Preserve media in mixed JSON results. Blob refs restore across hosts; hash/write failures use inline fallback. Internal outputs bypass workspace/delivery scans. Publication, write elision and bash exit marker: [contract](TOOL_OUTPUT_PERSISTENCE.md). Elision uses shared `tool_call_args`; retain `keep_recent_writes` newest writes. Controls: `elide_superseded_writes`, `superseded_write_min_chars`.
3. **ToolResultSanitizationMiddleware** - Neutralizes framework/injection tags (e.g. `<system-reminder>`) and boundary markers in *remote-content* tool results (`web_fetch`/`web_search`/`image_search`/`web_capture`) so attacker-controlled fetched pages cannot forge trusted framework context. Mirrors `InputSanitizationMiddleware`'s user-input guardrail for the other untrusted-content entry point; sits inner of `ToolOutputBudgetMiddleware` (neutralizes the raw output, then the budget truncates). Local tool output (bash/read_file) is left untouched. Scope is a name-based allowlist for the first-party web tools, plus every MCP-sourced tool via its `deerflow_mcp` metadata tag, so an MCP server naming its fetcher `fetch_url` is still covered
Result-rewriting middlewares between the raw callable boundary and the

View File

@ -43,6 +43,18 @@ configured fallback. Disabled budgets, raised limits and tool exemptions apply
before provider normalization. Pure text/media results without JSON retain their
existing pass-through behavior.
## Bash exit marker
The subagent executor reads a bash result's exit status only from its end
(`Exit Code: N`, or a whole-output `Command exited with code N`); with no marker
it falls back to `deerflow_tool_meta`, which reports `success` for a nonzero
exit. A model-visible rewrite of a `bash`/`bash_tool` result must therefore keep
the original trailing marker last: the preview re-appends it after the `Access:`
footer, and the storage-unavailable fallback reserves it from `max_chars` like
sandbox truncation. The local marker regex mirrors
`sandbox.tools._BASH_EXIT_MARKER_TAIL_RE` and a test pins them together. The
persisted output is the original bytes; other tools are not rewritten.
## Unclean shutdown
SIGKILL, OOM termination, and host failure bypass exception cleanup and can leave

View File

@ -29,6 +29,7 @@ import json
import logging
import os
import posixpath
import re
import shlex
import tempfile
import uuid
@ -81,6 +82,15 @@ _MAX_TOOL_OUTPUT_BLOB_BYTES = 64 * 1024 * 1024
# even when the operator disabled the ordinary disk-unavailable fallback.
_DURABLE_FAILURE_FALLBACK_MAX_CHARS = 30_000
# Mirrors ``sandbox.tools._BASH_EXIT_MARKER_TAIL_RE`` (pinned by a test). The
# subagent executor reads the exit status only from the end of a bash result
# (trailing ``Exit Code: N``, or whole-output ``Command exited with code N``),
# so a budget rewrite must keep the marker last or a failed command is harvested
# as success. Restricted to the same names ``executor._BASH_EVIDENCE_TOOL_NAMES``
# uses; other tools may print the phrase as ordinary text.
_BASH_EXIT_MARKER_TAIL_RE = re.compile(r"(?:\nExit Code: -?\d+|\n?Command exited with code -?\d+)\s*$")
_EXIT_MARKER_TOOL_NAMES = frozenset({"bash", "bash_tool"})
class _BudgetedContent(NamedTuple):
replacement: str
@ -352,6 +362,32 @@ def _externalize_to_sandbox(
# ---------------------------------------------------------------------------
def _canonical_exit_marker(content: str) -> str | None:
"""Return the trailing bash exit marker as a single line, or ``None``."""
match = _BASH_EXIT_MARKER_TAIL_RE.search(content)
if match is None:
return None
return match.group(0).strip()
def _keep_exit_marker_last(original: str, rewritten: str, *, tool_name: str) -> str:
"""Re-append the original trailing exit marker after a budget rewrite.
Preview construction always ends with an Access footer, so the harvest
regex (``Exit Code: N`` anchored at the end) would otherwise miss the
status and fall back to ``deerflow_tool_meta``'s success. Restricted to
bash-family tools; other tools' text is left alone.
"""
if tool_name not in _EXIT_MARKER_TOOL_NAMES:
return rewritten
marker = _canonical_exit_marker(original)
if marker is None:
return rewritten
if rewritten.rstrip().endswith(marker):
return rewritten
return f"{rewritten.rstrip()}\n{marker}"
def _build_preview(
content: str,
*,
@ -361,13 +397,14 @@ def _build_preview(
tail_chars: int,
) -> str:
"""Build a typed synopsis preview with a file reference for externalized output."""
return render_tool_output_preview(
preview = render_tool_output_preview(
content,
tool_name=tool_name,
virtual_path=virtual_path,
head_chars=head_chars,
tail_chars=tail_chars,
)
return _keep_exit_marker_last(content, preview, tool_name=tool_name)
def _build_fallback(
@ -381,26 +418,45 @@ def _build_fallback(
"""Build a head+tail truncation when disk persistence is unavailable.
The returned string is guaranteed to be no longer than *max_chars*.
A trailing bash exit marker (``Exit Code: N`` / ``Command exited with
code N``) is reserved from the budget the way sandbox truncation does,
so harvest can still recover the shell status. Only when *max_chars*
is smaller than the marker line (including its leading newline) does
the marker drop.
"""
total = len(content)
if max_chars <= 0 or total <= max_chars:
return content
preserved = ""
body = content
if tool_name in _EXIT_MARKER_TOOL_NAMES:
match = _BASH_EXIT_MARKER_TAIL_RE.search(content)
if match is not None:
line = f"\n{match.group(0).strip()}"
if len(line) <= max_chars:
body = content[: match.start()]
preserved = line
max_chars -= len(preserved)
if len(body) <= max_chars:
return body + preserved
total = len(body)
marker_template = "\n\n[... {n} chars omitted from {tn} output. Persistent storage unavailable. Consider narrowing the query or using more specific parameters.]\n\n"
marker_overhead = len(marker_template.format(n=total, tn=tool_name))
if marker_overhead >= max_chars:
return content[:max_chars]
return body[:max_chars] + preserved
budget = max_chars - marker_overhead
effective_head = min(head_chars, budget)
effective_tail = min(tail_chars, max(0, budget - effective_head))
head_end = _snap_to_line_boundary(content, min(effective_head, total))
tail_start = _snap_start_to_line_boundary(content, max(head_end, total - effective_tail))
head_end = _snap_to_line_boundary(body, min(effective_head, total))
tail_start = _snap_start_to_line_boundary(body, max(head_end, total - effective_tail))
head = content[:head_end]
tail = content[tail_start:] if tail_start < total else ""
head = body[:head_end]
tail = body[tail_start:] if tail_start < total else ""
omitted = total - len(head) - len(tail)
marker = marker_template.format(n=omitted, tn=tool_name)
@ -408,7 +464,7 @@ def _build_fallback(
parts = [head, marker]
if tail:
parts.append(tail)
return "".join(parts)
return "".join(parts) + preserved
# ---------------------------------------------------------------------------

View File

@ -3,7 +3,8 @@
Covers: pass-through, disk externalization, fallback truncation, UTF-8
boundaries, Command results, model-request history patching, superseded
write_file payload elision (issue #5328), config variations, exempt tools,
per-tool overrides, edge cases, and both sync/async code paths.
per-tool overrides, bash exit-marker preservation after budget rewrites,
edge cases, and both sync/async code paths.
"""
from __future__ import annotations
@ -11,11 +12,13 @@ from __future__ import annotations
import contextlib
import csv
import hashlib
import importlib.util
import io
import json
import os
import pathlib
import re
import sys
import tempfile
from types import SimpleNamespace
@ -25,12 +28,14 @@ from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langgraph.types import Command
from deerflow.agents.middlewares.tool_output_budget_middleware import (
_BASH_EXIT_MARKER_TAIL_RE,
TOOL_OUTPUT_BLOB_KEY,
ToolOutputBudgetMiddleware,
_build_fallback,
_build_preview,
_effective_trigger,
_externalize,
_keep_exit_marker_last,
_message_text,
_needs_budget,
_patch_model_messages,
@ -99,6 +104,22 @@ def _tm(content: str = "ok", name: str = "tool", tool_call_id: str = "tc-1") ->
return ToolMessage(content=content, name=name, tool_call_id=tool_call_id)
def _pytest_like_output(exit_line: str = "Exit Code: 1") -> str:
"""Pytest-shaped bash output in the default 12k–20k budget window."""
return "============================= test session starts =============================\n" + ("collected 12 items\n" + "." * 12 + "\n") * 500 + f"12 passed in 1.23s\n{exit_line}"
def _bash_tm(content: str, name: str = "bash") -> ToolMessage:
return ToolMessage(
content=content,
name=name,
tool_call_id="tc-1",
id="tool-msg-1",
artifact={"k": "v"},
additional_kwargs={"deerflow_tool_meta": {"status": "success", "source": "normalized"}},
)
# ===========================================================================
# Unit tests for helper functions
# ===========================================================================
@ -797,6 +818,185 @@ class TestBuildFallback:
assert len(result) <= 50
# ===========================================================================
# Bash exit-marker preservation (harvest reads Exit Code: N from the end)
# ===========================================================================
class TestBashExitMarkerPreservation:
"""Budget rewrites must keep a trailing bash exit marker last.
The subagent executor harvests status with ``Exit Code: N\\s*$``. Preview
construction always ends with an Access footer, so without this the
marker is no longer last and a failed command is harvested as success.
"""
def test_keep_exit_marker_last_appends_for_bash(self):
original = "out\nExit Code: 1"
rewritten = "preview\nAccess:\n- Use read_file"
assert _keep_exit_marker_last(original, rewritten, tool_name="bash") == "preview\nAccess:\n- Use read_file\nExit Code: 1"
def test_keep_exit_marker_last_is_noop_for_other_tools(self):
original = "out\nExit Code: 1"
rewritten = "preview\nAccess:\n- Use read_file"
assert _keep_exit_marker_last(original, rewritten, tool_name="web_fetch") == rewritten
def test_keep_exit_marker_last_is_noop_when_already_last(self):
original = "out\nExit Code: 0"
rewritten = "preview\nExit Code: 0"
assert _keep_exit_marker_last(original, rewritten, tool_name="bash") == rewritten
def test_wrap_tool_call_keeps_failed_exit_marker_last(self):
content = _pytest_like_output("Exit Code: 1")
assert 12_000 < len(content) <= 20_000
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
msg = _bash_tm(content)
result = mw.wrap_tool_call(_make_request(tool_name="bash", outputs_path=tmpdir), lambda _: msg)
assert isinstance(result, ToolMessage)
assert result is not msg
assert str(result.content).rstrip().endswith("Exit Code: 1")
assert "Access:" in result.content
assert "read_file" in result.content
assert result.additional_kwargs["deerflow_tool_meta"] == {"status": "success", "source": "normalized"}
assert result.artifact == {"k": "v"}
assert result.id == "tool-msg-1"
@pytest.mark.anyio
async def test_awrap_tool_call_keeps_failed_exit_marker_last(self):
content = _pytest_like_output("Exit Code: 1")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
msg = _bash_tm(content)
async def handler(_):
return msg
result = await mw.awrap_tool_call(_make_request(tool_name="bash", outputs_path=tmpdir), handler)
assert str(result.content).rstrip().endswith("Exit Code: 1")
assert "Access:" in result.content
assert result.artifact == {"k": "v"}
assert result.id == "tool-msg-1"
def test_wrap_tool_call_keeps_successful_exit_marker_last(self):
content = _pytest_like_output("Exit Code: 0")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
result = mw.wrap_tool_call(_make_request(tool_name="bash", outputs_path=tmpdir), lambda _: _bash_tm(content))
assert str(result.content).rstrip().endswith("Exit Code: 0")
assert "Access:" in result.content
def test_wrap_tool_call_without_marker_ends_at_access_footer(self):
content = _pytest_like_output("12 failed in 1.23s")
assert not content.rstrip().endswith("Exit Code: 1")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
result = mw.wrap_tool_call(_make_request(tool_name="bash", outputs_path=tmpdir), lambda _: _bash_tm(content))
assert "Access:" in result.content
assert "read_file" in result.content
assert not str(result.content).rstrip().endswith("Exit Code: 1")
assert "Exit Code:" not in str(result.content).rsplit("Access:", 1)[-1]
def test_non_bash_tool_does_not_promote_exit_marker(self):
content = _pytest_like_output("Exit Code: 1")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
result = mw.wrap_tool_call(
_make_request(tool_name="web_fetch", outputs_path=tmpdir),
lambda _: _bash_tm(content, name="web_fetch"),
)
assert "Access:" in result.content
assert not str(result.content).rstrip().endswith("Exit Code: 1")
def test_bash_tool_alias_also_preserves_marker(self):
content = _pytest_like_output("Exit Code: 1")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
result = mw.wrap_tool_call(
_make_request(tool_name="bash_tool", outputs_path=tmpdir),
lambda _: _bash_tm(content, name="bash_tool"),
)
assert str(result.content).rstrip().endswith("Exit Code: 1")
def test_fallback_tiny_budget_keeps_marker_and_stays_within_max_chars(self):
content = _pytest_like_output("Exit Code: 1")
result = _build_fallback(content, tool_name="bash", max_chars=80, head_chars=8000, tail_chars=3000)
assert len(result) <= 80
assert result.rstrip().endswith("Exit Code: 1")
def test_fallback_default_budget_leaves_payload_unchanged(self):
content = _pytest_like_output("Exit Code: 1")
result = _build_fallback(content, tool_name="bash", max_chars=30_000, head_chars=8000, tail_chars=3000)
assert result == content
assert result.rstrip().endswith("Exit Code: 1")
def test_fallback_never_exceeds_max_chars_with_exit_marker(self):
content = _pytest_like_output("Exit Code: 1")
marker_line = "\nExit Code: 1"
for max_chars in [10, len(marker_line) - 1, len(marker_line), 20, 80, 200, 500, 1000, 5000, 20000]:
result = _build_fallback(content, tool_name="bash", max_chars=max_chars, head_chars=max_chars // 2, tail_chars=max_chars // 4)
assert len(result) <= max_chars, f"max_chars={max_chars}: got {len(result)}"
if max_chars >= len(marker_line):
assert result.rstrip().endswith("Exit Code: 1"), f"max_chars={max_chars}"
else:
assert not result.rstrip().endswith("Exit Code: 1")
def test_exit_marker_regex_matches_sandbox_truncation_tail_shapes(self):
from deerflow.sandbox.tools import _BASH_EXIT_MARKER_TAIL_RE as sandbox_re
for content in ("out\nExit Code: 1", "out\nExit Code: -9 \n", "out\nCommand exited with code 3", "Command exited with code 3"):
ours = _BASH_EXIT_MARKER_TAIL_RE.search(content)
theirs = sandbox_re.search(content)
assert ours is not None and theirs is not None
assert ours.start() == theirs.start(), content
def test_budgeted_failed_pytest_is_harvested_as_error_not_pass(self):
"""End-to-end: wrap_tool_call rewrite → harvest status=error → leaf holds=False.
``tests/conftest.py`` mocks ``deerflow.subagents.executor``, so the
production harvest helpers are loaded under a unique module name.
"""
path = pathlib.Path(__file__).parents[1] / "packages/harness/deerflow/subagents/executor.py"
spec = importlib.util.spec_from_file_location("_budget_exit_marker_executor", path)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
try:
sys.modules[spec.name] = module
spec.loader.exec_module(module)
from deerflow.subagents import acceptance_checks
content = _pytest_like_output("Exit Code: 1")
with tempfile.TemporaryDirectory() as tmpdir:
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig())
result = mw.wrap_tool_call(_make_request(tool_name="bash", outputs_path=tmpdir), lambda _: _bash_tm(content))
assert str(result.content).rstrip().endswith("Exit Code: 1")
cmd = "pytest -q"
ai = AIMessage(
content="",
tool_calls=[{"name": "bash", "args": {"command": cmd}, "id": "tc-1", "type": "tool_call"}],
)
state = {"messages": [HumanMessage(content="task"), ai, result]}
executions = module._harvest_bash_executions(state)
assert executions, "harvest returned no bash executions"
for entry in executions:
entry["shell_persistent"] = False
latest = executions[-1]
assert latest["status"] == "error"
assert latest["status_marker"] == "Exit Code: 1"
leaf = acceptance_checks._check_tests_passed_leaf(cmd, executions)
assert leaf["checked"] is True
assert leaf["holds"] is False
finally:
sys.modules.pop(spec.name, None)
# Absent when exec_module failed part-way; don't mask that error.
shutdown = getattr(module, "_shutdown_isolated_subagent_loop", None)
if shutdown is not None:
shutdown()
# ===========================================================================
# Middleware integration tests — wrap_tool_call
# ===========================================================================