mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-14 16:08:41 +00:00
a06a6fed7e
* feat(harness): deterministic acceptance checklist for subagent delegations (RFC #4651, layer 2) PR4 of RFC #4651: check lead-supplied acceptance_criteria in code when a subagent completes, so objectively checkable requirements can never be silently passed by a self-report. - subagents/acceptance_checks.py: deterministic leaf families — file:<path> exists|non-empty and file_written:<path> read through read_current_file_content scoped to the shared thread workspace; the read uses the sandbox-native virtual path form (the local read validator and provider mount tables resolve /mnt/user-data/... paths, not host paths); the scope decision canonicalizes with realpath on the local sandbox so workspace symlinks cannot escape into uploads; a remote provider's "Error: ..." return string is normalized to a failed check (provider-typed via is_local_sandbox); a UnicodeDecodeError marks a binary deliverable as existing and non-empty; out-of-scope paths degrade to UNVERIFIED. tests_passed:<command> anchors to a matching recorded bash execution with status=success and a test-summary shape; matching is shell-structure aware with control-flow attribution (span must end at the last segment with provable execution), negating-option values are ineligible evidence and a target negated anywhere in the command degrades the match, extra flags must be selection-preserving, extra positionals widen only after a path-scoped criterion, truncated commands degrade via command_truncated, the summary shape is read only from output attributable to the matched segment (preceding segments provably silent by invocation form), and pass shapes require a nonzero passed count. Criterion text is neutralized with neutralize_untrusted_tags before storage/rendering. Anything else renders UNVERIFIED, never silently passed. - executor: accumulate bounded bash command/output evidence per streamed chunk (merged by tool_call_id, newest-capped) so subagent summarization compacting earlier messages cannot erase a recorded execution; the recorded status is the actual shell exit status parsed from the output's exit marker (signed codes included; the remote Command exited with code N form is accepted only as the whole trimmed output), falling back to deerflow_tool_meta only when no marker exists. - sandbox providers: e2b/opensandbox/tenki/boxlite append the LocalSandbox-style "Exit Code: N" marker on nonzero exit even with non-empty output; aio propagates the SDK's structured exit_code on both exec paths the same way; local timeouts append Exit Code: 124; and _truncate_bash_output always preserves a trailing exit marker (signed included) inside its budget, with a 32-char floor raising any smaller configured limit, so the actual shell outcome always survives in the output text. - task_tool: run the checklist offloaded (asyncio.to_thread) on the completed branch, failure-isolated; stamp the verdict into result metadata and render the per-criterion section into the model-visible result text. - status contract: additive subagent_acceptance_verdict transport with read-side structural validation. - delegation ledger: entry carries the verdict and renders a compact acceptance segment; gateway strips caller-forged verdicts from both ledger entries and message metadata, like the citation verdict. - blocking-IO anchor pins the offload (teeth proven red->green); leaf read errors catch only OSError/SandboxError so unexpected errors reach the task-tool-level isolation instead of being mislabeled. * fix(harness): close acceptance evidence gaps from review (RFC #4651 PR4) - negating options: overlap with a matched criterion target is now checked by path/nodeid prefix, not exact token equality — excluding a sub-path of the criterion's selection (pytest tests --deselect tests/unit/test_auth.py) degrades to UNVERIFIED instead of holds - output attribution: any redirection token in the matched final segment makes the recorded tail non-attributable (> / >> / 2> are word characters to the parser, so redirection was invisible to the matcher) - silent-source allowlist narrowed from any *activate suffix to the */bin/activate shape - status_contract docstring: restore the shared-fixture sentence and note subagent_acceptance_verdict is deliberately outside the fixture - executor: update_bash_executions publishes [] (stream carried no bash-family calls) instead of collapsing it into None, mirroring update_tool_receipts * fix(harness): close acceptance residual gaps from re-review (RFC #4651 PR4) - tests_passed: add error outcomes to the fail shapes — "4 passed, 1 error" and pytest's "ERROR <nodeid>" short summary no longer satisfy the pass shape when the exit status is swallowed (|| true) or absent; zero-error counts stay clean. - file leaves: bound the deliverable read — a "wc -c" shell size probe answers files above 50k bytes without loading ~2x their size, honoring the host-bash kill switch and falling back to the full read on any non-integer rendering, so verdicts never get less sound. - executor: record the exit marker text as status_marker on harvested bash evidence; the leaf detail now reports the marker actually seen instead of asserting a failure indistinguishable from the command's own trailing text. - extend the blocking-IO anchor to drive the probe branch inside the offload; teeth re-verified red->green. * fix(harness): close acceptance forgery and bound gaps from P2 re-review (RFC #4651 PR4) - file leaves: never read unbounded — size is established first (os.stat on the validated local host path, so the host-bash-disabled configuration needs no shell; a guarded wc -c on remote providers that renders missing/unreadable in its own words). Above the 50k cap the leaf answers from the size alone, at/below it the full read runs, and an unestablishable size degrades to UNVERIFIED instead of an unlimited fallback read. - output attribution: source/. prefixes are never provably silent — a crafted */bin/activate path shape says nothing about what the script prints, so sourced segments can no longer lend a passing summary. - executable identity: an explicitly path-spelled criterion now requires the same normalized executable path; the basename rule stays only for deliberately bare criterion commands. * fix(harness): run acceptance size probe outside subagent-controlled state (RFC #4651 PR4) - remote probe no longer runs in the sandbox's persistent shell: a fresh env -i /bin/sh with absolute-path stat/realpath (poisoned functions, aliases, PATH, exported functions, IFS, locale cannot steer it), plus a marker env routing AIO onto a fresh per-call bash.exec session. - metadata-only: stat never opens content, so a FIFO deliverable cannot block the parent for the provider's idle timeout; non-regular files (fifo/dir/symlink) degrade to UNVERIFIED. - containment canonicalized against the literal mount root: a final-component symlink or a swapped parent directory (root included) cannot redirect the check outside shared storage; unprovable layouts degrade to UNVERIFIED. * fix(harness): canonicalize probe containment against the canonical mount root (RFC #4651 PR4) Literal-root equality made every remote file leaf permanently UNVERIFIED on e2b and Tenki, which realize /mnt/user-data as a symlink to the home dir by default (e2b bootstrap 'sudo ln -sfn', Tenki best-effort symlink). Containment now compares the file's realpath against the mount root's realpath — exactly what the provider's own read path resolves, so probe and read-back stay consistent; final-component symlinks stay rejected by the non-dereferencing stat, and an intermediate dir-link escape under a sane root still lands ESCAPED. The inner script is a module constant and the suite now executes the composed probe for real against on-disk layouts (real dir, symlinked prefix, final symlink, fifo, missing, dir-link escape), which the canned-output stub could not see. * fix(harness): close bare-criterion negation and CDPATH summary channels (RFC #4651 PR4) - matching: a criterion with no positional selection target (bare pytest, make test) stands for the runner's default selection, so ANY negating option (--ignore/--deselect/...) makes the recorded run a different selection — unprovable. The overlap guard only sees consumed criterion tokens, which a bare criterion does not have; scoped criteria keep the unrelated-exclusion behavior. - attribution: cd is no longer blanket-silent — CDPATH makes cd print the resolved (subagent-chosen) destination and the pass shapes match as substrings, so one mkdir 'all tests passed' plus an export minted a pass for any quiet command. A cd argument or CDPATH= value (export or leading assignment) carrying any summary shape makes the segment non-silent; shape-free cd dir wrappers keep matching. - docs: _truncate_bash_output states the effective 32-char floor (the guarantee previously read as an unconditional max_chars bound). * fix(harness): close env-assignment and expansion channels in acceptance matching (RFC #4651 PR4) Self-audit in the shape of the last review rounds — channels the matcher classified as accounted-for that can change what runs, narrow the selection, or lend the summary text: - env assignments are no longer blanket-stripped: only an allowlist of inert display/CI knobs (CI, NO_COLOR, PY_COLORS, ...) may prefix a matched span, and a non-allowlisted assignment in any preceding segment (pure-assignment or export NAME=) is state pollution — PATH redirects the executable, LD_PRELOAD/PYTHONPATH/NODE_OPTIONS inject code, PYTEST_ADDOPTS/GOFLAGS/MAKEFILES inject selection-changing inputs, BASH_ENV runs arbitrary shell startup. All degrade to unprovable. - runtime expansions: any span token carrying /$( )/backticks, any negating-option value carrying an expansion or glob (unknown excluded set), and any extra executed token carrying glob metacharacters (crafted option-looking filenames narrow invisibly) are unprovable. Criterion-side globs stay self-consistent (literal match). - cd: an argument carrying a runtime expansion or glob is non-silent (unknown destination, unknown print); CDPATH= assignments are now handled as state pollution at the match layer, subsuming the value-shape special case. * fix(harness): persistent-shell evidence, exact env sets, option-arity scoping (RFC #4651 PR4) - tests_passed: on a persistent-shell provider (new Sandbox.persistent_shell_sessions capability, set by AioSandbox) every leaf degrades to UNVERIFIED — any earlier call in the shared session could have mutated the state the clean-looking run executed in, and only a fresh controlled session (RFC section 6 verifier) can prove otherwise. The flag is read from the provider registry without acquiring a sandbox. - env assignments: the allowlist is gone — no variable is provably inert across repositories (CI/DEBUG are routinely read by tests). The span's assignment prefix must equal the criterion's exactly (values included, order-insensitive); any assignment or export NAME= in a preceding segment is state pollution. - scoping: positional targets are now read by option arity, so a path embedded in an option (--basetemp=/tmp/p, --junitxml=/tmp/r.xml) never counts as a selection target and an extra positional after such a criterion narrows the default selection it denotes. * fix(harness): stamp shell provenance at harvest, close export/unset and arity gaps (RFC #4651 PR4) * fix(harness): split physical newlines as shell separators in acceptance matching (RFC #4651 PR4) * fix(harness): scope cd wrappers to thread data roots, pin accepted boundaries (RFC #4651 PR4) * fix(harness): preserve criterion connectors, prove file_written readable, fail-closed shell capability (RFC #4651 PR4) * fix(harness): compare only the connector prefix, tolerate trailing criterion semicolons (RFC #4651 PR4) * fix(harness): preserve continuation-line operators, keep ./-spelled executable identity (RFC #4651 PR4) * fix(harness): render criteria single-line so a multiline criterion cannot inject a forged checklist line (RFC #4651 PR4) * fix(harness): reject parent-traversal executable tokens in acceptance matching (RFC #4651 PR4) * fix(harness): reject parent-traversal negated values in acceptance matching (RFC #4651 PR4)
277 lines
11 KiB
Python
277 lines
11 KiB
Python
"""Unit tests for tool output truncation functions.
|
|
|
|
These functions truncate long tool outputs to prevent context window overflow.
|
|
- _truncate_bash_output: middle-truncation (head + tail), for bash tool
|
|
- _truncate_read_file_output: head-truncation, for read_file tool
|
|
- _truncate_ls_output: head-truncation, for ls tool
|
|
"""
|
|
|
|
from deerflow.sandbox.tools import _truncate_bash_output, _truncate_ls_output, _truncate_read_file_output
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# _truncate_bash_output
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestTruncateBashOutput:
|
|
def test_short_output_returned_unchanged(self):
|
|
output = "hello world"
|
|
assert _truncate_bash_output(output, 20000) == output
|
|
|
|
def test_trailing_exit_marker_survives_truncation(self):
|
|
"""PR review: a failing command's pass-shaped text must not lose the
|
|
authoritative exit marker to truncation — evidence consumers parse it."""
|
|
output = "1 passed\n" + "M" * 30000 + "\nExit Code: 1"
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert result.endswith("\nExit Code: 1")
|
|
assert len(result) <= 20000
|
|
|
|
def test_trailing_exit_marker_survives_a_tiny_budget(self):
|
|
output = "x" * 5000 + "\nExit Code: 124"
|
|
result = _truncate_bash_output(output, 100)
|
|
assert result.endswith("\nExit Code: 124")
|
|
assert len(result) <= 100
|
|
|
|
def test_command_exited_with_code_form_is_preserved(self):
|
|
output = "M" * 5000 + "\nCommand exited with code 3"
|
|
result = _truncate_bash_output(output, 1000)
|
|
assert result.endswith("Command exited with code 3")
|
|
|
|
def test_signed_signal_exit_marker_is_preserved(self):
|
|
"""Signal-killed processes report signed codes (Exit Code: -9)."""
|
|
output = "5 passed\n" + "M" * 5000 + "\nExit Code: -9"
|
|
result = _truncate_bash_output(output, 100)
|
|
assert result.endswith("\nExit Code: -9")
|
|
assert len(result) <= 100
|
|
|
|
def test_marker_preserved_when_limit_is_below_marker_length(self):
|
|
"""PR review: a configured limit smaller than the exit marker must
|
|
not silently discard failure status — the floor keeps it."""
|
|
result = _truncate_bash_output("1 passed\nExit Code: 1", 10)
|
|
assert result.endswith("\nExit Code: 1")
|
|
assert len(result) <= 32 # the marker-preserving floor
|
|
|
|
def test_small_limit_without_marker_uses_the_floor(self):
|
|
result = _truncate_bash_output("x" * 500, 10)
|
|
assert len(result) <= 32
|
|
|
|
def test_output_without_marker_truncates_as_before(self):
|
|
output = "A" * 30000
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert len(result) <= 20000
|
|
assert not result.endswith("Exit Code: 1")
|
|
|
|
def test_output_equal_to_limit_returned_unchanged(self):
|
|
output = "A" * 20000
|
|
assert _truncate_bash_output(output, 20000) == output
|
|
|
|
def test_long_output_is_truncated(self):
|
|
output = "A" * 30000
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert len(result) < len(output)
|
|
|
|
def test_result_never_exceeds_max_chars(self):
|
|
output = "A" * 30000
|
|
max_chars = 20000
|
|
result = _truncate_bash_output(output, max_chars)
|
|
assert len(result) <= max_chars
|
|
|
|
def test_head_is_preserved(self):
|
|
head = "HEAD_CONTENT"
|
|
output = head + "M" * 30000
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert result.startswith(head)
|
|
|
|
def test_tail_is_preserved(self):
|
|
tail = "TAIL_CONTENT"
|
|
output = "M" * 30000 + tail
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert result.endswith(tail)
|
|
|
|
def test_middle_truncation_marker_present(self):
|
|
output = "A" * 30000
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert "[middle truncated:" in result
|
|
assert "chars skipped" in result
|
|
|
|
def test_skipped_chars_count_is_correct(self):
|
|
output = "A" * 25000
|
|
result = _truncate_bash_output(output, 20000)
|
|
# Extract the reported skipped count and verify it equals len(output) - kept.
|
|
# (kept = max_chars - marker_max_len, where marker_max_len is computed from
|
|
# the worst-case marker string — so the exact value is implementation-defined,
|
|
# but it must equal len(output) minus the chars actually preserved.)
|
|
import re
|
|
|
|
m = re.search(r"(\d+) chars skipped", result)
|
|
assert m is not None
|
|
reported_skipped = int(m.group(1))
|
|
# Verify the number is self-consistent: head + skipped + tail == total
|
|
assert reported_skipped > 0
|
|
# The marker reports exactly the chars between head and tail
|
|
head_and_tail = len(output) - reported_skipped
|
|
assert result.startswith(output[: head_and_tail // 2])
|
|
|
|
def test_max_chars_zero_disables_truncation(self):
|
|
output = "A" * 100000
|
|
assert _truncate_bash_output(output, 0) == output
|
|
|
|
def test_50_50_split(self):
|
|
# head and tail should each be roughly max_chars // 2
|
|
output = "H" * 20000 + "M" * 10000 + "T" * 20000
|
|
result = _truncate_bash_output(output, 20000)
|
|
assert result[:100] == "H" * 100
|
|
assert result[-100:] == "T" * 100
|
|
|
|
def test_small_max_chars_does_not_crash(self):
|
|
output = "A" * 1000
|
|
result = _truncate_bash_output(output, 10)
|
|
# Limits below the marker-preserving floor (32) are raised so a
|
|
# failing command's exit marker always fits; see
|
|
# _BASH_OUTPUT_MIN_LIMIT_CHARS.
|
|
assert len(result) <= 32
|
|
|
|
def test_result_never_exceeds_max_chars_various_sizes(self):
|
|
output = "X" * 50000
|
|
for max_chars in [100, 1000, 5000, 20000, 49999]:
|
|
result = _truncate_bash_output(output, max_chars)
|
|
assert len(result) <= max_chars, f"failed for max_chars={max_chars}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# _truncate_read_file_output
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestTruncateReadFileOutput:
|
|
def test_short_output_returned_unchanged(self):
|
|
output = "def foo():\n pass\n"
|
|
assert _truncate_read_file_output(output, 50000) == output
|
|
|
|
def test_output_equal_to_limit_returned_unchanged(self):
|
|
output = "X" * 50000
|
|
assert _truncate_read_file_output(output, 50000) == output
|
|
|
|
def test_long_output_is_truncated(self):
|
|
output = "X" * 60000
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert len(result) < len(output)
|
|
|
|
def test_result_never_exceeds_max_chars(self):
|
|
output = "X" * 60000
|
|
max_chars = 50000
|
|
result = _truncate_read_file_output(output, max_chars)
|
|
assert len(result) <= max_chars
|
|
|
|
def test_head_is_preserved(self):
|
|
head = "import os\nimport sys\n"
|
|
output = head + "X" * 60000
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert result.startswith(head)
|
|
|
|
def test_truncation_marker_present(self):
|
|
output = "X" * 60000
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert "[truncated:" in result
|
|
assert "showing first" in result
|
|
|
|
def test_total_chars_reported_correctly(self):
|
|
output = "X" * 60000
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert "of 60000 chars" in result
|
|
|
|
def test_start_line_hint_present(self):
|
|
output = "X" * 60000
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert "start_line" in result
|
|
assert "end_line" in result
|
|
|
|
def test_max_chars_zero_disables_truncation(self):
|
|
output = "X" * 100000
|
|
assert _truncate_read_file_output(output, 0) == output
|
|
|
|
def test_tail_is_not_preserved(self):
|
|
# head-truncation: tail should be cut off
|
|
output = "H" * 50000 + "TAIL_SHOULD_NOT_APPEAR"
|
|
result = _truncate_read_file_output(output, 50000)
|
|
assert "TAIL_SHOULD_NOT_APPEAR" not in result
|
|
|
|
def test_small_max_chars_does_not_crash(self):
|
|
output = "X" * 1000
|
|
result = _truncate_read_file_output(output, 10)
|
|
assert len(result) <= 10
|
|
|
|
def test_result_never_exceeds_max_chars_various_sizes(self):
|
|
output = "X" * 50000
|
|
for max_chars in [100, 1000, 5000, 20000, 49999]:
|
|
result = _truncate_read_file_output(output, max_chars)
|
|
assert len(result) <= max_chars, f"failed for max_chars={max_chars}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# _truncate_ls_output
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestTruncateLsOutput:
|
|
def test_short_output_returned_unchanged(self):
|
|
output = "dir1\ndir2\nfile1.txt"
|
|
assert _truncate_ls_output(output, 20000) == output
|
|
|
|
def test_output_equal_to_limit_returned_unchanged(self):
|
|
output = "X" * 20000
|
|
assert _truncate_ls_output(output, 20000) == output
|
|
|
|
def test_long_output_is_truncated(self):
|
|
output = "\n".join(f"file_{i}.txt" for i in range(5000))
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert len(result) < len(output)
|
|
|
|
def test_result_never_exceeds_max_chars(self):
|
|
output = "\n".join(f"subdir/file_{i}.txt" for i in range(5000))
|
|
max_chars = 20000
|
|
result = _truncate_ls_output(output, max_chars)
|
|
assert len(result) <= max_chars
|
|
|
|
def test_head_is_preserved(self):
|
|
head = "first_dir\nsecond_dir\n"
|
|
output = head + "\n".join(f"file_{i}" for i in range(5000))
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert result.startswith(head)
|
|
|
|
def test_truncation_marker_present(self):
|
|
output = "\n".join(f"file_{i}.txt" for i in range(5000))
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert "[truncated:" in result
|
|
assert "showing first" in result
|
|
|
|
def test_total_chars_reported_correctly(self):
|
|
output = "X" * 30000
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert "of 30000 chars" in result
|
|
|
|
def test_hint_suggests_specific_path(self):
|
|
output = "X" * 30000
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert "Use a more specific path" in result
|
|
|
|
def test_max_chars_zero_disables_truncation(self):
|
|
output = "\n".join(f"file_{i}.txt" for i in range(10000))
|
|
assert _truncate_ls_output(output, 0) == output
|
|
|
|
def test_tail_is_not_preserved(self):
|
|
output = "H" * 20000 + "TAIL_SHOULD_NOT_APPEAR"
|
|
result = _truncate_ls_output(output, 20000)
|
|
assert "TAIL_SHOULD_NOT_APPEAR" not in result
|
|
|
|
def test_small_max_chars_does_not_crash(self):
|
|
output = "\n".join(f"file_{i}.txt" for i in range(100))
|
|
result = _truncate_ls_output(output, 10)
|
|
assert len(result) <= 10
|
|
|
|
def test_result_never_exceeds_max_chars_various_sizes(self):
|
|
output = "\n".join(f"file_{i}.txt" for i in range(5000))
|
|
for max_chars in [100, 1000, 5000, 20000, len(output) - 1]:
|
|
result = _truncate_ls_output(output, max_chars)
|
|
assert len(result) <= max_chars, f"failed for max_chars={max_chars}"
|