mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
fix(sandbox): keep line endings in local sandbox reads, writes and str_replace (#6343)
* fix(sandbox): keep line endings in local sandbox reads, writes and str_replace LocalSandbox opened files in text mode with the default newline=None. Reads translated CRLF to LF, so str_replace on a CRLF file rewrote every line as LF and the read-before-write hash missed newline-only changes; on Windows, writes turned LF content into CRLF (breaking `bash run.sh`). Read with newline="\n" and write with newline="", keeping line endings as stored like the remote providers; reads still end lines only at "\n", the rule count_file_lines and the truncation marker number lines by. str_replace spells an LF-written old_str/new_str with CRLF when the file uses it, so multi-line edits still match CRLF files on any provider. The tool-output budget middleware wrote externalized output the same way: on Windows the file no longer matched its stamped blob ref and the next model call deleted it when no blob store was configured. It now writes with newline="" too. * docs(changelog): reference #6343 for the line-ending fix * fix(sandbox): end local grep lines where read_file does find_grep_matches still opened files with universal newlines, so a bare "\r" split a line for grep but not for LocalSandbox.read_file, and a hit's line_number stopped matching the start_line that reads it back. Read with newline="\n" like read_file, and fold a CRLF terminator back to "\n" so $-anchored patterns keep matching CRLF lines as before.
This commit is contained in:
parent
3f8f91d064
commit
ab126d419c
14
CHANGELOG.md
14
CHANGELOG.md
@ -494,6 +494,19 @@ This release closes that milestone with **301 merged pull requests**.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **sandbox:** The local sandbox no longer rewrites line endings. `read_file`
|
||||
translated CRLF to LF, so `str_replace` on a CRLF file wrote every line back as
|
||||
LF, and the read-before-write gate could not see a change that only touched
|
||||
line endings; on Windows, `write_file` turned LF content into CRLF, breaking
|
||||
scripts such as `bash run.sh`. Local reads and writes now keep line endings as
|
||||
stored, like the remote providers, and `str_replace` spells a `\n`-written
|
||||
`old_str`/`new_str` with CRLF when the file uses it, which also lets
|
||||
multi-line edits match CRLF files on remote providers. Local `grep` ends lines
|
||||
at `\n` like `read_file`, so a hit's line number is the line a ranged read
|
||||
returns even when the file contains a bare `\r`. On Windows, oversized
|
||||
tool output saved under `outputs/.tool-results/` was likewise written as CRLF,
|
||||
no longer matched its stamped blob reference, and was deleted on the next
|
||||
model call when no blob store was configured; it is now written byte-exact. ([#6343])
|
||||
- **skills:** Editing a custom skill no longer runs filesystem work on the event
|
||||
loop. `PUT /api/skills/custom/{name}` built the user-scoped skill storage,
|
||||
probed the custom, public, legacy and integration roots, validated the
|
||||
@ -7825,3 +7838,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
[#6326]: https://github.com/bytedance/deer-flow/pull/6326
|
||||
[#6328]: https://github.com/bytedance/deer-flow/pull/6328
|
||||
[#6332]: https://github.com/bytedance/deer-flow/pull/6332
|
||||
[#6343]: https://github.com/bytedance/deer-flow/pull/6343
|
||||
|
||||
@ -432,6 +432,14 @@
|
||||
|
||||
### 修复
|
||||
|
||||
- **沙箱:** 本地沙箱不再改写换行符。`read_file` 此前把 CRLF 转成 LF,导致
|
||||
`str_replace` 编辑 CRLF 文件后整份文件都变成 LF,且读后写校验无法察觉仅改动换行符的
|
||||
变更;在 Windows 上 `write_file` 会把 LF 内容写成 CRLF,使 `bash run.sh` 等脚本失败。
|
||||
现在本地读写与远程沙箱一致,按原样保留换行符;当文件使用 CRLF 时,`str_replace` 会把
|
||||
以 `\n` 书写的 `old_str`/`new_str` 转为 CRLF,远程沙箱上的多行编辑也因此能匹配 CRLF 文件。
|
||||
本地 `grep` 与 `read_file` 一样只在 `\n` 处断行,即使文件含有单独的 `\r`,命中的行号也与按行号范围读取的结果一致。
|
||||
同样在 Windows 上,保存到 `outputs/.tool-results/` 的超长工具输出此前也被写成 CRLF,与其 blob
|
||||
引用不再一致,未配置 blob 存储时会在下一次模型调用时被删除;现在按原样逐字节写入。([#6343])
|
||||
- **技能:** 编辑自定义技能时,不再在事件循环上执行文件系统操作。
|
||||
`PUT /api/skills/custom/{name}` 此前在事件循环上构建用户级技能存储、探测自定义、
|
||||
内置、旧版共享与集成目录、把草稿写入临时目录以校验 frontmatter,并读取将被替换的
|
||||
@ -6508,3 +6516,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子
|
||||
[#6319]: https://github.com/bytedance/deer-flow/pull/6319
|
||||
[#6326]: https://github.com/bytedance/deer-flow/pull/6326
|
||||
[#6332]: https://github.com/bytedance/deer-flow/pull/6332
|
||||
[#6343]: https://github.com/bytedance/deer-flow/pull/6343
|
||||
|
||||
@ -14,7 +14,10 @@ file publication, allowing the tool wrappers to use the configured inline fallba
|
||||
creation (`open(..., "x")`). Its mode is `0o666 & ~umask`, preserving ordinary
|
||||
file-creation permissions without reading or changing the process-wide umask.
|
||||
This matters when a mounted sandbox reads the output under a different UID.
|
||||
Restrictive operator umasks remain restrictive.
|
||||
Restrictive operator umasks remain restrictive. It also passes `newline=""`, so
|
||||
the file holds exactly the bytes of the stamped blob ref on every platform;
|
||||
Windows text mode would write CRLF, and restore discards a file whose size or
|
||||
digest does not match.
|
||||
|
||||
The writer closes its file before atomically replacing the deterministic final
|
||||
path. Concurrent publishers of identical output can safely reuse that path;
|
||||
|
||||
@ -222,7 +222,9 @@ def _externalize(
|
||||
candidate_path = os.path.join(storage_dir, f".tool-output-{uuid.uuid4().hex}.tmp")
|
||||
# Exclusive creation keeps per-writer ownership while honoring umask,
|
||||
# unlike NamedTemporaryFile's fixed 0600 mode on mounted outputs.
|
||||
with open(candidate_path, "x", encoding="utf-8") as f:
|
||||
# newline="" keeps the bytes equal to the stamped blob ref; Windows
|
||||
# text mode would write CRLF and restore would discard the file.
|
||||
with open(candidate_path, "x", encoding="utf-8", newline="") as f:
|
||||
tmp_path = candidate_path
|
||||
f.write(content)
|
||||
os.replace(tmp_path, filepath)
|
||||
|
||||
@ -116,6 +116,6 @@ symlinks, and visible depth/size bounds.
|
||||
- `ls` - Directory listing (tree format, max 2 levels). Remote commands precheck root existence and emit `__DF_FIND_STATUS__:missing`; `find` status 1 is always an incomplete-traversal `OSError`. `AioSandbox.list_dir` parses only `completed`/`None` on #5634's selected generation: `hard_timeout` raises `TimeoutError` and keeps that generation reusable; an ambiguous timeout or status raises `OSError`, replays nothing, and fences that generation. Missing-session 404s drop the stale id without replay. Persistent/scoped shells fence all `httpx.TransportError`s before cleanup.
|
||||
- `glob` - Find files or directories below a root directory with bounded results
|
||||
- `grep` - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `read_file` - Read file contents with optional line range; full reads keep stored line endings (Local `read_file`/`grep` read `newline="\n"`, writes `newline=""`), ranged reads join with `\n`
|
||||
- `write_file` - Write/append to files, creates directories; overwrites by default and exposes the `append` argument in the model-facing schema for end-of-file writes; subject to the read-before-write gate when `read_before_write.enabled` (see Middleware Chain)
|
||||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process; subject to the read-before-write gate when `read_before_write.enabled` (see Middleware Chain)
|
||||
- `str_replace` - Substring replacement (single or all occurrences); `_match_line_endings` gives `old_str`/`new_str` a CRLF file's endings; same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process; subject to the read-before-write gate when `read_before_write.enabled` (see Middleware Chain)
|
||||
|
||||
@ -832,7 +832,11 @@ class LocalSandbox(Sandbox):
|
||||
resolved_path = self._resolve_path(path)
|
||||
should_slice = start_line is not None or end_line is not None
|
||||
try:
|
||||
with open(resolved_path, encoding="utf-8") as f:
|
||||
# newline="\n" returns line endings as stored, like the remote
|
||||
# providers (a translated read hid CRLF from str_replace, which then
|
||||
# wrote the whole file back as LF), and ends lines only at "\n", the
|
||||
# rule count_file_lines and read_file's truncation marker count by.
|
||||
with open(resolved_path, encoding="utf-8", newline="\n") as f:
|
||||
if not should_slice:
|
||||
content = f.read()
|
||||
|
||||
@ -892,7 +896,9 @@ class LocalSandbox(Sandbox):
|
||||
# using the content-specific resolver (forward-slash safe)
|
||||
resolved_content = self._resolve_paths_in_content(content)
|
||||
mode = "a" if append else "w"
|
||||
with open(resolved_path, mode, encoding="utf-8") as f:
|
||||
# newline="" writes the content as given; the default would turn
|
||||
# every "\n" into "\r\n" on Windows (breaking `bash run.sh`).
|
||||
with open(resolved_path, mode, encoding="utf-8", newline="") as f:
|
||||
f.write(resolved_content)
|
||||
# Track this path so read_file knows to reverse-resolve on read.
|
||||
# Only agent-written files get reverse-resolved; user uploads and
|
||||
|
||||
@ -22,15 +22,15 @@ READ_FILE_TRUNCATION_PREFIX = "... [truncated:"
|
||||
def count_file_lines(content: str) -> int:
|
||||
"""Count file lines as LocalSandbox and read_file's truncation marker see them.
|
||||
|
||||
Uses ``count("\\n") + (1 if not endswith("\\n") else 0)`` — matching Python's
|
||||
text-mode line iteration used by ``LocalSandbox.read_file`` and the formula in
|
||||
Uses ``count("\\n") + (1 if not endswith("\\n") else 0)`` — matching the
|
||||
``newline="\\n"`` line iteration used by ``LocalSandbox.read_file`` and the formula in
|
||||
``_truncate_read_file_output``. This ensures the line count shown in the gate
|
||||
block message agrees with the ``start_line``/``end_line`` values the model can
|
||||
pass back to ``read_file``.
|
||||
|
||||
Unlike ``splitlines()``, this function does not treat ``\\f``, ``\\v``, ``\\x85``,
|
||||
``\\u2028``, or ``\\u2029`` as line separators — consistent with LocalSandbox's
|
||||
text-mode iteration where those characters are just regular content.
|
||||
line iteration where those characters are just regular content.
|
||||
"""
|
||||
if not content:
|
||||
return 0
|
||||
|
||||
@ -249,8 +249,14 @@ def find_grep_matches(
|
||||
continue
|
||||
if file_path.stat().st_size > max_file_size or is_binary_file(file_path):
|
||||
continue
|
||||
with file_path.open(encoding="utf-8", errors="replace") as handle:
|
||||
# newline="\n" ends lines where LocalSandbox.read_file does, so a
|
||||
# hit's line_number is the start_line that reads it back; a bare
|
||||
# "\r" stays content. A CRLF terminator still reads as "\n", so $
|
||||
# anchors match as before.
|
||||
with file_path.open(encoding="utf-8", errors="replace", newline="\n") as handle:
|
||||
for line_number, line in enumerate(handle, start=1):
|
||||
if line.endswith("\r\n"):
|
||||
line = line[:-2] + "\n"
|
||||
if len(line) > _max_line_chars:
|
||||
continue
|
||||
if regex.search(line):
|
||||
|
||||
@ -2984,6 +2984,27 @@ async def _write_file_tool_async(
|
||||
write_file_tool.coroutine = _write_file_tool_async
|
||||
|
||||
|
||||
def _to_crlf(text: str) -> str:
|
||||
return text.replace("\r\n", "\n").replace("\n", "\r\n")
|
||||
|
||||
|
||||
def _match_line_endings(content: str, old_str: str, new_str: str) -> tuple[str, str]:
|
||||
"""Spell ``old_str`` and ``new_str`` with the CRLF line endings ``content`` uses.
|
||||
|
||||
Full reads return line endings as stored, but the model tends to write
|
||||
``\\n`` (and ranged reads join lines with ``\\n``), so a multi-line
|
||||
``old_str`` would never match a CRLF file and inserted lines would be LF.
|
||||
A CRLF-only file takes both strings in CRLF; a mixed file does so only
|
||||
when the LF spelling is absent and the CRLF one is present.
|
||||
"""
|
||||
if "\r\n" not in content:
|
||||
return old_str, new_str
|
||||
crlf_old = _to_crlf(old_str)
|
||||
if content.count("\r\n") == content.count("\n") or (old_str not in content and crlf_old in content):
|
||||
return crlf_old, _to_crlf(new_str)
|
||||
return old_str, new_str
|
||||
|
||||
|
||||
@tool("str_replace", parse_docstring=True)
|
||||
def str_replace_tool(
|
||||
runtime: Runtime,
|
||||
@ -3022,6 +3043,7 @@ def str_replace_tool(
|
||||
# A no-op edit. str.replace("", new_str) would insert new_str at
|
||||
# every character boundary, so this cannot fall through.
|
||||
return "OK"
|
||||
old_str, new_str = _match_line_endings(content, old_str, new_str)
|
||||
if not content or old_str not in content:
|
||||
return f"Error: String to replace not found in file: {requested_path}"
|
||||
if replace_all:
|
||||
|
||||
@ -868,7 +868,7 @@ def test_lock_scope_resolves_overwrite_wrapped_sandbox():
|
||||
("write_file", "", "0 lines"),
|
||||
("str_replace", "line1\nline2\n", "2 lines"),
|
||||
# Exotic separators: \f and \u2028 are NOT line boundaries in LocalSandbox
|
||||
# (Python text-mode iterates only \n/\r\n/\r), so count must match LocalSandbox.
|
||||
# (it ends lines only at \n), so count must match LocalSandbox.
|
||||
("write_file", "x\fx\n", "1 line"),
|
||||
("write_file", "x\u2028x\n", "1 line"),
|
||||
],
|
||||
|
||||
218
backend/tests/test_sandbox_line_endings.py
Normal file
218
backend/tests/test_sandbox_line_endings.py
Normal file
@ -0,0 +1,218 @@
|
||||
"""Line endings survive local-sandbox reads, writes and ``str_replace``.
|
||||
|
||||
``LocalSandbox`` opened files in text mode with the default ``newline=None``.
|
||||
Reads translated CRLF to LF, so ``str_replace`` on a CRLF file edited one line
|
||||
and wrote every line back as LF, and the read-before-write gate hashed the
|
||||
translated text and missed a change that only touched line endings. Writes
|
||||
translated ``\\n`` to ``os.linesep``, so on Windows LF content landed as CRLF.
|
||||
Reads now use ``newline="\\n"`` and writes ``newline=""``: both keep line
|
||||
endings as stored, matching the remote providers, and ``str_replace`` spells
|
||||
``old_str``/``new_str`` the way the file does.
|
||||
"""
|
||||
|
||||
import builtins
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import deerflow.sandbox.local.local_sandbox as local_sandbox
|
||||
from deerflow.sandbox.local.local_sandbox import LocalSandbox
|
||||
from deerflow.sandbox.read_file_contract import count_file_lines
|
||||
from deerflow.sandbox.tools import read_current_file_content, str_replace_tool
|
||||
|
||||
CRLF_SOURCE = b"first\r\nsecond\r\nthird\r\n"
|
||||
|
||||
|
||||
def _local_runtime(tmp_path: Path) -> SimpleNamespace:
|
||||
for sub in ("workspace", "uploads", "outputs"):
|
||||
(tmp_path / sub).mkdir(parents=True, exist_ok=True)
|
||||
thread_data = {
|
||||
"workspace_path": str(tmp_path / "workspace"),
|
||||
"uploads_path": str(tmp_path / "uploads"),
|
||||
"outputs_path": str(tmp_path / "outputs"),
|
||||
}
|
||||
return SimpleNamespace(
|
||||
state={"sandbox": {"sandbox_id": "local:t1"}, "thread_data": thread_data},
|
||||
context={"thread_id": "t1"},
|
||||
)
|
||||
|
||||
|
||||
def _use_local_sandbox(monkeypatch) -> None:
|
||||
monkeypatch.setattr("deerflow.sandbox.tools.ensure_sandbox_initialized", lambda runtime: LocalSandbox("t1"))
|
||||
monkeypatch.setattr("deerflow.sandbox.tools.ensure_thread_directories_exist", lambda runtime: None)
|
||||
|
||||
|
||||
def _str_replace(tmp_path, monkeypatch, source: bytes, *, old_str: str, new_str: str, replace_all: bool = False) -> tuple[str, bytes]:
|
||||
runtime = _local_runtime(tmp_path)
|
||||
target = tmp_path / "outputs" / "file.txt"
|
||||
target.write_bytes(source)
|
||||
_use_local_sandbox(monkeypatch)
|
||||
result = str_replace_tool.func(
|
||||
runtime=runtime,
|
||||
description="replace",
|
||||
path="/mnt/user-data/outputs/file.txt",
|
||||
old_str=old_str,
|
||||
new_str=new_str,
|
||||
replace_all=replace_all,
|
||||
)
|
||||
return result, target.read_bytes()
|
||||
|
||||
|
||||
def _emulate_windows_text_writes(monkeypatch) -> None:
|
||||
"""Make text-mode writes translate ``\\n`` the way Windows does by default."""
|
||||
base = builtins.open
|
||||
|
||||
def windows_open(file, mode="r", *args, **kwargs):
|
||||
if "b" not in mode and any(flag in mode for flag in "wax+"):
|
||||
kwargs.setdefault("newline", "\r\n")
|
||||
return base(file, mode, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(local_sandbox, "open", windows_open, raising=False)
|
||||
|
||||
|
||||
def test_read_file_returns_crlf_as_stored(tmp_path) -> None:
|
||||
path = tmp_path / "crlf.txt"
|
||||
path.write_bytes(CRLF_SOURCE)
|
||||
|
||||
assert LocalSandbox("t").read_file(str(path)) == "first\r\nsecond\r\nthird\r\n"
|
||||
|
||||
|
||||
def test_ranged_read_file_still_joins_lines_with_lf(tmp_path) -> None:
|
||||
# Ranged reads join selected lines with "\n" in every provider; keep that.
|
||||
path = tmp_path / "crlf.txt"
|
||||
path.write_bytes(CRLF_SOURCE)
|
||||
|
||||
assert LocalSandbox("t").read_file(str(path), start_line=2, end_line=3) == "second\nthird"
|
||||
|
||||
|
||||
def test_ranged_read_line_numbers_agree_with_count_file_lines(tmp_path) -> None:
|
||||
# The gate's block message and read_file's truncation marker number lines
|
||||
# with count_file_lines over the full read; ranged reads must end lines at
|
||||
# the same place, so a bare CR is content, not a line break.
|
||||
path = tmp_path / "mixed.txt"
|
||||
path.write_bytes(b"a\rb\r\nc\nd")
|
||||
sandbox = LocalSandbox("t")
|
||||
full = sandbox.read_file(str(path))
|
||||
|
||||
assert full == "a\rb\r\nc\nd"
|
||||
assert count_file_lines(full) == 3
|
||||
assert [sandbox.read_file(str(path), start_line=n, end_line=n) for n in (1, 2, 3, 4)] == ["a\rb", "c", "d", ""]
|
||||
|
||||
|
||||
def test_grep_line_numbers_are_the_lines_ranged_read_file_returns(tmp_path) -> None:
|
||||
# grep ends lines where read_file does, so a hit's line_number is the
|
||||
# start_line that reads it back, even with a bare CR inside a line.
|
||||
path = tmp_path / "mixed.txt"
|
||||
path.write_bytes(b"alpha\nlineB\rbeta\ngamma\r\n")
|
||||
sandbox = LocalSandbox("t")
|
||||
|
||||
matches, _ = sandbox.grep(str(path), "beta|gamma")
|
||||
|
||||
assert [(match.line_number, match.line) for match in matches] == [(2, "lineB\rbeta"), (3, "gamma")]
|
||||
for match in matches:
|
||||
assert sandbox.read_file(str(path), start_line=match.line_number, end_line=match.line_number) == match.line
|
||||
|
||||
|
||||
def test_grep_end_anchor_still_matches_a_crlf_line(tmp_path) -> None:
|
||||
path = tmp_path / "crlf.txt"
|
||||
path.write_bytes(CRLF_SOURCE)
|
||||
|
||||
matches, _ = LocalSandbox("t").grep(str(path), "second$")
|
||||
|
||||
assert [(match.line_number, match.line) for match in matches] == [(2, "second")]
|
||||
|
||||
|
||||
def test_write_file_keeps_lf_under_windows_text_defaults(tmp_path, monkeypatch) -> None:
|
||||
path = tmp_path / "run.sh"
|
||||
_emulate_windows_text_writes(monkeypatch)
|
||||
|
||||
LocalSandbox("t").write_file(str(path), "#!/bin/bash\necho ok\n")
|
||||
LocalSandbox("t").write_file(str(path), "echo more\n", append=True)
|
||||
|
||||
assert path.read_bytes() == b"#!/bin/bash\necho ok\necho more\n"
|
||||
|
||||
|
||||
def test_str_replace_multiline_lf_old_str_keeps_a_crlf_file_crlf(tmp_path, monkeypatch) -> None:
|
||||
result, after = _str_replace(tmp_path, monkeypatch, CRLF_SOURCE, old_str="first\nsecond\n", new_str="first\ninserted\nsecond\n")
|
||||
|
||||
assert result == "OK"
|
||||
assert after == b"first\r\ninserted\r\nsecond\r\nthird\r\n"
|
||||
|
||||
|
||||
def test_str_replace_single_line_edit_leaves_other_crlf_lines_alone(tmp_path, monkeypatch) -> None:
|
||||
result, after = _str_replace(tmp_path, monkeypatch, CRLF_SOURCE, old_str="second", new_str="SECOND")
|
||||
|
||||
assert result == "OK"
|
||||
assert after == b"first\r\nSECOND\r\nthird\r\n"
|
||||
|
||||
|
||||
def test_str_replace_lines_inserted_after_a_single_line_anchor_are_crlf(tmp_path, monkeypatch) -> None:
|
||||
# old_str has no newline to translate, so only the file decides new_str's.
|
||||
result, after = _str_replace(tmp_path, monkeypatch, CRLF_SOURCE, old_str="second", new_str="second\nextra")
|
||||
|
||||
assert result == "OK"
|
||||
assert after == b"first\r\nsecond\r\nextra\r\nthird\r\n"
|
||||
|
||||
|
||||
def test_str_replace_all_on_a_crlf_file_keeps_crlf(tmp_path, monkeypatch) -> None:
|
||||
result, after = _str_replace(tmp_path, monkeypatch, b"a\r\nb\r\na\r\nb\r\n", old_str="a\nb", new_str="c\nd", replace_all=True)
|
||||
|
||||
assert result == "OK"
|
||||
assert after == b"c\r\nd\r\nc\r\nd\r\n"
|
||||
|
||||
|
||||
def test_str_replace_on_a_mixed_file_edits_only_the_matched_lines(tmp_path, monkeypatch) -> None:
|
||||
mixed = b"lf1\nlf2\ncrlf1\r\ncrlf2\r\n"
|
||||
|
||||
result, after = _str_replace(tmp_path, monkeypatch, mixed, old_str="lf1\nlf2\n", new_str="LF1\nLF2\n")
|
||||
assert result == "OK"
|
||||
assert after == b"LF1\nLF2\ncrlf1\r\ncrlf2\r\n"
|
||||
|
||||
result, after = _str_replace(tmp_path, monkeypatch, mixed, old_str="crlf1\ncrlf2\n", new_str="CRLF1\nCRLF2\n")
|
||||
assert result == "OK"
|
||||
assert after == b"lf1\nlf2\nCRLF1\r\nCRLF2\r\n"
|
||||
|
||||
|
||||
def test_str_replace_on_an_lf_file_writes_new_str_as_given(tmp_path, monkeypatch) -> None:
|
||||
result, after = _str_replace(tmp_path, monkeypatch, b"one\ntwo\n", old_str="one\n", new_str="uno\r\n")
|
||||
|
||||
assert result == "OK"
|
||||
assert after == b"uno\r\ntwo\n"
|
||||
|
||||
|
||||
def test_str_replace_on_a_crlf_file_still_reports_a_missing_old_str(tmp_path, monkeypatch) -> None:
|
||||
result, after = _str_replace(tmp_path, monkeypatch, CRLF_SOURCE, old_str="second\nfourth", new_str="x")
|
||||
|
||||
assert result.startswith("Error: String to replace not found in file")
|
||||
assert after == CRLF_SOURCE
|
||||
|
||||
|
||||
def test_str_replace_matches_crlf_from_a_provider_that_reads_raw(monkeypatch) -> None:
|
||||
# Remote providers already returned CRLF untranslated, so an LF old_str
|
||||
# spanning lines never matched there either.
|
||||
sandbox = MagicMock()
|
||||
sandbox.id = "remote"
|
||||
sandbox.read_file.return_value = "first\r\nsecond\r\n"
|
||||
monkeypatch.setattr("deerflow.sandbox.tools.ensure_sandbox_initialized", lambda runtime: sandbox)
|
||||
monkeypatch.setattr("deerflow.sandbox.tools.ensure_thread_directories_exist", lambda runtime: None)
|
||||
monkeypatch.setattr("deerflow.sandbox.tools.is_local_sandbox", lambda runtime: False)
|
||||
|
||||
result = str_replace_tool.func(runtime=MagicMock(), description="d", path="/mnt/user-data/outputs/f.txt", old_str="first\nsecond", new_str="1\n2")
|
||||
|
||||
assert result == "OK"
|
||||
sandbox.write_file.assert_called_once_with("/mnt/user-data/outputs/f.txt", "1\r\n2\r\n")
|
||||
|
||||
|
||||
def test_read_before_write_content_changes_when_only_line_endings_change(tmp_path, monkeypatch) -> None:
|
||||
# The gate hashes read_current_file_content; a CRLF -> LF rewrite by
|
||||
# another process must invalidate the model's earlier read.
|
||||
runtime = _local_runtime(tmp_path)
|
||||
target = tmp_path / "outputs" / "file.txt"
|
||||
_use_local_sandbox(monkeypatch)
|
||||
|
||||
target.write_bytes(CRLF_SOURCE)
|
||||
before = read_current_file_content(runtime, "/mnt/user-data/outputs/file.txt")
|
||||
target.write_bytes(CRLF_SOURCE.replace(b"\r\n", b"\n"))
|
||||
after = read_current_file_content(runtime, "/mnt/user-data/outputs/file.txt")
|
||||
|
||||
assert before != after
|
||||
@ -731,6 +731,37 @@ class TestToolOutputBlobPersistence:
|
||||
}
|
||||
]
|
||||
|
||||
def test_host_externalized_file_keeps_its_blob_ref_bytes_under_windows_newlines(self, monkeypatch, tmp_path):
|
||||
# Windows text mode wrote "\n" as "\r\n", so the file no longer matched
|
||||
# the ref stamped from content.encode() and the next model call on this
|
||||
# Gateway deleted it as a mismatch when no blob store was configured.
|
||||
from deerflow.agents.middlewares import tool_output_budget_middleware as mod
|
||||
|
||||
content = "first line\nsecond line\n" * 20
|
||||
builtin_open = open
|
||||
|
||||
def windows_open(file, mode="r", *args, **kwargs):
|
||||
if "b" not in mode and any(flag in mode for flag in "wax+"):
|
||||
kwargs.setdefault("newline", "\r\n")
|
||||
return builtin_open(file, mode, *args, **kwargs)
|
||||
|
||||
class RecordingStore:
|
||||
def put_bytes(self, data: bytes, **kwargs) -> BlobRef:
|
||||
return BlobRef(sha256=hashlib.sha256(data).hexdigest(), size=len(data), kind=kwargs["kind"], content_type=kwargs["content_type"])
|
||||
|
||||
monkeypatch.setattr(mod, "open", windows_open, raising=False)
|
||||
monkeypatch.setattr(mod, "get_blob_store_if_enabled", lambda: RecordingStore())
|
||||
mw = ToolOutputBudgetMiddleware(config=ToolOutputConfig(externalize_min_chars=50, preview_head_chars=20, preview_tail_chars=10))
|
||||
result = mw.wrap_tool_call(_make_request(outputs_path=str(tmp_path)), lambda _: _tm(content, name="remote_executor"))
|
||||
saved = tmp_path / ".tool-results" / os.path.basename(result.additional_kwargs[TOOL_OUTPUT_BLOB_KEY]["virtual_path"])
|
||||
assert saved.read_bytes() == content.encode()
|
||||
|
||||
monkeypatch.setattr(mod, "get_blob_store_if_enabled", lambda: None)
|
||||
request = ModelRequest(model=None, messages=[result], tools=[], state={"thread_data": {"outputs_path": str(tmp_path)}})
|
||||
mw.wrap_model_call(request, lambda prepared: [])
|
||||
|
||||
assert saved.read_bytes() == content.encode()
|
||||
|
||||
def test_configured_blob_write_failure_falls_back_inline(self, monkeypatch, tmp_path):
|
||||
from deerflow.agents.middlewares import tool_output_budget_middleware as mod
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user