fix(sandbox): prune ignored entries before the listing limit (#5676)

* fix(sandbox): prune ignored entries before the listing limit

* docs(sandbox): clarify listing filter assumptions

README.md

@@ -1635,6 +1635,8 @@ Each task gets its own execution environment with a full filesystem view — ski
 
 The built-in `grep` tool searches either one text file or all matching text files below a directory, so an agent can search an uploaded document directly without first broadening the request to the entire uploads directory.
 
+Remote `ls` excludes ignored descendants before applying its 500-entry listing limit, so dependency and build trees do not crowd out visible files. Explicitly listing an ignored directory still lists its contents; normal depth and output limits remain in effect.
+
 Uploaded Markdown outlines recognize ATX heading syntax, clean closing markers with a linear suffix scan, and skip fenced code examples, so hashtags and code comments do not
 crowd out real document sections from the agent's heading preview.
 UTF-8 Markdown files with or without a byte-order mark (BOM) produce the same

backend/packages/harness/deerflow/sandbox/AGENTS.md

@@ -103,6 +103,12 @@ consume events into a human-input card when no human can respond.
 - Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
 
 **Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
+
+Remote `list_dir` prints the root, then prunes ignored descendants before the
+500-entry cap. Preserve root literals, host case policy, status markers and the
+subshell. Keep parser filtering as a backstop; test ignored roots, metacharacters,
+symlinks, and visible depth/size bounds.
+
 - Every sandbox tool keeps a model-visible `description` field for a human-readable progress label, but the field is optional and defaults to an empty string. Tool execution must depend only on its operational arguments; the frontend supplies localized fallback labels when a provider omits `description`.
 - `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), output on POSIX and Windows is captured through bounded pipe-drain threads and stdin is `/dev/null`; Windows capture decodes with the platform text encoding and applies universal-newline translation, matching the former `subprocess.run(..., text=True)` behavior for locale-code-page output, Python UTF-8 Mode, CRLF, and bare CR. That translation is Windows-only so the pre-existing POSIX output contract remains byte-decoded without newline rewriting. On POSIX, a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole POSIX process group or Windows process tree is killed and the agent gets a notice telling it to background long-lived processes. The shared bash tool description scopes host environment detection to LocalSandbox: start with `uname -s`, follow with `sw_vers` on Darwin, and read Linux host system files only when the active policy permits them. Local path and `file://` rejections provide the same conditional recovery guidance: command-only probes for environment questions, allowed virtual paths otherwise, and no repetition of the rejected path. The description also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command`, its platform runners, and `bash_tool`'s docstring.
 - `ls` - Directory listing (tree format, max 2 levels). Remote commands precheck root existence and emit `__DF_FIND_STATUS__:missing`; `find` status 1 is always an incomplete-traversal `OSError`, including when no entries were printed. Do not infer a missing path from status 1 alone.

backend/packages/harness/deerflow/sandbox/remote_list_dir.py

@@ -14,9 +14,10 @@ not an error.
 
 from __future__ import annotations
 
+import os
 import shlex
 
-from deerflow.sandbox.search import should_ignore_path
+from deerflow.sandbox.search import IGNORE_PATTERNS, should_ignore_path
 
 _STATUS_PREFIX = "__DF_FIND_STATUS__:"
 _MISSING_ROOT = "missing"
@@ -31,6 +32,12 @@ def remote_list_dir_command(path: str, max_depth: int, *, limit: int = _LIST_LIM
     quoted = shlex.quote(path)
     depth = int(max_depth)
     n = int(limit)
+    # Match the parser's host-platform case policy. Prune ignored descendants
+    # before head so they cannot consume the visible listing's output budget.
+    name_test = "-iname" if os.path.normcase("A") == "a" else "-name"
+    # IGNORE_PATTERNS must contain basename patterns (no '/'); -name/-iname do not match paths.
+    ignored = " -o ".join(f"{name_test} {shlex.quote(pattern)}" for pattern in IGNORE_PATTERNS)
+    prune = f"\\( {ignored} \\) -prune -o " if ignored else ""
     # Status file is written by the find side of the pipe, then printed AFTER
     # head so a 500-line listing cannot truncate the marker. ``set +e`` undoes
     # a login-profile ``set -e`` so a failing find still records $?. End with
@@ -45,7 +52,11 @@ def remote_list_dir_command(path: str, max_depth: int, *, limit: int = _LIST_LIM
     return (
         f"set +e; ( if [ ! -e {quoted} ]; then printf '%s\\n' {_STATUS_PREFIX}{_MISSING_ROOT}; exit 1; fi; "
         f"_st=/tmp/df_find_$$; "
-        f"{{ find -H {quoted} -maxdepth {depth} \\( -type f -o -type d \\) 2>/dev/null; "
+        # Print the explicit root separately and only filter its descendants.
+        # Unlike a -path root exemption, this treats glob metacharacters in
+        # the root literally and still permits listing an ignored root itself.
+        f"{{ printf '%s\\n' {quoted}; "
+        f"find -H {quoted} -mindepth 1 -maxdepth {depth} {prune}\\( -type f -o -type d \\) -print 2>/dev/null; "
         f'echo $? > "$_st"; }} | head -n {n}; '
         f'st=$(cat "$_st" 2>/dev/null); '
         f"printf '\\n%s\\n' {_STATUS_PREFIX}$st; "
@@ -121,8 +132,7 @@ def parse_remote_list_dir_output(
             if not should_ignore_path(entry[len(prefix) :]):
                 kept.append(entry)
         else:
-            # ``find -H`` prints the resolved target when the root is a symlink,
-            # so an entry may not carry the requested prefix. Keep it: a path
-            # that cannot be placed relative to the root must not disappear.
+            # Defensive fallback for unexpected entries outside the requested
+            # prefix: root-relative ignore matching cannot classify them.
             kept.append(entry)
     return kept

backend/tests/test_remote_list_dir.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import os
+import shlex
 import shutil
 import stat
 import subprocess
@@ -210,8 +211,10 @@ def test_list_dir_command_head_truncation_is_not_an_error(tmp_path) -> None:
     )
     entries = parse_remote_list_dir_output(proc.stdout, "/dir", pipeline_exit_code=proc.returncode)
     assert len(entries) == 500
-    assert entries[0] == "/dir/f1"
-    assert entries[-1] == "/dir/f500"
+    # The explicitly emitted root shares the same 500-entry output budget.
+    assert entries[0] == str(tmp_path)
+    assert entries[1] == "/dir/f1"
+    assert entries[-1] == "/dir/f499"
 
 
 @_POSIX_SH
@@ -314,3 +317,77 @@ def test_list_dir_command_lists_an_explicitly_requested_ignored_directory(tmp_pa
     entries = parse_remote_list_dir_output(proc.stdout, str(root), pipeline_exit_code=proc.returncode)
     assert str(root) in entries
     assert str(root / "notes.txt") in entries
+
+
+@_POSIX_SH
+@pytest.mark.skipif(shutil.which("find") is None or shutil.which("sort") is None, reason="system find and sort required")
+@pytest.mark.parametrize("root_name", ["workspace", "build", "project [one]'s"])
+@pytest.mark.parametrize("ignored_kind", ["directory", "files"])
+def test_ignored_entries_do_not_consume_listing_budget(tmp_path, root_name, ignored_kind) -> None:
+    """A deterministic real-find order puts ignored entries before the useful file."""
+    root = tmp_path / root_name
+    root.mkdir()
+    if ignored_kind == "directory":
+        ignored = root / "node_modules"
+        ignored.mkdir()
+        for index in range(600):
+            (ignored / f"dependency_{index:04}.js").touch()
+    else:
+        for index in range(600):
+            (root / f"ignored_{index:04}.log").touch()
+    visible = root / "zz_report.txt"
+    visible.write_text("report", encoding="utf-8")
+    # Only enumeration order is normalized. The real find still evaluates the
+    # production arguments and traversal/pruning expression against real files.
+    find = shlex.quote(shutil.which("find"))
+    sort = shlex.quote(shutil.which("sort"))
+    fake_bin = _write_fake_find(tmp_path, f'#!/bin/sh\n{find} "$@" | LC_ALL=C {sort}\n')
+    proc = _run_list_dir_script(remote_list_dir_command(str(root), 2), env=_env_with_bin(str(fake_bin)))
+    entries = parse_remote_list_dir_output(proc.stdout, str(root), pipeline_exit_code=proc.returncode)
+    assert entries == [str(root), str(visible)]
+
+
+@_POSIX_SH
+@pytest.mark.skipif(shutil.which("find") is None, reason="system find required")
+def test_visible_listing_still_obeys_depth_and_output_limit(tmp_path) -> None:
+    root = tmp_path / "workspace"
+    root.mkdir()
+    for index in range(8):
+        (root / f"visible_{index}.txt").touch()
+    (root / "nested" / "too-deep").mkdir(parents=True)
+    (root / "nested" / "too-deep" / "report.txt").touch()
+    proc = _run_list_dir_script(remote_list_dir_command(str(root), 1, limit=4))
+    entries = parse_remote_list_dir_output(proc.stdout, str(root), pipeline_exit_code=proc.returncode)
+    assert len(entries) == 4
+    assert entries[0] == str(root)
+    assert all("too-deep" not in entry for entry in entries)
+
+
+@_POSIX_SH
+@pytest.mark.skipif(shutil.which("find") is None, reason="system find required")
+def test_pruning_preserves_an_ignored_symlinked_root(tmp_path) -> None:
+    target = tmp_path / "actual"
+    target.mkdir()
+    (target / "report.txt").touch()
+    (target / "node_modules").mkdir()
+    (target / "node_modules" / "dependency.js").touch()
+    root = tmp_path / "build"
+    root.symlink_to(target, target_is_directory=True)
+    proc = _run_list_dir_script(remote_list_dir_command(str(root), 2))
+    entries = parse_remote_list_dir_output(proc.stdout, str(root), pipeline_exit_code=proc.returncode)
+    assert entries == [str(root), str(root / "report.txt")]
+
+
+@_POSIX_SH
+@pytest.mark.skipif(shutil.which("find") is None, reason="system find required")
+def test_pruning_uses_the_parsers_case_policy(tmp_path, monkeypatch) -> None:
+    root = tmp_path / "workspace"
+    (root / "BUILD").mkdir(parents=True)
+    (root / "BUILD" / "ignored.txt").touch()
+    (root / "report.txt").touch()
+    # Exercise the policy a Windows Gateway applies to a POSIX remote sandbox.
+    monkeypatch.setattr(os.path, "normcase", lambda value: value.lower())
+    proc = _run_list_dir_script(remote_list_dir_command(str(root), 2))
+    assert "BUILD" not in proc.stdout
+    entries = parse_remote_list_dir_output(proc.stdout, str(root), pipeline_exit_code=proc.returncode)
+    assert entries == [str(root), str(root / "report.txt")]