mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
fix(sandbox): exempt awk field arithmetic from the shell-variable path guard (#7023)
* fix(sandbox): exempt awk field arithmetic from the shell-variable path guard Since #6930 every `$NAME/` in a host-bash command was treated as a variable-prefixed host path unless both sides were digits, so legitimate awk programs such as `awk '{print $NF/2}' data.txt` or `awk '{print $sum/$n}' data.txt` on files under /mnt/user-data were rejected with "Unsafe absolute paths". Exempt a `$` only inside the single-quoted inline program of awk, gawk, mawk or nawk (after `-F`, `-v` and `--`), where bash expands nothing and awk reads `$NAME` as a field reference. awk string literals, option values, input operands, double-quoted programs, text re-read by `sh -c` or `eval`, heredocs, awk functions/aliases, unknown options such as gawk's `--assign`, `-f` programs, and programs whose `"` may sit in a regex or comment keep the strict scan. Fixes bytedance/deer-flow#7021 * fix(sandbox): read awk string literals and comments before judging quotes Review follow-up: the quote-reliability check gave up the awk exemption for any `/` not preceded by an identifier, number or `]`, including a `/` inside a plain string literal, so `awk '{n=split($0,p,"/"); print $1/p[n]}'` was still rejected. Walk the program once with the same string-literal tracker that computes the code spans: `/` and `#` inside a literal are text, `#` outside one starts a comment to end of line, and only a `/` outside a literal that cannot be division (regex literal) forfeits the exemption. * fix(sandbox): scan the awk exemption the way the shell parses the command Review follow-up for the awk field-arithmetic exemption. Bypasses closed: a backslash-newline let `aw\<nl>k() { eval "$1"; }` define an awk function past the literal blocker regex, so any continuation now forfeits the exemption; the word scanner emitted `)`, `$(` and `>|` as command separators, so an `awk` that was only an argument (`echo $(true) awk '...'`) looked like a command; `$'...'`, comments and quotes nested in `"$(...)"` could desynchronise quote pairing; the alias/function blocker only matched literal spellings and ignored eval, source, hash, PATH, autoload and zsh's `functions[awk]=` / `aliases[awk]=` arrays. The scanner now follows bash: single and double quotes with escapes, `#` comments at a word start, fd redirections, the usual separators, and `$(...)` as a nested command context with fresh quoting. Anything it does not model (backslash-newline, backticks, `$'...'`, heredocs, an unmatched `)`, unbalanced quotes) returns no exemption. The command word is found like `_validate_local_bash_shell_tokens` finds it: after assignments, prefix keywords and wrappers, reduced to its basename, inside every `$(...)`. A command that could redefine awk or re-read quoted text gets no exemption at all, and the docstring says the enumeration is best-effort. False positives removed: `($2+$3)/2` and `length($0)/2` (a `/` after `)` is division unless the `)` closes an if/while/for condition, and the absolute-path scan is now exempt inside awk code too), `/usr/bin/awk`, `LC_ALL=C awk`, `echo "$(awk ...)"`, `"<<"` inside an awk string. The awk spans are computed lazily, only when a `$NAME/` or absolute-path match needs them, and the division check scans backward instead of re-slicing the program. * fix(sandbox): exempt only single-segment absolute-path matches in awk code Review follow-up. Inside awk code spans the absolute-path scan now skips only a match with no second `/` (`/2`, `/n`, `/NR`), so a `/a/b` shape is still rejected even if a span were ever misjudged; previously `awk '{print ($1)/etc/passwd}'` and `awk 'BEGIN{x = 1 /etc/shadow}'` were accepted. `enable` joins the commands that withhold the exemption, since `enable -f` can load a builtin under any name. The division check reads the preceding `)` through the backward scan instead of re-slicing the program for every `/`. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
AochenShen99
GitHub
parent
b6ce047510
commit
8a1acf8009
20
CHANGELOG.md
20
CHANGELOG.md
@ -6022,6 +6022,26 @@ This release closes that milestone with **621 merged pull requests**.
|
||||
names the same file as the equivalent relative path, unless the command
|
||||
rebinds `PWD` itself. Variables inside URLs are unaffected. The guard
|
||||
remains best-effort and is not an isolation boundary. ([#6930])
|
||||
- **sandbox:** With host bash enabled, the local path guard accepts awk
|
||||
field arithmetic again, such as `awk '{print $NF/2}' data.txt`,
|
||||
`awk '{print $sum/$n}' data.txt` and `($2+$3)/2`. Since [#6930] every
|
||||
`$NAME/` was treated as a variable-prefixed host path unless both sides were
|
||||
digits, and a `/` after `)` had always read as an absolute path. A `$` or
|
||||
`/` inside the single-quoted inline program of `awk`, `gawk`, `mawk` or
|
||||
`nawk` (after options such as `-F` and `-v`, found the way the shell finds
|
||||
the command word: after an assignment prefix, with a path reduced to its
|
||||
basename, inside `$(...)`) is now exempt, because bash expands nothing there
|
||||
and awk reads it as a field reference or division; for the absolute-path
|
||||
scan only a single-segment match such as `/2` or `/NR` is exempt, so a
|
||||
`/a/b` shape stays rejected even inside awk code. Text inside awk string
|
||||
literals, options such as `-v`, input operands, double-quoted programs and
|
||||
text re-read by `sh -c` or `eval` are still scanned. No exemption is granted
|
||||
to a command that could redefine `awk` or re-read quoted text (`alias`,
|
||||
`function`, `name()`, `eval`, `source`, `.`, `hash`, `PATH=`, a heredoc), nor
|
||||
to syntax the scanner does not model (a backslash-newline, backticks,
|
||||
`$'...'`, an unmatched `)`), nor to a program whose `"` may sit in an awk
|
||||
regex literal; `/` and `#` inside an awk string literal are plain text.
|
||||
(issue #7021)
|
||||
- **sandbox:** On the local sandbox, `ls` now accepts the `/mnt/user-data`
|
||||
root itself and lists `workspace`, `uploads` and `outputs` as it does on
|
||||
AIO. The provider maps that root on purpose, but the tool layer allowed
|
||||
|
||||
@ -6,6 +6,7 @@ import os
|
||||
import posixpath
|
||||
import re
|
||||
import shlex
|
||||
import string
|
||||
from collections.abc import AsyncIterator, Callable, Iterator
|
||||
from contextlib import asynccontextmanager, contextmanager
|
||||
from contextvars import ContextVar
|
||||
@ -78,6 +79,10 @@ _ABSOLUTE_PATH_PATTERN = re.compile(r"(?<![:\w])(?<!:/)/(?:[^\s\"'`;&|<>()]+)")
|
||||
# through the absolute-path regex because ``}``, ``~``, ``@`` and ``?`` are not
|
||||
# word chars. The remainder is captured in a lookahead so a skipped prefix does
|
||||
# not hide a later one (``$1/$2/etc/passwd``).
|
||||
# The one structural exemption is a ``$`` inside the single-quoted inline
|
||||
# program of an awk-family command (``awk '{print $NF/2}'``): bash expands
|
||||
# nothing there and awk reads ``$NAME`` as a field reference. See
|
||||
# ``_awk_program_field_spans`` for what keeps that exemption narrow.
|
||||
_SHELL_VARIABLE_PATH_PATTERN = re.compile(r"\$([A-Za-z_][A-Za-z0-9_]*|[0-9])/(?=([^\s\"'`;&|<>()]*))")
|
||||
# ``~name/`` expands to that user's home (``~root/.ssh/id_rsa``) and slips the
|
||||
# absolute-path regex for the same reason. Tilde expansion only applies at the
|
||||
@ -1179,6 +1184,334 @@ def _split_shell_tokens(command: str) -> list[str]:
|
||||
|
||||
_SHELL_PATH_HARD_TERMINATORS = frozenset("\"'`;&|<>()")
|
||||
|
||||
_AWK_COMMAND_NAMES = frozenset({"awk", "gawk", "mawk", "nawk"})
|
||||
# awk options whose value is the next word. ``-f`` is handled separately: it
|
||||
# means there is no inline program, so the first operand is an input file. Any
|
||||
# other option (gawk's ``--assign``, ``-e``, ...) may also consume the next
|
||||
# word, so it forfeits the exemption.
|
||||
_AWK_OPTIONS_WITH_VALUE = frozenset({"-F", "-v"})
|
||||
# Command words that can make a later ``awk`` word run something other than
|
||||
# awk, or re-read quoted text: definitions (``alias``, ``function``, ``name()``
|
||||
# handled in the walk), ``eval``, sourcing, ``hash -p``, ``enable -f`` (loads a
|
||||
# builtin under any name) and a ``PATH`` rebind.
|
||||
# Any of them anywhere in the command forfeits the exemption for the whole
|
||||
# command; the words are compared after basename reduction.
|
||||
_AWK_EXEMPTION_BLOCKING_COMMANDS = frozenset({".", "alias", "autoload", "enable", "eval", "function", "hash", "source"})
|
||||
# ``name[...]=`` at a command start is never an environment prefix for awk, but
|
||||
# zsh's ``functions[awk]=...`` / ``aliases[awk]=...`` define a function or alias.
|
||||
_SHELL_ARRAY_ASSIGNMENT_PATTERN = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\[")
|
||||
# awk keywords after which ``/`` opens a regex literal (``print /re/``), and the
|
||||
# keywords whose closing ``)`` is followed by a pattern (``if (x) /re/``) rather
|
||||
# than a divisor (``($2+$3)/2``).
|
||||
_AWK_REGEX_KEYWORDS = frozenset({"case", "delete", "do", "else", "exit", "getline", "in", "print", "printf", "return"})
|
||||
_AWK_CONDITION_KEYWORDS = frozenset({"for", "if", "while"})
|
||||
_AWK_WORD_CHARS = frozenset(string.ascii_letters + string.digits + "_")
|
||||
_SHELL_WORD_OPERATOR_CHARS = frozenset(";&|()<>\n\r")
|
||||
|
||||
|
||||
def _awk_word_before(program: str, position: int) -> str | None:
|
||||
"""The identifier, number or ``]`` that ends just before *position*, or None.
|
||||
|
||||
Scans backward over spaces and tabs only, so the token must sit on the same
|
||||
line. Returns ``"]"`` or ``")"`` for a closing bracket, the word for an
|
||||
identifier or number, and None for anything else (operators, punctuation,
|
||||
start of line).
|
||||
"""
|
||||
end = position
|
||||
while end > 0 and program[end - 1] in " \t":
|
||||
end -= 1
|
||||
if end == 0:
|
||||
return None
|
||||
if program[end - 1] in "])":
|
||||
return program[end - 1]
|
||||
start = end
|
||||
while start > 0 and program[start - 1] in _AWK_WORD_CHARS:
|
||||
start -= 1
|
||||
return program[start:end] or None
|
||||
|
||||
|
||||
def _awk_program_code_spans(program: str, offset: int) -> list[tuple[int, int]] | None:
|
||||
"""Spans of *program* (shifted by *offset*) that awk reads as code, not string text.
|
||||
|
||||
A ``$NAME/`` in awk code is a field reference; inside a string literal it
|
||||
is text that ``system("...")`` or ``getline < "..."`` may hand to a shell,
|
||||
so string literals are left out. Rather than lex awk fully, walk the
|
||||
program once: a ``"`` toggles a string literal (``\\`` escapes inside),
|
||||
``#`` outside one starts a comment that runs to the end of the line, and
|
||||
inside a literal ``/`` and ``#`` are plain text (``split($0, p, "/")``).
|
||||
A ``/`` outside a literal must be division, which in awk follows an
|
||||
identifier, number, ``]`` or a ``)`` that does not close an ``if``,
|
||||
``while`` or ``for`` condition, all on the same line. Any other ``/``
|
||||
(``gsub(/"/, ...)``, ``if (x) /re/``, a ``/`` at the start of a line)
|
||||
opens a regex literal whose ``"`` would desynchronise the tracker, so
|
||||
return None and the caller grants no exemption.
|
||||
"""
|
||||
spans: list[tuple[int, int]] = []
|
||||
in_string = False
|
||||
segment_start = 0
|
||||
open_parens: list[int] = []
|
||||
last_closed_opener: int | None = None
|
||||
position = 0
|
||||
while position < len(program):
|
||||
char = program[position]
|
||||
if in_string:
|
||||
if char == "\\":
|
||||
position += 1
|
||||
elif char == '"':
|
||||
in_string = False
|
||||
segment_start = position + 1
|
||||
elif char == '"':
|
||||
spans.append((offset + segment_start, offset + position))
|
||||
in_string = True
|
||||
elif char == "#":
|
||||
newline = program.find("\n", position)
|
||||
position = len(program) if newline == -1 else newline
|
||||
continue
|
||||
elif char == "(":
|
||||
open_parens.append(position)
|
||||
elif char == ")":
|
||||
last_closed_opener = open_parens.pop() if open_parens else None
|
||||
elif char == "/":
|
||||
word = _awk_word_before(program, position)
|
||||
if word == ")":
|
||||
if last_closed_opener is None or _awk_word_before(program, last_closed_opener) in _AWK_CONDITION_KEYWORDS:
|
||||
return None
|
||||
elif word is None or word in _AWK_REGEX_KEYWORDS:
|
||||
return None
|
||||
position += 1
|
||||
if not in_string:
|
||||
spans.append((offset + segment_start, offset + len(program)))
|
||||
return spans
|
||||
|
||||
|
||||
class _ShellWord:
|
||||
"""One word of a scanned shell command.
|
||||
|
||||
``text`` is the word with quotes removed; ``single_quoted`` is the span of
|
||||
the word's content in the original command when the whole word is exactly
|
||||
one single-quoted string, else None. ``kind`` is ``"word"``, ``"op"`` (a
|
||||
command separator, including ``(`` and ``)``) or ``"redir"``.
|
||||
"""
|
||||
|
||||
__slots__ = ("kind", "single_quoted", "text")
|
||||
|
||||
def __init__(self, kind: str, text: str, single_quoted: tuple[int, int] | None = None) -> None:
|
||||
self.kind = kind
|
||||
self.text = text
|
||||
self.single_quoted = single_quoted
|
||||
|
||||
|
||||
def _scan_shell_words(command: str) -> list[list[_ShellWord]] | None:
|
||||
"""Split *command* into word lists, one per command context, bash-style.
|
||||
|
||||
The first list is the top-level command; every ``$(...)`` substitution adds
|
||||
its own list, since quoting starts afresh inside it and its first word is
|
||||
a command. Single quotes, double quotes with ``\\`` escapes, ``\\x``
|
||||
escapes, ``#`` comments at the start of a word, ``2>``-style redirections
|
||||
and the ``;``, ``&&``, ``||``, ``|``, ``|&``, ``&``, ``(``, ``)`` and
|
||||
newline separators follow bash. Returns None, so that no exemption is
|
||||
granted, for anything it does not model: a backslash-newline (bash splices
|
||||
the lines before parsing, which can spell ``awk`` across lines), backticks,
|
||||
``$'...'`` strings, a heredoc, an unmatched ``)`` and unbalanced quotes.
|
||||
"""
|
||||
if "\\\n" in command or "\\\r" in command:
|
||||
return None
|
||||
contexts: list[list[_ShellWord]] = []
|
||||
|
||||
def scan(position: int, nested: bool) -> int | None:
|
||||
items: list[_ShellWord] = []
|
||||
contexts.append(items)
|
||||
text: list[str] = []
|
||||
quoted_spans: list[tuple[int, int]] = []
|
||||
in_word = False
|
||||
plain_so_far = True # no quotes or substitutions seen in this word yet
|
||||
depth = 0
|
||||
n = len(command)
|
||||
|
||||
def flush() -> None:
|
||||
nonlocal text, quoted_spans, in_word, plain_so_far
|
||||
if in_word:
|
||||
joined = "".join(text)
|
||||
span = quoted_spans[0] if len(quoted_spans) == 1 and quoted_spans[0][1] - quoted_spans[0][0] == len(joined) else None
|
||||
items.append(_ShellWord("word", joined, span))
|
||||
text, quoted_spans, in_word, plain_so_far = [], [], False, True
|
||||
|
||||
while position < n:
|
||||
char = command[position]
|
||||
if char in " \t":
|
||||
flush()
|
||||
position += 1
|
||||
elif char == "#" and not in_word:
|
||||
newline = command.find("\n", position)
|
||||
position = n if newline == -1 else newline
|
||||
elif char in _SHELL_WORD_OPERATOR_CHARS:
|
||||
if char in "<>" and in_word and plain_so_far and "".join(text).isdigit():
|
||||
text, in_word = [], False # ``2>``: the digits belong to the operator
|
||||
flush()
|
||||
if char == ")":
|
||||
if depth == 0:
|
||||
return position + 1 if nested else None
|
||||
depth -= 1
|
||||
items.append(_ShellWord("op", ")"))
|
||||
position += 1
|
||||
continue
|
||||
if char == "(":
|
||||
depth += 1
|
||||
end = position + 1
|
||||
group = "&|" if char in "&|" else "<>&|" if char in "<>" else ""
|
||||
while end < n and command[end] in group:
|
||||
end += 1
|
||||
operator = command[position:end]
|
||||
if operator.startswith("<<") and not operator.startswith("<<<"):
|
||||
return None # heredoc bodies are re-read by the shell
|
||||
items.append(_ShellWord("redir" if char in "<>" else "op", operator))
|
||||
position = end
|
||||
elif char == "`":
|
||||
return None
|
||||
elif char == "'":
|
||||
end = command.find("'", position + 1)
|
||||
if end == -1:
|
||||
return None
|
||||
quoted_spans.append((position + 1, end))
|
||||
text.append(command[position + 1 : end])
|
||||
in_word, plain_so_far = True, False
|
||||
position = end + 1
|
||||
elif char == '"':
|
||||
position += 1
|
||||
while position < n and command[position] != '"':
|
||||
if command[position] == "\\":
|
||||
position += 2
|
||||
elif command[position] == "`":
|
||||
return None
|
||||
elif command.startswith("$(", position):
|
||||
after = scan(position + 2, True)
|
||||
if after is None:
|
||||
return None
|
||||
position = after
|
||||
else:
|
||||
position += 1
|
||||
if position >= n:
|
||||
return None
|
||||
text.append("\x00") # opaque: content is never a bare awk program
|
||||
in_word, plain_so_far = True, False
|
||||
position += 1
|
||||
elif command.startswith("$'", position):
|
||||
return None
|
||||
elif command.startswith("$(", position):
|
||||
after = scan(position + 2, True)
|
||||
if after is None:
|
||||
return None
|
||||
text.append("\x00")
|
||||
in_word, plain_so_far = True, False
|
||||
position = after
|
||||
elif char == "\\":
|
||||
if position + 1 >= n:
|
||||
return None
|
||||
text.append(command[position + 1])
|
||||
in_word, plain_so_far = True, False
|
||||
position += 2
|
||||
else:
|
||||
text.append(char)
|
||||
in_word = True
|
||||
position += 1
|
||||
flush()
|
||||
if nested or depth:
|
||||
return None
|
||||
return position
|
||||
|
||||
if scan(0, False) is None:
|
||||
return None
|
||||
return contexts
|
||||
|
||||
|
||||
def _awk_inline_program_span(items: list[_ShellWord], index: int) -> tuple[tuple[int, int] | None, int]:
|
||||
"""Return the span of a wholly single-quoted awk program and the next index."""
|
||||
while index < len(items):
|
||||
item = items[index]
|
||||
if item.kind == "op":
|
||||
break
|
||||
if item.kind == "redir" or item.text in _AWK_OPTIONS_WITH_VALUE:
|
||||
index += 2
|
||||
continue
|
||||
if item.text == "--file" or item.text.startswith(("-f", "--file=")):
|
||||
return None, index + 1
|
||||
if item.text == "--":
|
||||
index += 1
|
||||
break
|
||||
if not item.text.startswith("-") or item.text == "-":
|
||||
break
|
||||
if not item.text.startswith(("-F", "-v")):
|
||||
return None, index + 1
|
||||
index += 1
|
||||
if index < len(items) and items[index].kind == "word" and items[index].single_quoted is not None:
|
||||
return items[index].single_quoted, index + 1
|
||||
return None, index + 1
|
||||
|
||||
|
||||
def _awk_program_field_spans(command: str) -> list[tuple[int, int]]:
|
||||
"""Spans where ``$NAME/`` is an awk field reference rather than a path.
|
||||
|
||||
Only the single-quoted inline program of an awk-family command qualifies:
|
||||
bash performs no expansion inside single quotes and awk reads ``$`` as its
|
||||
field operator. The command word is found the way
|
||||
``_validate_local_bash_shell_tokens`` finds it (after separators,
|
||||
assignments, prefix keywords and ``command``/``builtin``, reduced to its
|
||||
basename), at the top level and inside every ``$(...)``. Text
|
||||
re-interpreted by another shell (``sh -c '...'``, ``eval``, double-quoted
|
||||
programs) never qualifies, because the command word there is not awk. The
|
||||
exemption assumes the word ``awk`` runs the real awk, so a command that
|
||||
defines or could redefine commands (``alias``, ``function``, ``name()``,
|
||||
``autoload``, zsh's ``functions[...]=`` / ``aliases[...]=`` arrays,
|
||||
``eval``, ``source``, ``hash``, ``enable``, ``PATH=``) gets no exemption
|
||||
at all. That
|
||||
list is best-effort, like the rest of this guard. Inside the program, awk string
|
||||
literals are left out: ``system("cat $HOME/x")`` hands that text to a
|
||||
shell. A program whose ``"`` may sit in a regex literal gets no exemption.
|
||||
"""
|
||||
contexts = _scan_shell_words(command)
|
||||
if contexts is None:
|
||||
return []
|
||||
|
||||
field_spans: list[tuple[int, int]] = []
|
||||
for items in contexts:
|
||||
at_command_start = True
|
||||
index = 0
|
||||
while index < len(items):
|
||||
item = items[index]
|
||||
if item.text.startswith("PATH=") or (index + 1 < len(items) and items[index + 1].kind == "op" and items[index + 1].text.startswith("(")):
|
||||
return [] # PATH rebind or ``name ()`` function definition
|
||||
if at_command_start and item.kind == "word" and _SHELL_ARRAY_ASSIGNMENT_PATTERN.match(item.text):
|
||||
return [] # zsh ``functions[awk]=...`` / ``aliases[awk]=...``
|
||||
if item.kind == "op":
|
||||
at_command_start = True
|
||||
index += 1
|
||||
continue
|
||||
if item.kind == "redir":
|
||||
index += 1 if item.text.endswith("&") and item.text not in ("&>", "&>>") else 2
|
||||
continue
|
||||
if not at_command_start:
|
||||
index += 1
|
||||
continue
|
||||
if _is_shell_assignment(item.text):
|
||||
index += 1
|
||||
continue
|
||||
name = item.text.rsplit("/", 1)[-1]
|
||||
if name in _LOCAL_BASH_COMMAND_PREFIX_KEYWORDS or name in _LOCAL_BASH_COMMAND_END_KEYWORDS or name in _LOCAL_BASH_COMMAND_WRAPPERS:
|
||||
index += 1
|
||||
continue
|
||||
if name in _AWK_EXEMPTION_BLOCKING_COMMANDS:
|
||||
return []
|
||||
at_command_start = False
|
||||
if name not in _AWK_COMMAND_NAMES:
|
||||
index += 1
|
||||
continue
|
||||
program, index = _awk_inline_program_span(items, index + 1)
|
||||
if program is None:
|
||||
continue
|
||||
start, end = program
|
||||
field_spans.extend(_awk_program_code_spans(command[start:end], start) or [])
|
||||
return field_spans
|
||||
|
||||
|
||||
def _absolute_path_token_occurrences(command: str, tokens: list[str]) -> list[tuple[int, int, str]]:
|
||||
"""Locate contiguous occurrences of path-bearing shell tokens in *command*.
|
||||
@ -1472,7 +1805,11 @@ def validate_local_bash_command_paths(command: str, thread_data: ThreadDataState
|
||||
validated whole, so a space inside quotes is not treated as a new segment.
|
||||
A shell variable used as a path prefix (``$HOME/...``, ``$1/...``) or a named
|
||||
home (``~root/...``) is rejected like ``${HOME}/...`` and ``~/...``, since its
|
||||
runtime value is unknown here;
|
||||
runtime value is unknown here, except inside the single-quoted inline
|
||||
program of awk/gawk/mawk/nawk (outside awk string literals), where bash
|
||||
expands nothing and ``$NF/2`` or ``($2+$3)/2`` is field arithmetic; that
|
||||
exemption is withheld from any command that could redefine ``awk`` or
|
||||
re-read quoted text (see ``_awk_program_field_spans``);
|
||||
``$PWD/...`` is checked like the relative path it is equivalent to.
|
||||
A windows-incompatible segment still rejects a path that is not already
|
||||
stored on the host; an existing host file may be read or renamed.
|
||||
@ -1492,9 +1829,20 @@ def validate_local_bash_command_paths(command: str, thread_data: ThreadDataState
|
||||
path_token_occurrences = _absolute_path_token_occurrences(command, tokens)
|
||||
url_spans = _non_file_url_spans(command)
|
||||
|
||||
# Inside the single-quoted program of an awk command a ``/`` after ``)``
|
||||
# is division (``($2+$3)/2``); only awk string literals, which the spans
|
||||
# leave out, can carry text to a shell. Only a single-segment match
|
||||
# (``/2``, ``/n``, ``/NR``) is exempt, so anything shaped like ``/a/b``
|
||||
# is still rejected should the scanner ever misjudge a span.
|
||||
awk_field_spans: list[tuple[int, int]] | None = None
|
||||
for match in _ABSOLUTE_PATH_PATTERN.finditer(command):
|
||||
if _is_in_spans(match.start(), url_spans):
|
||||
continue
|
||||
if "/" not in match.group()[1:]:
|
||||
if awk_field_spans is None:
|
||||
awk_field_spans = _awk_program_field_spans(command)
|
||||
if _is_in_spans(match.start(), awk_field_spans):
|
||||
continue
|
||||
absolute_path = match.group()
|
||||
if _is_non_path_literal_fragment(absolute_path):
|
||||
continue
|
||||
@ -1516,6 +1864,10 @@ def validate_local_bash_command_paths(command: str, thread_data: ThreadDataState
|
||||
for match in _SHELL_VARIABLE_PATH_PATTERN.finditer(command):
|
||||
if _is_in_spans(match.start(), url_spans):
|
||||
continue
|
||||
if awk_field_spans is None:
|
||||
awk_field_spans = _awk_program_field_spans(command)
|
||||
if _is_in_spans(match.start(), awk_field_spans):
|
||||
continue
|
||||
name, remainder = match.group(1), match.group(2)
|
||||
if name == "PWD" and pwd_is_workspace:
|
||||
continue
|
||||
|
||||
@ -1084,6 +1084,68 @@ def test_validate_local_bash_command_paths_blocks_brace_expansion_host_paths(com
|
||||
("cat ~root/.ssh/id_rsa", "~root/.ssh/id_rsa"),
|
||||
("ls ~deploy/", "~deploy/"),
|
||||
("make --directory=~deploy/app", "~deploy/app"),
|
||||
# The awk exemption covers only the single-quoted inline program.
|
||||
("awk '{print $1}' $HOME/.ssh/id_rsa", "$HOME/.ssh/id_rsa"),
|
||||
("awk -f prog.awk '$HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("awk -v 'cmd=cat $HOME/.ssh/id_rsa' 'BEGIN{system(cmd)}'", "$HOME/.ssh/id_rsa"),
|
||||
# A string literal inside the program can reach a shell via system().
|
||||
("awk 'BEGIN{system(\"cat $HOME/.ssh/id_rsa\")}'", "$HOME/.ssh/id_rsa"),
|
||||
# A quote inside an awk regex or comment must not hide a later string.
|
||||
('awk \'/"/ {system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'$0 ~ /"/ {system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'{print /"/; system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'{gsub(/"/, ""); system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'{x = "a" # "\nsystem("cat $HOME/.ssh/id_rsa") }\'', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'BEGIN { # "\nsystem("cat $HOME/.ssh/id_rsa") }\'', "$HOME/.ssh/id_rsa"),
|
||||
# In awk a ``/`` after ``)`` or at the start of a line opens a regex.
|
||||
('awk \'{ if (1) /"b/; system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'{ while (0) /"b/; system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
('awk \'NR==1\n/"b/{system("cat $HOME/.ssh/id_rsa")}\' /mnt/user-data/workspace/data.txt', "$HOME/.ssh/id_rsa"),
|
||||
# An unrecognised option may take the next word as its value (gawk's
|
||||
# ``--assign`` is ``-v``), so the following word is not the program.
|
||||
("gawk --assign 'cmd=cat $HOME/.ssh/id_rsa' 'BEGIN{system(cmd)}'", "$HOME/.ssh/id_rsa"),
|
||||
("awk --assign 'cmd=cat $HOME/.ssh/id_rsa' 'BEGIN{system(cmd)}'", "$HOME/.ssh/id_rsa"),
|
||||
# Re-interpreted or expanded program text is not exempt.
|
||||
("sh -c 'awk \"{print \\$HOME/x}\"'", "$HOME/x"),
|
||||
("bash -c 'awk {print $HOME/.ssh/id_rsa}'", "$HOME/.ssh/id_rsa"),
|
||||
("eval awk '{print $HOME/.ssh/id_rsa}'", "$HOME/.ssh/id_rsa"),
|
||||
('awk "{print $HOME/.ssh/id_rsa}"', "$HOME/.ssh/id_rsa"),
|
||||
("bash <<EOF\nawk '{print $HOME/.ssh/id_rsa}'\nEOF", "$HOME/.ssh/id_rsa"),
|
||||
("awk() { bash -c \"$1\"; }; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
# bash splices a backslash-newline before parsing, so these define an
|
||||
# ``awk`` function or alias too; a continuation anywhere forfeits the exemption.
|
||||
('aw\\\nk() { eval "$1"; }; awk \'printf "%s" $HOME/.ssh/id_rsa\'', "$HOME/.ssh/id_rsa"),
|
||||
('function aw\\\nk() { eval "$1"; }; awk \'printf "%s" $HOME/.ssh/id_rsa\'', "$HOME/.ssh/id_rsa"),
|
||||
("alia\\\ns awk=eval; shopt -s expand_aliases; awk 'printf \"%s\" $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
# Commands that can redefine what ``awk`` runs forfeit the exemption.
|
||||
("awk () { eval \"$1\"; }; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("function awk { eval \"$1\"; }; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("hash -p /mnt/user-data/workspace/evil awk; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("PATH=/mnt/user-data/workspace:$PATH awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("export PATH=/mnt/user-data/workspace:$PATH; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("source /mnt/user-data/workspace/defs.sh; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
(". /mnt/user-data/workspace/defs.sh; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("eval \"$(cat /mnt/user-data/workspace/defs.sh)\"; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
# zsh defines functions and aliases through special arrays.
|
||||
("functions[awk]='eval \"$1\"'; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("functions[awk]='eval \"$1\"'; awk 'cat /etc/passwd'", "/etc/passwd"),
|
||||
("aliases[awk]=eval; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("autoload -Uz awk; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
("enable -f /mnt/user-data/workspace/evil.so awk; awk 'cat $HOME/.ssh/id_rsa'", "$HOME/.ssh/id_rsa"),
|
||||
# Inside awk code only a single-segment ``/x`` skips the absolute-path
|
||||
# scan, so a multi-segment path stays rejected even after ``)``.
|
||||
("awk '{print ($1)/etc/passwd}' /mnt/user-data/workspace/data.txt", "/etc/passwd"),
|
||||
("awk 'BEGIN{x = 1 /etc/shadow}'", "/etc/shadow"),
|
||||
# An ``awk`` word that is an argument of another command is not a command.
|
||||
("echo $(true) awk '{print $NF/2}'", "$NF/2"),
|
||||
("echo >| awk '{print $NF/2}'", "$NF/2"),
|
||||
("echo x) awk '{print $NF/2}'", "$NF/2"),
|
||||
("eval $(true) awk '{print $HOME/.ssh/id_rsa}'", "$HOME/.ssh/id_rsa"),
|
||||
("echo `true` awk '{print $HOME/.ssh/id_rsa}'", "$HOME/.ssh/id_rsa"),
|
||||
('echo "$(echo "x") awk \'{print $HOME/.ssh/id_rsa}\'"', "$HOME/.ssh/id_rsa"),
|
||||
# Quoting the scanner does not model fails closed.
|
||||
("awk $'{print $HOME/.ssh/id_rsa}'", "$HOME/.ssh/id_rsa"),
|
||||
("awk '{print $NF/2}' $'/mnt/user-data/workspace/data.txt' 'x' $HOME/.ssh/id_rsa", "$HOME/.ssh/id_rsa"),
|
||||
],
|
||||
)
|
||||
def test_validate_local_bash_command_paths_blocks_variable_prefixed_host_paths(command: str, rejected: str) -> None:
|
||||
@ -1113,6 +1175,38 @@ def test_validate_local_bash_command_paths_blocks_variable_prefixed_host_paths(c
|
||||
# awk field arithmetic is not a path.
|
||||
"du -k /mnt/user-data/workspace | awk '{print $1/1024}'",
|
||||
"awk -F, '{print $1/$2}' /mnt/user-data/uploads/data.csv",
|
||||
# Inside a single-quoted awk program bash expands nothing and ``$``
|
||||
# is awk's field operator (#7021).
|
||||
"awk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"awk '{print $sum/$n}' /mnt/user-data/workspace/data.txt",
|
||||
"awk -F, '{total+=$3} END {print total/NR}' /mnt/user-data/workspace/data.csv",
|
||||
"gawk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"mawk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"nawk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"awk -F: -v n=2 '{print $NF/n}' /mnt/user-data/workspace/data.txt",
|
||||
"cat /mnt/user-data/workspace/data.txt | awk -- '{print $NF/2}' && echo done",
|
||||
"awk '{print \"avg\", $sum/$n}' /mnt/user-data/workspace/data.txt",
|
||||
"awk '{print $NF/2 \" MB\"}' /mnt/user-data/workspace/data.txt",
|
||||
# ``/`` and ``#`` inside an awk string literal are plain text.
|
||||
"awk '{n=split($0,p,\"/\"); print $1/p[n]}' /mnt/user-data/workspace/data.csv",
|
||||
"awk '{split($0, a, \"/\"); print $1/a[1]}' /mnt/user-data/workspace/data.csv",
|
||||
"awk -F'#' '{print \"#\" $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
# ``/`` after ``)`` is division unless the ``)`` closes an if/while/for
|
||||
# condition; the absolute-path scan is exempt inside awk code too.
|
||||
"awk '{avg=($2+$3)/2; print $NF/avg}' /mnt/user-data/workspace/data.txt",
|
||||
"awk '{print ($NF+1)/2, $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"awk '{print length($0)/2}' /mnt/user-data/workspace/data.txt",
|
||||
"awk '{a[$1]/=2; print a[$1]/2}' /mnt/user-data/workspace/data.txt",
|
||||
# The command word is found like the shell does: basename, after an
|
||||
# assignment prefix, inside ``$(...)``, before a comment.
|
||||
"/usr/bin/awk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"LC_ALL=C awk '{print $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"echo \"$(awk '{print $NF/2}' /mnt/user-data/workspace/data.txt)\"",
|
||||
"x=$(awk '{print $NF/2}' /mnt/user-data/workspace/data.txt); echo \"$x\"",
|
||||
"awk '{print $NF/2}' /mnt/user-data/workspace/data.txt # don't worry",
|
||||
"awk '{print \"<<\", $NF/2}' /mnt/user-data/workspace/data.txt",
|
||||
"(awk '{print $NF/2}' /mnt/user-data/workspace/data.txt)",
|
||||
"awk '{print $NF/2}' /mnt/user-data/workspace/data.txt 2>/dev/null",
|
||||
],
|
||||
)
|
||||
def test_validate_local_bash_command_paths_allows_non_path_variable_uses(command: str) -> None:
|
||||
@ -1134,6 +1228,11 @@ def test_validate_local_bash_command_paths_rejects_cdpath_relocation(command: st
|
||||
validate_local_bash_command_paths(command, _THREAD_DATA)
|
||||
|
||||
|
||||
def test_validate_local_bash_command_paths_rejects_absolute_path_after_awk_program() -> None:
|
||||
with pytest.raises(PermissionError, match="/etc/passwd"):
|
||||
validate_local_bash_command_paths("awk '{print $NF/2}' /etc/passwd", _THREAD_DATA)
|
||||
|
||||
|
||||
def test_validate_local_bash_command_paths_rejects_pwd_traversal_like_relative_traversal() -> None:
|
||||
for command in ("cat ../../etc/passwd", "cat $PWD/../../etc/passwd"):
|
||||
with pytest.raises(PermissionError, match="path traversal"):
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user