mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-21 12:06:18 +00:00
6469833886
* fix(skills): close SkillScan bypasses in the skill review gate The public skill review gate re-materialized a package snapshot into a temp directory for SkillScan, but copied only entries the reader had decoded as text and skipped every file under any evals/fixtures/ directory. Executable binaries and nested archives never reached the package rules, and a fixture-shaped path hid any script from the scan. Readers now keep binary bytes as content_base64, the analyzer writes every non-symlink file byte for byte, and only eval fixture SKILL.md samples stay exempt. Files are created exclusively, so a duplicate archive member or a case-folded name fails the scan closed instead of overwriting an earlier file. SkillScan itself skipped any file that was not NUL-free UTF-8. One Latin-1 byte in a comment hid a reverse shell from the review gate, and a NUL byte skipped static analysis at install. Code files that fail strict decoding now raise package-undecodable-script (HIGH) and are analyzed over a lossy decode, so CRITICAL matches keep blocking. "Code file" and "executable magic" were defined separately in the installer and SkillScan and had drifted: SkillScan missed 32-bit little-endian and fat Mach-O variants the installer blocks. Both rules now live in skills/package_files.py, shared by the installer, the export guard, and SkillScan. * docs(changelog): link the skill review gate fix to #5431 * fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables The review analyzer skipped any snapshot entry it could not turn into bytes. Readers only emit such entries for oversized files, and they also mark the snapshot truncated, but content_base64 is optional in the contract, so a reader regression or a hand-built snapshot would silently drop a file from SkillScan. An entry without bytes now fails the scan closed (not_assessed: skillscan) unless the snapshot is truncated, and a text entry without content no longer materializes as an empty file. A real executable under scripts/ is a code file, so SkillScan decoded it lossily and ran the text rules over its string tables. An OpenSSH binary produced a CRITICAL secret-private-key finding from the key-format banner it embeds. An undecodable file with executable magic still reports package-undecodable-script, and its CRITICAL package-executable-binary finding already blocks it, so it now skips the text rules. Decodable files keep full text analysis. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
46 lines
1.9 KiB
Python
46 lines
1.9 KiB
Python
"""Shared classification of skill-package files by name and leading bytes.
|
|
|
|
The installer, the export guard, and SkillScan must agree on which files are
|
|
code and which bytes mark an executable, so both rules live only here.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import PurePath, PurePosixPath
|
|
|
|
CODE_SUFFIXES = frozenset({".bash", ".cjs", ".js", ".mjs", ".php", ".pl", ".ps1", ".py", ".rb", ".sh", ".ts", ".zsh"})
|
|
# Full magics per variant — a shorter shared prefix would also match
|
|
# non-executable data files.
|
|
_EXECUTABLE_MAGIC_PREFIXES = (
|
|
b"\x7fELF", # ELF
|
|
b"MZ", # PE/DOS
|
|
b"\xfe\xed\xfa\xce", # Mach-O 32-bit big-endian
|
|
b"\xfe\xed\xfa\xcf", # Mach-O 64-bit big-endian
|
|
b"\xce\xfa\xed\xfe", # Mach-O 32-bit little-endian
|
|
b"\xcf\xfa\xed\xfe", # Mach-O 64-bit little-endian
|
|
b"\xca\xfe\xba\xbe", # Mach-O fat binary big-endian
|
|
b"\xbe\xba\xfe\xca", # Mach-O fat binary little-endian
|
|
b"\xca\xfe\xba\xbf", # Mach-O fat64 binary big-endian
|
|
b"\xbf\xba\xfe\xca", # Mach-O fat64 binary little-endian
|
|
)
|
|
|
|
|
|
def _posix(path: str | PurePath) -> PurePosixPath:
|
|
return PurePosixPath(str(path).replace("\\", "/"))
|
|
|
|
|
|
def is_code_path(path: str | PurePath) -> bool:
|
|
"""Return whether a package-relative path is code by name: a ``scripts/`` member or a code suffix."""
|
|
posix = _posix(path)
|
|
return (bool(posix.parts) and posix.parts[0] == "scripts") or posix.suffix.lower() in CODE_SUFFIXES
|
|
|
|
|
|
def is_code_file(path: str | PurePath, head: bytes) -> bool:
|
|
"""Return whether a package file is code; an extensionless file also counts when ``head`` starts with a shebang."""
|
|
return is_code_path(path) or (not _posix(path).suffix and head.startswith(b"#!"))
|
|
|
|
|
|
def is_executable_binary_prefix(prefix: bytes) -> bool:
|
|
"""Detect ELF, PE, and Mach-O executables by magic bytes."""
|
|
return prefix.startswith(_EXECUTABLE_MAGIC_PREFIXES)
|