Zheng Feng dcb2e687d5
feat(channels): add GitHub as a webhook-driven channel (#3754)
* feat(channels): add GitHub event-driven agents (#3754)

Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation.

* fix(llm-middleware): classify bare IndexError as transient

Upstream chat providers occasionally return 200 OK with an empty
generations list (observed against Volces "coding" on
ark.cn-beijing.volces.com). When that happens,
langchain_core.language_models.chat_models.ainvoke raises
``IndexError: list index out of range`` at
``llm_result.generations[0][0].message`` and kills the run.

Treat a bare IndexError reaching the middleware as a transient
upstream-payload glitch and route it through the existing
retry/backoff path instead of failing the whole agent run. The
retry budget and backoff schedule are unchanged.

Adds three regression tests covering the classifier and both the
recover-on-retry and exhausted-retries paths.

* fix(runtime): ignore stale LLM fallback markers from prior runs

When a run on a thread ends with the LLM-error-handling middleware emitting
a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError
empty-generations classification fix lands), that message is persisted to
the thread's checkpoint as part of the messages channel. LangGraph replays
the full message history in `stream_mode="values"` chunks, so every
subsequent run on the same thread re-streams the stale fallback marker —
and the worker's chunk scanner faithfully picks it up, flipping
`RunStatus.success` to `RunStatus.error` for runs that themselves had
no LLM failure at all.

Snapshot the set of pre-existing message ids from the pre-run checkpoint
and thread it through `_extract_llm_error_fallback_message` /
`_try_extract_from_message` as a filter. Markers on history messages are
ignored; markers on fresh messages produced during this run still trip
the error path. Falls back to an empty set when the checkpointer is
absent or the snapshot can't be captured, preserving the prior behavior
on first-run / no-state paths.

Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`)
plus an integration test exercising the full `run_agent` path with a stale
history checkpointer.

* fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs

GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed
the langgraph_sdk default 300s read deadline. The manager's runs.wait
call kept an HTTP stream open for the entire run lifetime, so the long
run blew up with httpx.ReadTimeout and the outer except branch then
released the dedupe key and emitted a false 'internal error' outbound.

The GitHub channel's outbound send is log-only by design: agents post to
the issue/PR via the gh CLI in the sandbox when they choose to comment
or create a PR. There is nothing for the manager to ferry back, so the
long-poll was pure overhead.

This change adds ChannelRunPolicy.fire_and_forget (default False) and
sets it True for the github channel. When fire_and_forget is True,
_handle_chat dispatches via client.runs.create (short POST, returns
once the run is pending) instead of client.runs.wait, and skips the
response-extraction + outbound-publish block. ConflictError on a busy
thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on
the busy case is preserved for any future non-github fire-and-forget
channel.

Other (non-github) channels are unchanged: their policy defaults
fire_and_forget=False and they continue to dispatch via runs.wait.

Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget:
- Default ChannelRunPolicy.fire_and_forget is False.
- The github policy registers fire_and_forget=True.
- github inbound calls runs.create, not runs.wait, with the right kwargs.
- github inbound publishes no outbound on success.
- ConflictError from runs.create still emits THREAD_BUSY_MESSAGE.
- Non-github channels (slack) still dispatch via runs.wait.

* test(lead-agent): accept user_id kwarg in skill-policy test stubs

The two GitHub-channel tests added in #3754 stubbed
_load_enabled_skills_for_tool_policy with a lambda that only accepted
`available_skills` and `app_config`, but the real function (and its call
site in agent.py) also passes `user_id`. This raised TypeError on every
run, failing backend-unit-tests.

Add `user_id=None` to match the three sibling stubs in the same file.

* refactor(gateway): disambiguate context-key set names

The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS
shared a confusable "CONTEXT_ONLY" token in different orders, and the
first broke the _CONTEXT_<X>_KEYS pattern of its sibling
_CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit:

  _CONTEXT_INTERNAL_CALLER_KEYS  - WHO: internal callers (scheduler) only
  _CONTEXT_RUNTIME_ONLY_KEYS     - WHERE: runtime context only, never configurable

Pure rename, no behavior change.
2026-07-04 22:56:24 +08:00

176 lines
6.5 KiB
Python

import re
from abc import ABC, abstractmethod
from deerflow.sandbox.search import GrepMatch
# POSIX env-var name rule: letter or underscore, then letters/digits/underscores.
# Used to validate ``env`` keys before they reach a sandbox implementation.
# No current implementation splices a key into a shell string — the local
# sandbox passes the dict to ``subprocess.run(env=...)`` (no shell), the AIO
# sandbox forwards it via the ``bash.exec`` structured ``env`` field, and e2b
# forwards it as the SDK's ``envs``. The check is defense-in-depth for the
# contract: a future shell-splicing implementation must not have to re-derive
# its own rule.
_ENV_NAME_PATTERN = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
def _validate_extra_env(extra_env: dict[str, str] | None) -> None:
"""Reject ``env`` keys that are not valid POSIX env-var names.
The :meth:`Sandbox.execute_command` contract accepts arbitrary ``str``
keys. Today no implementation splices a key into a shell string — the
local sandbox passes the dict to ``subprocess.run(env=...)`` (no shell),
the AIO sandbox forwards it via the ``bash.exec`` structured ``env``
field (no command-string splice), and e2b forwards it as the SDK's
``envs``. Enforcing the POSIX env-name rule in the abstract layer is
defense-in-depth for the contract: a future implementation that does
route a key through a shell must not have to re-derive its own
validation rule, and a caller passing a key derived from config /
payload / user input fails fast with ``ValueError`` instead of silently
producing an exploit should a future implementation regress to splicing.
Raises:
ValueError: When ``extra_env`` is not None and any key does not
match ``^[A-Za-z_][A-Za-z0-9_]*$``. ``None`` and empty dicts
pass through unchanged.
"""
if not extra_env:
return
for key in extra_env:
if not isinstance(key, str) or not _ENV_NAME_PATTERN.fullmatch(key):
raise ValueError(f"extra_env key {key!r} is not a valid POSIX environment variable name (must match ^[A-Za-z_][A-Za-z0-9_]*$). This protects shell-using sandbox implementations from command injection via the key.")
class Sandbox(ABC):
"""Abstract base class for sandbox environments"""
_id: str
def __init__(self, id: str):
self._id = id
@property
def id(self) -> str:
return self._id
@abstractmethod
def execute_command(
self,
command: str,
env: dict[str, str] | None = None,
timeout: float | None = None,
) -> str:
"""Execute bash command in sandbox.
Args:
command: The command to execute.
env: Optional per-call environment variables to inject into the
command's process. Used to pass request-scoped secrets (e.g. a
short-lived end-user token for skill scripts, issue #3861, or a
GitHub App installation token for ``git push`` / ``gh``) without
placing them in the prompt, tool arguments, or the command
string. When ``None`` the sandbox uses its default environment.
Keys must be valid POSIX environment-variable names
(``^[A-Za-z_][A-Za-z0-9_]*$``); implementations validate
via :func:`_validate_extra_env` before use. Values are
arbitrary strings — shell-using implementations
``shlex.quote`` them on splice.
timeout: Optional per-call wall-clock timeout in seconds. Local
sandboxes use this to bound host bash commands so long-lived
foreground processes cannot hang a turn indefinitely. Remote/AIO
implementations may ignore it when their backend does not expose
an equivalent command-timeout control separate from its own API
timeouts.
Returns:
The standard or error output of the command.
Raises:
ValueError: when an ``env`` key is not a valid env-var name.
"""
pass
@abstractmethod
def read_file(self, path: str) -> str:
"""Read the content of a file.
Args:
path: The absolute path of the file to read.
Returns:
The content of the file.
"""
pass
@abstractmethod
def download_file(self, path: str) -> bytes:
"""Download the binary content of a file.
Args:
path: The absolute path of the file to download.
Returns:
Raw file bytes.
Raises:
PermissionError: If path traversal is detected or the path is outside
the allowed virtual prefix.
OSError: If the file cannot be read or does not exist. Both local
and remote implementations must raise ``OSError`` so callers
have a single exception type to handle.
"""
pass
@abstractmethod
def list_dir(self, path: str, max_depth=2) -> list[str]:
"""List the contents of a directory.
Args:
path: The absolute path of the directory to list.
max_depth: The maximum depth to traverse. Default is 2.
Returns:
The contents of the directory.
"""
pass
@abstractmethod
def write_file(self, path: str, content: str, append: bool = False) -> None:
"""Write content to a file.
Args:
path: The absolute path of the file to write to.
content: The text content to write to the file.
append: Whether to append the content to the file. If False, the file will be created or overwritten.
"""
pass
@abstractmethod
def glob(self, path: str, pattern: str, *, include_dirs: bool = False, max_results: int = 200) -> tuple[list[str], bool]:
"""Find paths that match a glob pattern under a root directory."""
pass
@abstractmethod
def grep(
self,
path: str,
pattern: str,
*,
glob: str | None = None,
literal: bool = False,
case_sensitive: bool = False,
max_results: int = 100,
) -> tuple[list[GrepMatch], bool]:
"""Search for matches inside text files under a directory."""
pass
@abstractmethod
def update_file(self, path: str, content: bytes) -> None:
"""Update a file with binary content.
Args:
path: The absolute path of the file to update.
content: The binary content to write to the file.
"""
pass