Zheng Feng dcb2e687d5
feat(channels): add GitHub as a webhook-driven channel (#3754)
* feat(channels): add GitHub event-driven agents (#3754)

Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation.

* fix(llm-middleware): classify bare IndexError as transient

Upstream chat providers occasionally return 200 OK with an empty
generations list (observed against Volces "coding" on
ark.cn-beijing.volces.com). When that happens,
langchain_core.language_models.chat_models.ainvoke raises
``IndexError: list index out of range`` at
``llm_result.generations[0][0].message`` and kills the run.

Treat a bare IndexError reaching the middleware as a transient
upstream-payload glitch and route it through the existing
retry/backoff path instead of failing the whole agent run. The
retry budget and backoff schedule are unchanged.

Adds three regression tests covering the classifier and both the
recover-on-retry and exhausted-retries paths.

* fix(runtime): ignore stale LLM fallback markers from prior runs

When a run on a thread ends with the LLM-error-handling middleware emitting
a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError
empty-generations classification fix lands), that message is persisted to
the thread's checkpoint as part of the messages channel. LangGraph replays
the full message history in `stream_mode="values"` chunks, so every
subsequent run on the same thread re-streams the stale fallback marker —
and the worker's chunk scanner faithfully picks it up, flipping
`RunStatus.success` to `RunStatus.error` for runs that themselves had
no LLM failure at all.

Snapshot the set of pre-existing message ids from the pre-run checkpoint
and thread it through `_extract_llm_error_fallback_message` /
`_try_extract_from_message` as a filter. Markers on history messages are
ignored; markers on fresh messages produced during this run still trip
the error path. Falls back to an empty set when the checkpointer is
absent or the snapshot can't be captured, preserving the prior behavior
on first-run / no-state paths.

Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`)
plus an integration test exercising the full `run_agent` path with a stale
history checkpointer.

* fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs

GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed
the langgraph_sdk default 300s read deadline. The manager's runs.wait
call kept an HTTP stream open for the entire run lifetime, so the long
run blew up with httpx.ReadTimeout and the outer except branch then
released the dedupe key and emitted a false 'internal error' outbound.

The GitHub channel's outbound send is log-only by design: agents post to
the issue/PR via the gh CLI in the sandbox when they choose to comment
or create a PR. There is nothing for the manager to ferry back, so the
long-poll was pure overhead.

This change adds ChannelRunPolicy.fire_and_forget (default False) and
sets it True for the github channel. When fire_and_forget is True,
_handle_chat dispatches via client.runs.create (short POST, returns
once the run is pending) instead of client.runs.wait, and skips the
response-extraction + outbound-publish block. ConflictError on a busy
thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on
the busy case is preserved for any future non-github fire-and-forget
channel.

Other (non-github) channels are unchanged: their policy defaults
fire_and_forget=False and they continue to dispatch via runs.wait.

Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget:
- Default ChannelRunPolicy.fire_and_forget is False.
- The github policy registers fire_and_forget=True.
- github inbound calls runs.create, not runs.wait, with the right kwargs.
- github inbound publishes no outbound on success.
- ConflictError from runs.create still emits THREAD_BUSY_MESSAGE.
- Non-github channels (slack) still dispatch via runs.wait.

* test(lead-agent): accept user_id kwarg in skill-policy test stubs

The two GitHub-channel tests added in #3754 stubbed
_load_enabled_skills_for_tool_policy with a lambda that only accepted
`available_skills` and `app_config`, but the real function (and its call
site in agent.py) also passes `user_id`. This raised TypeError on every
run, failing backend-unit-tests.

Add `user_id=None` to match the three sibling stubs in the same file.

* refactor(gateway): disambiguate context-key set names

The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS
shared a confusable "CONTEXT_ONLY" token in different orders, and the
first broke the _CONTEXT_<X>_KEYS pattern of its sibling
_CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit:

  _CONTEXT_INTERNAL_CALLER_KEYS  - WHO: internal callers (scheduler) only
  _CONTEXT_RUNTIME_ONLY_KEYS     - WHERE: runtime context only, never configurable

Pure rename, no behavior change.
2026-07-04 22:56:24 +08:00

200 lines
8.2 KiB
Python

"""Trigger filter logic for GitHub webhook dispatch.
Pure functions, no I/O. Given an event name, its payload, and the
agent-config's per-event trigger overrides, decide whether to fire the
agent and why (the reason string makes the gateway log line useful).
**Events are opt-in per binding.** If an event name does not appear as a
key in the binding's ``triggers:`` mapping, the agent is **not registered**
for that event — the dispatcher never even loads the agent for it. The
agent's ``config.yaml`` is the single source of truth for "which events
do I care about?".
:data:`DEFAULT_TRIGGERS` still exists, but it is no longer an
event-enablement list. It is the per-event **field-level defaults** that
get merged into the binding's override when an event IS listed. So:
* ``issue_comment: {}`` → registers the agent for ``issue_comment`` and
inherits ``require_mention: True`` from the default. (Same shape as
before — minimal config, sensible defaults.)
* Binding omits ``issue_comment`` entirely → the agent does **not** see
``issue_comment`` events at all. (New behavior.)
Trigger override merge is field-wise via Pydantic's ``exclude_unset``:
fields the binding explicitly set win; fields it omitted fall back to
the default. Fields with no default (``DEFAULT_TRIGGERS[event]`` is
``None``) just use the binding's literal value.
"""
from __future__ import annotations
import re
from typing import Any
from deerflow.config.agents_config import GitHubTriggerConfig
# Per-event field-level defaults. These are merged into a binding's
# override when the event IS listed in the binding's ``triggers:``. They
# no longer enable the event by themselves — the binding must list the
# event for the agent to register for it.
#
# ``None`` means "no per-event defaults; use whatever the binding set
# (or the model's own field defaults)".
DEFAULT_TRIGGERS: dict[str, GitHubTriggerConfig | None] = {
"ping": None,
"issues": None,
"pull_request_review": None,
"pull_request": GitHubTriggerConfig(actions=["opened"]),
"issue_comment": GitHubTriggerConfig(require_mention=True),
"pull_request_review_comment": GitHubTriggerConfig(require_mention=True),
}
def _action(payload: dict[str, Any]) -> str | None:
action = payload.get("action")
return action if isinstance(action, str) else None
def _comment_body(event: str, payload: dict[str, Any]) -> str:
"""Extract the human-typed text to scan for an ``@mention``.
For comment events this is the comment body. For ``issues`` and
``pull_request`` events there is no separate comment — the mention
would be in the issue/PR body itself — so we read that. For
``pull_request_review`` the body is the review summary. Other events
have no user-authored text to mention-check and return ``""``.
"""
if event in ("issue_comment", "pull_request_review_comment"):
body = (payload.get("comment") or {}).get("body")
return body if isinstance(body, str) else ""
if event == "issues":
body = (payload.get("issue") or {}).get("body")
return body if isinstance(body, str) else ""
if event == "pull_request":
body = (payload.get("pull_request") or {}).get("body")
return body if isinstance(body, str) else ""
if event == "pull_request_review":
body = (payload.get("review") or {}).get("body")
return body if isinstance(body, str) else ""
return ""
def _author_login(event: str, payload: dict[str, Any]) -> str | None:
"""Login of the human who triggered the event, for ``allow_authors``."""
if event in ("issue_comment", "pull_request_review_comment"):
login = (payload.get("comment") or {}).get("user", {}).get("login")
elif event == "pull_request":
login = (payload.get("pull_request") or {}).get("user", {}).get("login")
elif event == "pull_request_review":
login = (payload.get("review") or {}).get("user", {}).get("login")
elif event == "issues":
login = (payload.get("issue") or {}).get("user", {}).get("login")
else:
login = (payload.get("sender") or {}).get("login")
return login if isinstance(login, str) else None
def _resolved_trigger(
event: str,
binding_triggers: dict[str, GitHubTriggerConfig],
) -> GitHubTriggerConfig | None:
"""Merge the binding's override with per-event field defaults.
Returns ``None`` if the binding does not list the event at all — the
event is opt-in per binding.
Otherwise, returns a ``GitHubTriggerConfig`` where:
* fields the binding explicitly set win,
* fields the binding omitted fall back to ``DEFAULT_TRIGGERS[event]``,
* and if there is no per-event default the binding's own field
defaults (from the Pydantic model) apply.
Detection of "explicitly set" relies on Pydantic's
``model_fields_set`` — fields not present in the source YAML aren't
counted as set.
"""
override = binding_triggers.get(event)
if override is None:
return None
default = DEFAULT_TRIGGERS.get(event)
if default is None:
return override
# Field-wise merge: take fields the binding explicitly set,
# backfill the rest from the per-event default.
explicit = override.model_dump(exclude_unset=True)
merged = default.model_copy(update=explicit)
return merged
def _mentions(body: str, login: str) -> bool:
"""Return True if ``body`` @-mentions ``login`` with proper boundaries.
GitHub logins are ``[A-Za-z0-9-]+``, so the character immediately
after the login in a mention must NOT be one of those — otherwise
``@deerflow`` would falsely match ``@deerflow-bot`` (a different,
legitimate GitHub user). A plain substring ``in`` check is wrong for
this reason.
Also rejects mentions where the ``@`` is preceded by a login-class
character (e.g. ``foo@deerflow`` inside an email address) to avoid
incidental matches on URLs / pasted addresses.
Match is case-insensitive; GitHub itself is.
"""
pattern = rf"(?:^|[^A-Za-z0-9-])@{re.escape(login)}(?![A-Za-z0-9-])"
return re.search(pattern, body, flags=re.IGNORECASE) is not None
def event_should_fire(
event: str,
payload: dict[str, Any],
trigger: GitHubTriggerConfig,
default_mention_login: str,
) -> tuple[bool, str]:
"""Decide whether ``event`` fires the agent for this binding.
Args:
event: GitHub event name (``X-GitHub-Event``).
payload: Parsed webhook payload.
trigger: Pre-resolved trigger config for this ``(repo, event)``.
The caller (registry) has already merged the binding override
with per-event :data:`DEFAULT_TRIGGERS` field defaults, so this
function does not look the event up in any dict — it just
applies the gates the trigger declares.
default_mention_login: Bot login (without ``@``) used by
``require_mention`` when the trigger doesn't override
``mention_login``. Pass the agent name as a fallback.
Returns:
``(fire, reason)`` where ``fire`` is the decision and ``reason``
is a short label for logging (e.g. ``"action=opened"``,
``"mention"``, ``"disabled"``).
"""
# Action whitelist (e.g. only "opened" PRs).
if trigger.actions is not None:
action = _action(payload)
if action not in trigger.actions:
return False, f"action={action!r} not in {trigger.actions}"
# allow_authors bypasses require_mention entirely. Useful so a repo
# owner can talk to the bot without typing the handle every time.
if trigger.allow_authors:
author = _author_login(event, payload)
if author and author in trigger.allow_authors:
return True, f"allow_authors:{author}"
if trigger.require_mention:
login = trigger.mention_login or default_mention_login
body = _comment_body(event, payload)
# Boundary-aware @-mention match: ``@deerflow`` must NOT match
# ``@deerflow-bot`` (a distinct, legitimate GitHub login). See
# :func:`_mentions` for the full rationale.
if not login or not _mentions(body, login):
return False, f"mention required for @{login}"
# All gates passed.
action = _action(payload)
return True, f"action={action}" if action else "ok"