Nan Gao c542185a7f
feat(extensions): add gateway contribution points and packaged extension management (#4780)
* feat(extensions): add gateway services and routers

* feat(extensions): add standalone reference extension

* fix(extensions): harden contributed gateway routes

* docs(extensions): document gateway contribution points

* feat(extensions): add operator CLI for packaged extension management

Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.

Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.

Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.

Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.

Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.

Co-authored-by: Codex <codex@openai.com>

* fix(extensions): stop the managed plugins rewrite from destroying config

Two data-safety defects in the managed `plugins:` block writer.

The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.

The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.

Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.

* fix(extensions): harden the manager transaction and correct its docs

Follow-up hardening on the extension package manager.

Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
  `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
  the interpreter that the entry-point probe then imports and calls, and every
  later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
  validation the HTTPS-only source rule depends on. Neither is an index, proxy,
  cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
  `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
  on case transitions, so only the separated spellings were caught. Short
  generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
  the package's build backend, so a config the manager could never write to must
  fail before that code runs rather than afterwards through rollback.

Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
  runs without `--locked` when the checkout had no lock, so uv writes one while
  resolving; if that sync then failed, the restore was skipped and the operator
  kept a lock file they never had. A failing recovery sync now also reports the
  original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
  dependency resolve invites a second interrupt that escapes the handler and
  strands the checkout mid-transaction; the declarations are already restored
  and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
  gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
  contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
  line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
  builder is a different machine, but unlike an environment-driven wheelhouse
  resolution this is a source the operator typed deliberately, so it is reported
  rather than rolled back. Private-network indexes are untouched: a builder on
  that network can reach them.

Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.

* test(gateway): pin the request-path projection agreement

`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.

Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:

- projection edge cases, including the segment-boundary guard that keeps
  root_path="/api" from slicing "/apifoo/models" into a string the router
  would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
  mounted under the "/health" public prefix must still 401, and a POST
  mounted under "/api/webhooks" must still require a CSRF token

Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).

Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.

* ci: pin uv to the version production ships

ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.

backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.

Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.

Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.

* fix(gateway): state the extension route auth limit and abort a failed dev sync

Two scoped review follow-ups.

README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.

docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.

Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.

* fix(extensions): point Git SSH shorthand at the HTTPS correction

Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.

The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.

* docs: keep the root extension summary within its new budget

#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.

---------

Co-authored-by: Codex <codex@openai.com>
2026-08-13 23:55:30 +08:00

309 lines
11 KiB
Python

"""Command-line entry point and launch-mode planning for the DeerFlow TUI.
``plan_launch`` is a pure decision function (fully unit-tested): given argv, TTY
state and the environment, it decides whether to open the terminal UI or run a
headless one-shot. ``main`` wires that decision to the embedded ``DeerFlowClient``
and lazily imports the Textual app only when actually launching the UI, so the
``deerflow`` console script still runs headless commands without Textual present.
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Literal
_UNSET = object()
Mode = Literal["tui", "print", "json", "headless-help"]
@dataclass
class LaunchPlan:
mode: Mode
message: str | None = None
read_stdin: bool = False
thread_id: str | None = None
continue_recent: bool = False
forced_tui: bool = False
transparent: bool = False
recursion_limit: int | None = None
reason: str = ""
def _positive_int(value: str) -> int:
try:
parsed = int(value)
except ValueError as exc:
raise argparse.ArgumentTypeError("must be a positive integer") from exc
if parsed <= 0:
raise argparse.ArgumentTypeError("must be a positive integer")
return parsed
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="deerflow",
description="DeerFlow terminal workbench — a TUI over the embedded DeerFlow harness.",
epilog="Extension management: deerflow extensions --help",
add_help=True,
)
parser.add_argument("message", nargs="*", help="initial prompt for the TUI, or message in --cli mode")
parser.add_argument(
"--print",
dest="print",
nargs="?",
const=None,
default=_UNSET,
metavar="MESSAGE",
help="headless one-shot: print the final answer and exit (reads stdin if no MESSAGE)",
)
parser.add_argument(
"--json",
dest="json",
nargs="?",
const=None,
default=_UNSET,
metavar="MESSAGE",
help="headless streaming: emit newline-delimited JSON StreamEvents and exit",
)
parser.add_argument("--tui", action="store_true", help="force the terminal UI (error if unavailable)")
parser.add_argument(
"--tui-transparent",
action="store_true",
help="use the terminal's default background in the TUI",
)
parser.add_argument("--cli", action="store_true", help="force headless/classic mode for one invocation")
parser.add_argument("--continue", dest="continue_recent", action="store_true", help="resume the most recent thread")
parser.add_argument("--resume", dest="resume", metavar="THREAD", default=None, help="resume a thread by id or title")
parser.add_argument(
"--recursion-limit",
type=_positive_int,
metavar="N",
help="headless agent-loop super-step limit (default: 100)",
)
return parser
def _strip_chat(argv: Sequence[str]) -> list[str]:
"""Accept an optional leading ``chat`` subcommand as an alias for the default surface."""
argv = list(argv)
if argv and argv[0] == "chat":
return argv[1:]
return argv
def _truthy(value: object) -> bool:
return isinstance(value, str) and value.strip().lower() in {"1", "true", "yes", "on"}
def plan_launch(
argv: Sequence[str],
*,
stdin_isatty: bool,
stdout_isatty: bool,
env: dict[str, str],
) -> LaunchPlan:
"""Decide what surface to launch. Pure: no I/O, no client construction."""
parser = build_parser()
args = parser.parse_args(_strip_chat(argv))
positional = " ".join(args.message).strip() or None
resume = args.resume
continue_recent = bool(args.continue_recent)
headless_requested = args.print is not _UNSET or args.json is not _UNSET or args.cli
if args.recursion_limit is not None and not headless_requested:
parser.error("--recursion-limit requires --print, --json, or --cli")
if args.print is not _UNSET:
message = args.print if isinstance(args.print, str) else None
if message is None and stdin_isatty:
return LaunchPlan(mode="headless-help", reason="--print needs a MESSAGE argument or piped stdin.")
return LaunchPlan(
mode="print",
message=message,
read_stdin=message is None,
thread_id=resume,
continue_recent=continue_recent,
recursion_limit=args.recursion_limit,
)
if args.json is not _UNSET:
message = args.json if isinstance(args.json, str) else None
if message is None and stdin_isatty:
return LaunchPlan(mode="headless-help", reason="--json needs a MESSAGE argument or piped stdin.")
return LaunchPlan(
mode="json",
message=message,
read_stdin=message is None,
thread_id=resume,
continue_recent=continue_recent,
recursion_limit=args.recursion_limit,
)
if args.cli:
if positional:
return LaunchPlan(
mode="print",
message=positional,
thread_id=resume,
continue_recent=continue_recent,
recursion_limit=args.recursion_limit,
)
# Mirror --print: a piped message or --continue is enough to run headless.
if continue_recent or not stdin_isatty:
return LaunchPlan(
mode="print",
message=None,
read_stdin=True,
thread_id=resume,
continue_recent=continue_recent,
recursion_limit=args.recursion_limit,
)
return LaunchPlan(
mode="headless-help",
reason='--cli needs a message. Try: deerflow --print "your question".',
)
forced_tui = bool(args.tui)
transparent = bool(args.tui_transparent) or _truthy(env.get("DEER_FLOW_TUI_TRANSPARENT"))
if forced_tui or _truthy(env.get("DEER_FLOW_TUI")) or (stdin_isatty and stdout_isatty):
return LaunchPlan(
mode="tui",
message=positional,
thread_id=resume,
continue_recent=continue_recent,
forced_tui=forced_tui,
transparent=transparent,
)
return LaunchPlan(
mode="headless-help",
message=positional,
thread_id=resume,
continue_recent=continue_recent,
reason="No interactive terminal detected. Use --print MESSAGE for one-shot output, or --tui to force the UI.",
)
# --------------------------------------------------------------------------- #
# Runtime dispatch (not unit-tested here; covered by smoke + integration).
# --------------------------------------------------------------------------- #
_HEADLESS_HELP = """\
deerflow — DeerFlow terminal workbench
deerflow launch the terminal UI (TTY required)
deerflow --tui force the terminal UI
deerflow --tui-transparent use the terminal's default background
deerflow --continue resume the most recent thread in the UI
deerflow --resume THREAD resume a thread by id or title
deerflow --print "question" one-shot answer to stdout
deerflow --json "question" stream newline-delimited JSON events
deerflow --recursion-limit N --print "question"
set the headless agent-loop super-step limit
deerflow extensions --help install and manage trusted Python extensions
echo "question" | deerflow --print
"""
def _resolve_message(plan: LaunchPlan) -> str:
if plan.read_stdin:
return sys.stdin.read().strip()
return plan.message or ""
def _run_overrides(plan: LaunchPlan) -> dict[str, int]:
if plan.recursion_limit is None:
return {}
return {"recursion_limit": plan.recursion_limit}
def main(argv: Sequence[str] | None = None) -> int:
argv = list(sys.argv[1:] if argv is None else argv)
if argv and argv[0] == "extensions":
from deerflow.extensions.cli import main as extensions_main
return extensions_main(argv[1:])
plan = plan_launch(
argv,
stdin_isatty=sys.stdin.isatty(),
stdout_isatty=sys.stdout.isatty(),
env=dict(os.environ),
)
if plan.mode == "headless-help":
if plan.reason:
print(plan.reason, file=sys.stderr)
print(_HEADLESS_HELP, file=sys.stderr)
return 0 if not plan.reason else 2
if plan.mode == "print":
return _run_print(plan)
if plan.mode == "json":
return _run_json(plan)
return _run_tui(plan)
def _make_session():
# Imported lazily so the pure planning path never imports the heavy harness.
# Headless one-shots never use the threads_meta writer, so skip persistence
# (no background loop / engine / connection pool just to discard it).
from .session import open_session
return open_session(persistence=False)
def _run_print(plan: LaunchPlan) -> int:
message = _resolve_message(plan)
if not message:
print("No message provided.", file=sys.stderr)
return 2
session = _make_session()
thread_id = session.resolve_thread(plan)
answer = session.client.chat(message, thread_id=thread_id, **_run_overrides(plan))
print(answer)
return 0
def _run_json(plan: LaunchPlan) -> int:
message = _resolve_message(plan)
if not message:
print("No message provided.", file=sys.stderr)
return 2
session = _make_session()
thread_id = session.resolve_thread(plan)
for event in session.client.stream(message, thread_id=thread_id, **_run_overrides(plan)):
payload = {"type": event.type, "data": event.data}
sys.stdout.write(json.dumps(payload, ensure_ascii=False, default=str) + "\n")
sys.stdout.flush()
return 0
def _run_tui(plan: LaunchPlan) -> int:
try:
# Absolute import (not `from .app`) so the harness import-boundary check,
# which records relative module names verbatim, doesn't mistake the sibling
# `deerflow.tui.app` module for the forbidden top-level `app` package.
from deerflow.tui.app import run_tui
except ModuleNotFoundError as exc: # textual missing
if getattr(exc, "name", "") == "textual" or "textual" in str(exc):
msg = "The terminal UI needs the optional 'textual' dependency.\nInstall it with: uv pip install 'deerflow-harness[tui]' (or: pip install textual)\n"
if plan.forced_tui:
print(msg, file=sys.stderr)
return 1
print(msg + "\nFalling back to headless help:\n", file=sys.stderr)
print(_HEADLESS_HELP, file=sys.stderr)
return 0
raise
return run_tui(plan)
if __name__ == "__main__":
raise SystemExit(main())