mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
* feat(channels): select the model per conversation via /model IM channels could list models with /models but had no way to choose one: the model for a conversation came only from channel/session config or the server default, so a user who needed a stronger or cheaper model for one task had to reconfigure the whole channel or move to the web UI. - /model <name> pins a model to the current conversation, validated against the caller-visible /api/models list (list membership doubles as the authorization check), persisted in thread metadata so it survives a Gateway restart and applies from the next message in place — unlike /agent use, which must start a new conversation. - The pin is written to all three runtime carriers (run context, configurable, context) after the session layers merge, so it wins over channel/session configured models; like any explicitly requested model it also takes precedence over a Custom Agent's configured model. - /model (no args) reports the effective model and its source through one resolver that mirrors the run merge order (default -> channel -> user, context carrier over configurable, both session-config carriers scanned); /model default clears the pin; /models names the pinned model in a footer. - Pin lookups degrade instead of failing messages: the status query reports "Current model: unknown" on a lookup failure (matching the /models footer), and the reuse-thread preload leaves the in-process cache untouched so the next turn retries a transient threads.get. - Reserve "model" in the slash-skill reserved-name contract (backend, contract fixture, frontend), register the Telegram handler, and document the command in /help, README, and README_zh. * fix(channels): fence model pin cache and mirror runtime model resolution Review follow-up for /model: - Serialize cold pin loads with persist+cache publication under a per-manager lock, so a slow threads.get that captured pre-pin metadata can never overwrite a concurrently committed selection (commands are not run-serialized). - _resolve_configured_model_name now replicates the run path's shallow session merges and applies the real runtime precedence — config.context (preserved by build_run_config, unbeatable by the setdefault-filled body context) > top-level context > configurable — instead of scanning per-layer carriers; a later layer replacing a nested carrier wholesale no longer leaks a stale value into the status reply. - /api/models resolves the trusted bound owner for internal channel calls and filters the list with the owner's role, so /model selection gates on the same principal that run admission admits for that account. Tests: pin-write-vs-in-flight-load race, status compared against the real merged runtime config (parametrized across carriers), nested-carrier replacement, and owner-role filtering with the real RBAC provider; each verified red by reverting exactly its guard. * fix(channels): share model resolvers with the run path and harden the pin lifecycle The /model status reply re-implemented the session-layer merge and the Gateway carrier precedence, and it ignored the custom-agent model fallback — on a thread pinned to a custom agent whose config sets ``model:`` it reported "server default" while the run actually used the agent's model. Review also found the pin lock spanned a network call and a stale pin bricked its conversation behind the generic error fallback. - Extract ``_select_carried_model_name`` as the single carrier-precedence rule; the status reply reuses the run path's exact ``_merge_dicts`` layer shapes, so a future layering change has one place to touch. - Extract ``_resolve_message_assistant_id`` / ``_resolve_assistant_id`` and share the assistant precedence chain between the run path and the status reply. - Fall back to the resolved custom agent's configured ``model`` when no session layer carries one, mirroring the agent factory's ``_resolve_model_name(requested or agent_model_name)``; the reply now distinguishes "(from channel configuration)" and "(from agent configuration)". - Run the cold ``threads.get`` in ``_load_thread_model`` outside ``_model_pin_lock``; only cache publication takes the lock, with a post-GET re-check so an in-flight /model write still wins over the older snapshot. - Detect the run-admission allowlist rejection in ``_handle_message`` and recover: clear the stale pin (durable write plus cache under the lock) and reply with the reset and the /model recovery instead of the generic internal-error fallback. - Document /model and the ``channel_model_name`` pin semantics in backend/app/channels/AGENTS.md. * fix(channels): fence stale-pin recovery and cover streaming admission failures Review round 4 found the stale-pin recovery itself unsafe and incomplete: - Recovery cleared whatever pin was current when the delayed admission rejection arrived, so a rejection for an old pin could erase a replacement selection committed mid-flight (and tell the user the new model was unavailable). ``_stale_pin_recovery`` now takes the failed run's own selection (``expected_pin``, read from its run context) and clears only while that selection is still current under the lock; a replaced or already-reset selection gets a non-destructive reply instead. - Feishu/Telegram/DingTalk streaming turns never reached the recovery hook because ``_handle_streaming_chat`` swallows start_run's 400 into its terminal-error reply. The recovery now runs in the streaming terminal-error path too, delivered as the single final outbound with the same dedupe cleanup; blocking, streaming, and fire-and-forget run-creation paths all handle the rejection at the site that knows the submitted selection. * docs(channels): fit the /model entry within the agent-guidance size budget The base file sits one byte under the AGENTS.md hard limit, so documenting /model requires net-zero growth: keep the command-inventory addition and a compact pin-semantics sentence, and losslessly tighten the surrounding prose (the file was already at the limit). * fix(channels): apply the review suggestions on /model replies Three non-blocking findings from the latest review: - The /models pin footer decorated the failure string too, advertising a pin next to an error that lists nothing. The command now fetches the listing through ``_fetch_gateway_json`` and appends the footer only on a successful fetch. - The /model status reply labeled every session-layer model "from channel configuration", misreporting user- and default-layer models. The label is now "session configuration" for every session layer. - Stale-pin recovery replies read as if the turn went through, but the run was rejected before starting and the triggering message never reached the agent. All recovery variants now end with an explicit "Your message was not processed — please send it again." * fix(channels): recover stale model pins on the follow-up drain path The drain is the fourth run-creation path and the only one without stale-pin recovery: a /model pin that left the allowlist while follow-ups were buffered made every drained runs.create fail admission, requeueing the batch into the same 400 on every cycle with no user-facing recovery — contradicting the documented "stale pins self-clear" invariant. - Compute expected_pin from the drain's run context (mirroring _handle_chat_on_thread, initialized None since _resolve_run_params is itself inside the guarded try) and run _stale_pin_recovery on drain failure before requeueing; the pin self-clears and the requeued batch succeeds on the next cycle. - Pass the caller-authoritative thread_id into _stale_pin_recovery: every call site already holds it, so recovery never re-derives it from the message — a store-mapping rewrite between dispatch and recovery can never clear the wrong conversation's pin, and the lookup's failure mode (and its guard) disappears entirely. - Return StalePinRecovery(text, resolved) so requeueing callers can tell a cleared pin from a repeat failure: the drain logs INFO when the next cycle is free of this rejection and WARNING when recovery could not remove it (persist failed, or the selection came from session configuration, which still carries it) — no silent infinite requeue loop. Drain logs carry structured facts only: the reply text ends with a "please resend" note that would double-process buffered messages retried automatically (GitHub, the only buffering channel, sends log-only anyway). - Reword the config-carried recovery reply: with no pin held, the rejected selection came from session configuration — name that source and the fix instead of claiming a default takeover that would not happen. - Correct the _load_thread_model lock docstring (writes deliberately hold _model_pin_lock across threads.update) and read the agent config for /model status off the event loop via asyncio.to_thread, with a Blockbuster-gated blocking_io regression test. - Pin the under-lock re-check in _stale_pin_recovery with a concurrency test (a /model committed during the lock-wait window wins; recovery persists nothing), mutation-verified — the drain is the only caller running on a background task without the serialized-run lock, so it is the path most exposed to that window. * Revert "fix(channels): recover stale model pins on the follow-up drain path" This reverts commit 0b4652c689afb56791e5e670901c97a4bf44a156. The drain recovery work moves to a dedicated PR stacked on this branch, keeping #6534 scoped to the /model feature itself. * fix(channels): recover stale model pins on the follow-up drain path The drain is the fourth run-creation path and the only one without stale-pin recovery: a /model pin that left the allowlist while follow-ups were buffered made every drained runs.create fail admission, requeueing the batch into the same 400 on every cycle with no user-facing recovery — contradicting the documented "stale pins self-clear" invariant. - Compute expected_pin from the drain's run context (mirroring _handle_chat_on_thread, initialized None since _resolve_run_params is itself inside the guarded try) and run _stale_pin_recovery on drain failure before requeueing; the pin self-clears and the requeued batch succeeds on the next cycle. - Pass the caller-authoritative thread_id into _stale_pin_recovery: every call site already holds it, so recovery never re-derives it from the message — a store-mapping rewrite between dispatch and recovery can never clear the wrong conversation's pin, and the lookup's failure mode (and its guard) disappears entirely. - Return StalePinRecovery(text, resolved) so requeueing callers can tell a cleared pin from a repeat failure: the drain logs INFO when the next cycle is free of this rejection and WARNING when recovery could not remove it (persist failed, or the selection came from session configuration, which still carries it) — no silent infinite requeue loop. Drain logs carry structured facts only: the reply text ends with a "please resend" note that would double-process buffered messages retried automatically (GitHub, the only buffering channel, sends log-only anyway). - Reword the config-carried recovery reply: with no pin held, the rejected selection came from session configuration — name that source and the fix instead of claiming a default takeover that would not happen. - Correct the _load_thread_model lock docstring (writes deliberately hold _model_pin_lock across threads.update) and read the agent config for /model status off the event loop via asyncio.to_thread, with a Blockbuster-gated blocking_io regression test. - Pin the under-lock re-check in _stale_pin_recovery with a concurrency test (a /model committed during the lock-wait window wins; recovery persists nothing), mutation-verified — the drain is the only caller running on a background task without the serialized-run lock, so it is the path most exposed to that window. (cherry picked from commit 0b4652c689afb56791e5e670901c97a4bf44a156) * fix(channels): apply the third review round on /model command replies - Delete the dead kind == "models" branch in _fetch_gateway: /models has rendered the listing inline since the pin-footer fix, so the helper's copy was unreachable and had already diverged (unguarded m["name"], no pin footer). The internal-auth-header test now drives the surviving /memory kind. - Guard the inline /models listing comprehension like the _handle_model_command sibling (isinstance(m, Mapping) and isinstance(m.get("name"), str)), so a malformed payload entry can no longer turn the command into the generic internal-error reply; regression: test_models_command_skips_malformed_entries. - Drop the unused user parameter from _resolve_bound_owner_for_model_list: the resolved owner comes solely from the trusted header, and the parameter implied the caller's own principal shaped the filter. * fix(channels): apply the fourth review round on the /model lifecycle - Preserve fresh model pins across cache eviction: a cold threads.get racing a successful /model write could still publish its stale snapshot after clear-on-pressure evicted the fresh entry — the in-cache re-check cannot see an eviction. The model-pin cache now carries a write generation; a load whose generation moved during the GET refetches once instead of publishing over the committed selection. Regression: test_fresh_model_pin_survives_cache_eviction_during_cold_load (blocked GET, write, eviction, then the refetch wins), mutation-verified. - Resolve context-carried agents before reporting their model: /model status now mirrors the run path's effective-agent rule — explicit message/thread choice overwrites every carrier, then the merged carriers' agent_name (new _select_carried_agent_name, sharing the model-name precedence via _select_carried_value), then the session-layer assistant_id that only setdefaults into context. A session with context.agent_name=coder now reports coder-model instead of server default, and assistant_id=writer alongside context.agent_name=coder reports coder-model instead of writer-model. Regressions: context-carried, context-beats-assistant_id, and thread-pin-beats-context status tests. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
812 lines
33 KiB
Python
812 lines
33 KiB
Python
"""Phase 3 model-level authorization tests.
|
|
|
|
Covers two enforcement layers:
|
|
- Gateway routes (``list_models``, ``get_model``) — request-scoped Principal,
|
|
mirrors Phase 2A's ``resolve_route_permissions``.
|
|
- Runtime model resolution (``_authorize_model_name``) — context-scoped
|
|
Principal with graceful fallback, mirrors Phase 1B's ``apply_tool_authorization``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
from unittest.mock import AsyncMock
|
|
|
|
import pytest
|
|
from fastapi import FastAPI
|
|
from fastapi.testclient import TestClient
|
|
|
|
from app.gateway.routers import models as models_router
|
|
from deerflow.authz.provider import AuthzDecision, AuthzReason
|
|
from deerflow.authz.rbac import RbacAuthorizationProvider
|
|
from deerflow.config.app_config import AppConfig
|
|
from deerflow.config.authorization_config import AuthorizationConfig
|
|
from deerflow.config.model_config import ModelConfig
|
|
from deerflow.config.sandbox_config import SandboxConfig
|
|
from deerflow.config.token_usage_config import TokenUsageConfig
|
|
|
|
# ── Helpers ────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _user(**overrides):
|
|
values = {
|
|
"id": "user-123",
|
|
"system_role": "user",
|
|
"oauth_provider": "github",
|
|
"oauth_id": "oauth-456",
|
|
}
|
|
values.update(overrides)
|
|
return SimpleNamespace(**values)
|
|
|
|
|
|
def _make_app_config(model_names: list[str]) -> AppConfig:
|
|
"""Build a minimal AppConfig with the given model names."""
|
|
return AppConfig(
|
|
models=[ModelConfig(name=n, model=n, use="langchain_openai:ChatOpenAI") for n in model_names],
|
|
sandbox=SandboxConfig(use="deerflow.sandbox.local:LocalSandboxProvider"),
|
|
token_usage=TokenUsageConfig(enabled=False),
|
|
authorization=AuthorizationConfig(),
|
|
)
|
|
|
|
|
|
def _enable_authorization(monkeypatch, provider, *, fail_closed: bool = True, default_role: str = "user") -> None:
|
|
config = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=fail_closed,
|
|
default_role=default_role,
|
|
)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", lambda c: provider)
|
|
|
|
|
|
def _make_models_app(app_config: AppConfig) -> FastAPI:
|
|
"""Build a FastAPI app with the models router and a pinned config."""
|
|
app = FastAPI()
|
|
app.include_router(models_router.router)
|
|
# Pin the config dependency so routes use our test AppConfig.
|
|
app.dependency_overrides[models_router.get_config] = lambda: app_config
|
|
return app
|
|
|
|
|
|
class _RecordingProvider:
|
|
"""Provider that records all requests and can deny/error specific targets."""
|
|
|
|
name = "recording"
|
|
|
|
def __init__(self, *, denied: set[str] | None = None, errors: set[str] | None = None) -> None:
|
|
self.denied = denied or set()
|
|
self.errors = errors or set()
|
|
self.authorize_requests: list = []
|
|
self.filter_requests: list = []
|
|
|
|
def authorize(self, request):
|
|
self.authorize_requests.append(request)
|
|
if request.target in self.errors:
|
|
raise RuntimeError(f"provider failed for {request.target}")
|
|
allowed = request.target not in self.denied
|
|
return AuthzDecision(
|
|
allow=allowed,
|
|
reasons=[AuthzReason(code="authz.allowed" if allowed else "authz.denied")],
|
|
)
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
self.filter_requests.append((resource_type, list(candidates)))
|
|
if resource_type in self.errors:
|
|
raise RuntimeError(f"provider failed for {resource_type}")
|
|
return [c for c in candidates if c not in self.denied]
|
|
|
|
|
|
# ── list_models tests ──────────────────────────────────────────────────
|
|
|
|
|
|
def test_list_models_disabled_returns_all(monkeypatch):
|
|
"""When authorization is disabled, all models are visible."""
|
|
config = AuthorizationConfig(enabled=False)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
cached = AsyncMock(side_effect=AssertionError("disabled must not resolve provider"))
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", cached)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
cached.assert_not_called()
|
|
|
|
|
|
def test_list_models_anonymous_user_returns_all(monkeypatch):
|
|
"""Anonymous requests (user=None) are not filtered."""
|
|
provider = _RecordingProvider()
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=None),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
assert provider.filter_requests == []
|
|
|
|
|
|
def test_list_models_rbac_filters_by_allow(monkeypatch):
|
|
"""Role with allowlist sees only allowed models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4"]
|
|
|
|
|
|
def test_list_models_rbac_filters_by_deny(monkeypatch):
|
|
"""Role with deny hides denied models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": "*", "deny": ["claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "llama-3"]
|
|
|
|
|
|
def test_list_models_wildcard_returns_all(monkeypatch):
|
|
"""Role with allow: '*' sees all models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": "*"}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_count"),
|
|
[(True, 0), (False, 3)],
|
|
)
|
|
def test_list_models_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_count):
|
|
"""Provider error → empty (fail-closed) or all (fail-open)."""
|
|
provider = _RecordingProvider(errors={"model"})
|
|
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
assert len(response.json()["models"]) == expected_count
|
|
|
|
|
|
def _internal_caller_app(app_config: AppConfig, internal_user) -> FastAPI:
|
|
"""Models app whose request state mirrors AuthMiddleware's internal path."""
|
|
|
|
app = _make_models_app(app_config)
|
|
|
|
@app.middleware("http")
|
|
async def _stamp_internal_state(request, call_next):
|
|
request.state.user = internal_user
|
|
return await call_next(request)
|
|
|
|
return app
|
|
|
|
|
|
def test_list_models_internal_caller_with_bound_owner_uses_owner_role(monkeypatch):
|
|
"""Internal channel call with a trusted owner header filters by the owner's role.
|
|
|
|
Regression for the /model selection gate: with ``default_role=user``
|
|
restricted to cheap models, a bound admin must still see premium models —
|
|
run admission accepts them for that account, so the list (which doubles as
|
|
the selection validator) must too. Uses the real RBAC provider so the
|
|
owner role flows through the same principal pipeline as production.
|
|
"""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={
|
|
"user": {"models": {"allow": ["cheap"]}},
|
|
"admin": {"models": {"allow": "*"}},
|
|
},
|
|
)
|
|
_enable_authorization(monkeypatch, provider, default_role="user")
|
|
|
|
internal_user = _user(id="__internal__", system_role="internal", oauth_provider=None, oauth_id=None)
|
|
admin_owner = _user(id="admin-1", system_role="admin", oauth_provider=None, oauth_id=None)
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=internal_user),
|
|
)
|
|
monkeypatch.setattr(
|
|
"app.gateway.services.get_local_provider",
|
|
lambda: SimpleNamespace(get_user=AsyncMock(return_value=admin_owner)),
|
|
)
|
|
|
|
app_config = _make_app_config(["cheap", "premium"])
|
|
app = _internal_caller_app(app_config, internal_user)
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/api/models", headers={"X-DeerFlow-Owner-User-Id": "admin-1"})
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["cheap", "premium"]
|
|
|
|
|
|
def test_list_models_internal_caller_without_owner_still_uses_default_role(monkeypatch):
|
|
"""Internal calls without a trusted owner keep the default_role filter.
|
|
|
|
Pins the documented unbound-channel posture: no owner header → the
|
|
synthetic principal still falls under ``default_role``.
|
|
"""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["cheap"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider, default_role="user")
|
|
|
|
internal_user = _user(id="__internal__", system_role="internal", oauth_provider=None, oauth_id=None)
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=internal_user),
|
|
)
|
|
|
|
app_config = _make_app_config(["cheap", "premium"])
|
|
app = _internal_caller_app(app_config, internal_user)
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["cheap"]
|
|
|
|
|
|
# ── get_model tests ────────────────────────────────────────────────────
|
|
|
|
|
|
def test_get_model_disabled_returns_model(monkeypatch):
|
|
"""When authorization is disabled, get_model works as before."""
|
|
config = AuthorizationConfig(enabled=False)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["name"] == "gpt-4"
|
|
|
|
|
|
def test_get_model_404_when_not_found(monkeypatch):
|
|
"""Non-existent model returns 404 regardless of authorization."""
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": "*"}}})
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/nonexistent")
|
|
|
|
assert response.status_code == 404
|
|
|
|
|
|
def test_get_model_denied_returns_403(monkeypatch):
|
|
"""Role denied model:use → 403 (not 404)."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 403
|
|
|
|
|
|
def test_get_model_allowed_returns_200(monkeypatch):
|
|
"""Role allowed model:use → 200."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["name"] == "gpt-4"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_status"),
|
|
[(True, 403), (False, 200)],
|
|
)
|
|
def test_get_model_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
|
|
"""Provider error on model:use → 403 (fail-closed) or 200 (fail-open)."""
|
|
provider = _RecordingProvider(errors={"gpt-4"})
|
|
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == expected_status
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_status"),
|
|
[(True, 403), (False, 200)],
|
|
)
|
|
def test_get_model_provider_unavailable_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
|
|
"""Provider *resolution* failure → 403 (fail-closed) or 200 (fail-open).
|
|
|
|
Distinct from ``test_get_model_provider_error_fail_closed_vs_open``: that
|
|
test exercises a provider that resolves but errors inside ``authorize``.
|
|
This one exercises ``_AuthorizationUnavailable`` (the provider cannot be
|
|
resolved at all, e.g. misconfigured class path) and pins the fail-open
|
|
path so the docstring's "provider resolution error yields 403 (fail-closed)
|
|
or allows the request (fail-open)" claim is backed by a test.
|
|
"""
|
|
config = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=fail_closed,
|
|
default_role="user",
|
|
)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
|
|
# Force provider resolution to raise → _AuthorizationUnavailable.
|
|
def _boom(_config):
|
|
raise RuntimeError("provider class path invalid")
|
|
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", _boom)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == expected_status
|
|
|
|
|
|
# ── Runtime model resolution tests (_authorize_model_name) ─────────────
|
|
|
|
|
|
def _rbac_context(**overrides):
|
|
"""Build a minimal run context dict for build_principal_from_context."""
|
|
values = {
|
|
"user_id": "user-123",
|
|
"user_role": "user",
|
|
"oauth_provider": "github",
|
|
"oauth_id": "oauth-456",
|
|
"is_internal": False,
|
|
}
|
|
values.update(overrides)
|
|
return values
|
|
|
|
|
|
def _enable_runtime_authorization(monkeypatch, provider) -> AuthorizationConfig:
|
|
"""Patch resolve_authorization_provider in agent.py to return *provider*.
|
|
|
|
Returns an enabled AuthorizationConfig the caller assigns to app_config.
|
|
"""
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
return AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
|
|
def test_authorize_model_name_disabled_is_noop():
|
|
"""When authorization is disabled, model name is returned unchanged."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
# AuthorizationConfig() defaults to enabled=False.
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_allowed_returns_same(monkeypatch):
|
|
"""Allowed model → returned unchanged."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_denied_falls_back_gracefully(monkeypatch):
|
|
"""Denied model → falls back to first allowed model (RFC §9)."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["claude-3"]}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "claude-3"
|
|
|
|
|
|
def test_authorize_model_name_all_denied_fail_closed_raises(monkeypatch):
|
|
"""All models denied + fail_closed → ValueError."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": []}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
with pytest.raises(ValueError, match="No models are authorized"):
|
|
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
|
|
|
|
def test_authorize_model_name_all_denied_fail_open_returns_original(monkeypatch):
|
|
"""All models denied + fail_open → returns original model name."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": []}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=False,
|
|
default_role="user",
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_list_vs_use_divergence(monkeypatch):
|
|
"""Custom provider that allows 'list' but denies 'use' → model is denied.
|
|
|
|
Regression for willem-bd's forward-looking note: a custom provider that
|
|
distinguishes ``list`` from ``use`` must not let a model through the
|
|
runtime path just because ``filter_resources`` includes it. The runtime
|
|
path checks ``authorize("model", "use")`` first; only on deny does it
|
|
fall back to ``filter_resources`` to pick a replacement.
|
|
"""
|
|
|
|
class _ListButNotUseProvider:
|
|
"""Allows listing gpt-4 but denies using it."""
|
|
|
|
name = "list-not-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use" and request.target == "gpt-4":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
# gpt-4 is "visible" (listable) but not "usable"
|
|
return list(candidates)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _ListButNotUseProvider(),
|
|
)
|
|
|
|
# gpt-4 is listable but denied for use → falls back to claude-3
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result != "gpt-4"
|
|
assert result == "claude-3"
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_closed(monkeypatch):
|
|
"""All visible models denied for use + fail_closed → ValueError.
|
|
|
|
Regression for willem-bd's edge-case note: when ``filter_resources``
|
|
returns only models that are themselves denied for ``use``, the fallback
|
|
must NOT silently reselect a denied model. With ``fail_closed=True`` it
|
|
must raise; with ``fail_closed=False`` it returns the original name.
|
|
"""
|
|
|
|
class _AllListNoneUseProvider:
|
|
"""Lists all models but denies use for every one of them."""
|
|
|
|
name = "all-list-none-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
return list(candidates) # all visible
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _AllListNoneUseProvider(),
|
|
)
|
|
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
# gpt-4 denied for use; fallback candidates also denied → ValueError
|
|
with pytest.raises(ValueError, match="No models are authorized"):
|
|
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_open(monkeypatch):
|
|
"""All visible models denied for use + fail_open → returns original name."""
|
|
|
|
class _AllListNoneUseProvider:
|
|
name = "all-list-none-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
return list(candidates)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=False, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _AllListNoneUseProvider(),
|
|
)
|
|
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
# ── DeerFlowClient._ensure_agent path ─────────────────────────────────
|
|
# Regression for willem-bd's Round 4 coverage observation: the embedded/library
|
|
# lead-agent construction path (``DeerFlowClient._ensure_agent``) must enforce
|
|
# ``model:use`` too, not just the Gateway runtime path (``_make_lead_agent``).
|
|
# Otherwise a consumer that enables ``authorization`` with role-scoped model
|
|
# policies gets tools filtered yet can still run a model the role is denied
|
|
# ``use`` for, diverging from the contract this PR establishes.
|
|
|
|
|
|
def test_client_ensure_agent_enforces_model_use_when_authorized(monkeypatch):
|
|
"""``_ensure_agent`` routes the resolved model through ``_authorize_model_name``.
|
|
|
|
Real-path test: we let the genuine ``_authorize_model_name`` run against a
|
|
real RBAC provider (only ``resolve_authorization_provider`` is patched, as
|
|
in the runtime tests above) so the full ``client → authz gate → RBAC →
|
|
fallback → create_chat_model`` chain is exercised — not just "the gate was
|
|
called". The provider allows only ``claude-3`` for ``use``, so the denied
|
|
``gpt-4`` must be swapped for ``claude-3`` before reaching the model factory.
|
|
"""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
|
|
client = _bare_client(app_config)
|
|
config: RunnableConfig = {"configurable": {"model_name": "gpt-4", "user_id": "user-123", "user_role": "user"}}
|
|
client._ensure_agent(config)
|
|
|
|
# Denied ``gpt-4`` was swapped for the authorized fallback ``claude-3``.
|
|
assert captured_name["name"] == "claude-3"
|
|
assert client._effective_model_name == "claude-3"
|
|
|
|
|
|
def test_client_ensure_agent_resolves_none_default_before_authorization(monkeypatch):
|
|
"""A ``None`` model name is resolved to the default before the authz gate.
|
|
|
|
Guards the ``create_chat_model(name=None)`` semantic: when the caller omits
|
|
``model_name`` the implicit default (first configured model) must still pass
|
|
``model:use`` — otherwise the embedded path could run an unauthorized default.
|
|
"""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
# Deny the default ``gpt-4``; the gate must fallback to ``claude-3``.
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
|
|
client = _bare_client(app_config)
|
|
# No model_name supplied → defaults to ``gpt-4`` (first configured) → denied → fallback.
|
|
config: RunnableConfig = {"configurable": {"user_id": "user-123", "user_role": "user"}}
|
|
client._ensure_agent(config)
|
|
|
|
assert captured_name["name"] == "claude-3"
|
|
assert client._effective_model_name == "claude-3"
|
|
|
|
|
|
def test_client_ensure_agent_noop_when_authorization_disabled(monkeypatch):
|
|
"""When ``authorization.enabled`` is false, ``_ensure_agent`` leaves the model unchanged."""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
# AuthorizationConfig() defaults to enabled=False.
|
|
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
client = _bare_client(app_config)
|
|
config: RunnableConfig = {"configurable": {"model_name": "gpt-4"}}
|
|
client._ensure_agent(config)
|
|
|
|
# Disabled → gate is a no-op: original name passed straight through.
|
|
assert captured_name["name"] == "gpt-4"
|
|
|
|
|
|
def _stub_client_assembly(monkeypatch) -> dict[str, str]:
|
|
"""Stub the heavy dependencies ``_ensure_agent`` pulls in after the authz gate.
|
|
|
|
Returns a dict the caller can inspect to see what ``create_chat_model`` got.
|
|
Everything here is downstream of the contract under test, so we replace it
|
|
with no-ops to keep the test focused on the ``_authorize_model_name`` call.
|
|
"""
|
|
captured: dict[str, str] = {}
|
|
monkeypatch.setattr(
|
|
"deerflow.client.create_chat_model",
|
|
lambda **kwargs: captured.__setitem__("name", kwargs.get("name")) or object(),
|
|
)
|
|
monkeypatch.setattr("deerflow.client.create_agent", lambda **kwargs: object())
|
|
monkeypatch.setattr("deerflow.client.build_middlewares", lambda *args, **kwargs: [])
|
|
monkeypatch.setattr("deerflow.client.DeerFlowClient._get_tools", staticmethod(lambda *, model_name, subagent_enabled, mcp_plugins=None: [])) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_enabled_skills_for_config", lambda app_config, **kw: []) # noqa: ARG005
|
|
monkeypatch.setattr(
|
|
"deerflow.client.build_skill_search_setup",
|
|
lambda skills, *, enabled, container_base_path, skill_authorization=None: SimpleNamespace(describe_skill_tool=None, skill_names=frozenset()), # noqa: ARG005
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.client.assemble_deferred_tools",
|
|
lambda tools, *, enabled: ([], SimpleNamespace(deferred_names=frozenset())), # noqa: ARG005
|
|
)
|
|
monkeypatch.setattr("deerflow.client.build_mcp_routing_middleware", lambda *args, **kwargs: None) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_mcp_routing_hints_prompt_section", lambda *args, **kwargs: "") # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.apply_prompt_template", lambda **kwargs: "") # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_thread_state_schema", lambda *args, **kwargs: object()) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.normalize_middleware_state_schemas", lambda schemas, mode, freq: []) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_effective_user_id", lambda: "user-123")
|
|
# ``apply_tool_authorization`` (called with the empty tool list above) still
|
|
# resolves a provider via ``tool_filter.resolve_authorization_provider``; route
|
|
# it at an allow-all RBAC provider so the empty list stays empty. The skill
|
|
# filter (added in Phase 3 Skills PR) resolves via ``skill_filter`` namespace.
|
|
monkeypatch.setattr(
|
|
"deerflow.authz.tool_filter.resolve_authorization_provider",
|
|
lambda config: RbacAuthorizationProvider(roles={"user": {"tools": {"allow": "*"}}}),
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.authz.skill_filter.resolve_authorization_provider",
|
|
lambda config: RbacAuthorizationProvider(roles={"user": {"skills": {"allow": "*"}}}),
|
|
)
|
|
return captured
|
|
|
|
|
|
def _bare_client(app_config):
|
|
"""Construct a ``DeerFlowClient`` without running ``__init__``."""
|
|
from deerflow.client import DeerFlowClient
|
|
|
|
client = DeerFlowClient.__new__(DeerFlowClient)
|
|
client._app_config = app_config
|
|
client._agent_name = "default"
|
|
client._available_skills = None
|
|
client._checkpoint_channel_mode = "full"
|
|
client._checkpoint_snapshot_frequency = None
|
|
client._middlewares = []
|
|
client._agent = None
|
|
client._agent_config_key = None
|
|
# Non-None so ``_ensure_agent`` skips the real (postgres/sqlite) checkpointer
|
|
# resolution — the value is never used because ``create_agent`` is stubbed.
|
|
client._checkpointer = object()
|
|
return client
|