mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3) Phase 3 / Skills — the second of three resource-type PRs (Models, Skills, Sandbox). The RBAC provider already maps "skill" → config key "skills" (rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed. Layer 1 (assembly-time, mirrors Phase 1B): - filter_available_skills_by_authorization() in skill_filter.py filters the skill-name allowlist (set[str] | None) by the provider's "skill" policy. In lead agent, runs after _available_skill_names() so the catalog (describe_skill) and SkillActivationMiddleware share one filtered set. In subagent executor _load_skills(), filters config.skills before disk load. - filter_resources_by_authorization() in enforcement.py — generic version of filter_tools_by_authorization for any resource with a name attribute. Layer 2 (slash-activation): - No new middleware code needed. SkillActivationMiddleware._resolve_activation already checks "reference.name not in self._available_skills". Since Layer 1 filters the allowlist, denied skills cannot be slash-activated. authorization.enabled: false is a complete no-op. available_skills=None (no agent allowlist) + enabled: resolves all skill names from config and filters. 12 new tests + 243 existing authz/skills tests pass. * test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test * feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates * feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes * chore(authz): align the subagent filter's fallback user with the chain's bucket convention * feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly Review round 4 (willem-bd, CHANGES_REQUESTED): - [P1] Autonomous load path: describe_skill filters its match set on the action-scoped skill:activate decision (all-denied is indistinguishable from "no match"), and ToolErrorHandlingMiddleware stamps skill_context entries only when the action allows it — a denied SKILL.md read gets a skill_context_denied marker instead, so durable context, allowed-tools policy, and autonomous secret bindings never activate a denied skill. File visibility itself stays Layer 1 (sandbox-projection semantics). - [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now receive the authorization-filtered set instead of the raw _BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved when deferred discovery is on rather than round-tripping through None into the legacy full-metadata prompt that re-advertises a denied skill (deferred-off keeps the legacy rendering; the filtered empty allowlist blanks it on denial). - [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per assembly (_resolve_skill_authorization, memoized per executor) and shares it across the Layer 1 filter (authorization=), the activation middleware, and both new gates — uncached provider factories can no longer hand the layers different instances/policy snapshots. All gates share one helper (skill_filter.skill_activation_allowed) so slash, describe, and skill-file-load cannot drift; provider errors keep the configured fail-closed/fail-open policy. skill_authorization is threaded through _build_runtime_middlewares into both runtime builders. Tests: describe gating (deny/fail-closed/fail-open/disabled), read stamping (denied marker vs entry) + extract_skills skip without warning, bootstrap three-way param (allow/deny x deferred on/off), lead + subagent chain wiring for the stamp gate, and the distinct-instance provider factory repro. Each fix is mutation-verified (reverting it fails its test). Full A/B suite run: no new failures vs HEAD. * feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize Review round 5 (willem-bd): - [P1] Persisted entries: skill_context survives across runs, so both consumers now re-check the action-scoped skill:activate decision before reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a denied skill (same skip as a disabled/unallowlisted one; all-denied keeps the existing fail-closed-to-builtins semantics) and _in_context_secret_sources binds no secrets for a denied entry. skill_authorization is threaded into SkillToolPolicyMiddleware on both chains. - [P2] Async provider API: skill_activation_allowed_async mirrors the sync helper through await provider.aauthorize() (same pattern as authorize_sandbox_execution_async). Async hooks precompute an activation_decisions map on the event loop (slash target + persisted entry names, one batch of awaits) and hand it to the worker-thread handler, which consults the map instead of the sync API: SkillActivationMiddleware.awrap_model_call (activation + secret binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp awaits the decision before stamping), and SkillToolPolicyMiddleware's awrap_model_call/awrap_tool_call (entry names + slash source name). describe_skill becomes a StructuredTool with both a sync function (invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async agent execution never routes a loop-affine provider through the sync API while sync callers keep their semantics. Tests: allow-read-then-deny-next-run regressions for secret binding and allowed-tools (partial deny does not poison, all-denied fails closed to builtins), and an _AsyncOnlyProvider (sync authorize always fails, aauthorize works) driving four async-path assertions (activation, stamp allow/deny, describe, tool policy). All six fixes are mutation-verified; full A/B suite run against HEAD shows no new failures. * fix(authz): authorize skill activation by declared Skill.name, not directory basename ShenAC-SAC's P2: bundled skills may declare a name that differs from their directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy), but the runtime gates derived authorization targets from the container path. - New deerflow/skills/container_registry.py: build_container_path_registry() + canonical_skill_name() — the one path-to-Skill.name resolution shared by the activation, tool-policy, and read-stamping middlewares. - ToolErrorHandlingMiddleware resolves the read path through the (user-scoped, via new user_id kwarg threaded from both runtime builders) registry before the skill:activate decision, sync and async (registry lookup in a worker thread, aauthorize() on the loop). RBAC allow by the declared name now activates; allow by the directory name does not. - SkillToolPolicyMiddleware async hooks canonicalize the policy paths first (three-phase: thread registry -> loop aauthorize -> thread filter reusing the registry; the cached per-model-step decision still short-circuits), so the decision map is keyed by the names _active_skills_for_paths checks — no more map miss falling back to the sync provider API from the thread. _entry_names removed: persisted entry["name"] is path-derived by construction; entries are resolved by path instead. - SkillActivationMiddleware collects slash names (already canonical) plus persisted entry *paths*, canonicalized off-loop via _canonical_names_for_paths(). Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC allow declared-name/deny directory-name both directions, async stamp via _AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's tools with zero sync fallback, persisted-entry secret binding survives the name mismatch. All three mutations (stamp target, policy map key, entry canonicalization) fail their tests. * fix(authz): close the sync-fallback class on skill activation paths willem-bd's two P2s on the canonical-name fix, plus a fourth instance of the same class found while sweeping the membership: - SkillToolPolicyMiddleware: a transient prepass registry failure is now preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies its own fail-closed treatment instead of silently retrying storage, which recovered skills absent from the decision map and fell back to the synchronous provider API from the thread (fail-open retaining a denied skill's allowed-tools). - SkillActivationMiddleware: an empty entry-path list no longer pays a full skill-tree scan on ordinary authorization-enabled async model steps (previously a passive path with zero registry I/O). - 4th instance (reproduced with the PR fixtures, then fixed): the entry prepass failure on the secret-binding path let the consumer-side _load_skill_registry_by_path silently recover and resolve entry names outside the decision map — the same sync fallback, fail-open binding the aauthorize-denied skill's declared secret. The fix goes one step further than failure propagation: the prepass hands down the registry *snapshot* the decision map was keyed by, and entry sources resolve against that snapshot, so a map miss is impossible by construction (this also closes the narrower window where both loads succeed but disagree because storage changed mid-step). The slash source is deliberately unaffected: it does not consult the decision map (validated once at activation) and keeps its fresh per-call load. - Class membership swept: skill_activation_allowed() sync call sites are exactly four (stamping sync path and describe's sync implementation are legitimately sync); the only two async-reachable fallback points (the _activation_allowed helpers) are both construction-guarded now. Structural hardening (typed async-batch decision object where a map miss is fail-closed by construction) recorded in the plan notes as a follow-up, anchored at those two helpers. - _RegistryArg type alias replaces the degenerate dict|object|None union in the policy middleware signatures. Tests (test_skills_authorization.py 50->54): transient prepass failure -> builtins-only with exactly one storage attempt and zero provider calls; ordinary async step -> zero registry scans; two-call scenario -> entry binding zeroed while the slash binding survives; entries resolve against the prepass snapshot under a mid-step rename. Mutations verified red: marker reverted to None, over-broad slash kill, snapshot bypass. * fix(authz): correct secret-binding reference points and gate the rendered skill reminder Review round 9 (willem-bd) plus construction-audit follow-ups on the same class — every async divergence between two loads of the same data now has an explicit, pinned reference point: - Slash secret source resolves from a FRESH post-activation registry: the prepass snapshot is taken before the aauthorize awaits, so binding the slash source from it would activate NEW_KEY content while injecting OLD_KEY when a skill's declared secrets change in that window. The 'one scan per step' bonus that motivated sharing the snapshot is withdrawn — consistency wins. The two secret sources resolve independently: a transient fresh-load failure zeroes only the slash binding; entries keep binding from their own snapshot. - Same-skill dominance: when a skill is both the slash source (fresh) and a persisted entry (snapshot), only the activation-era declaration binds — unioning both eras injected a secret the skill no longer declares. Aligns with the tool-policy rule (slash activation dominates for the run); different-skill entries still stack additively. - The rendered 'Active skills' reminder (DurableContextMiddleware) no longer advertises skills the provider denies — surface consistency with describe_skill. Publication/consumption split: the activation middleware publishes per-step path-keyed skill:activate decisions into the run context (async: reused from the prepass snapshot, zero extra I/O; sync: one scan + authorize() only when entries exist — passive steps stay scan-free), the renderer consumes them and filters its render copy (state untouched; unpublished = permissive, matching the other run-context carriers). New carrier key listed in REDACTED_CONTEXT_KEYS with a redaction regression. - Wiring regressions on both chains for everything added above (user_id to the stamp gate, skill_authorization + token pairing + activation- before-durable ordering to the durable renderer): each mutation verified red (M4-M13 across this change set). - Tests 54->65: slash post-activation lookup, source independence, dominance, sync-chain sync-API path, render filter (async/sync/ disabled with load-count assertions), chain wiring x2, redaction listing. Plan notes record rounds 9-2..6 including the reference-point post-mortem and the audit axes exhausted. * fix(authz): anchor same-skill dominance on the slash identity, not binding results Review round 10 (willem-bd) plus a self-audit extension the new checklist items caught in the first fix itself: - R10 (P2): the same-skill exclusion was derived from successfully BOUND slash sources. When the prepass snapshots required-secrets [OLD_KEY] and the operator strips the declaration during the await window, the slash source binds nothing (required_secrets gate) — the exclusion set stayed empty and the stale entry view kept injecting OLD_KEY for a skill that now declares no secrets. The exclusion now derives from the AUTHENTICATED slash activation identity (run-context source path, owner-token checked): the identity's declared name resolves from the fresh registry first, then the entry snapshot, and joins the exclusion set regardless of binding success. - Self-audit (value->gate->downstream on the fix itself): the declared name can also change mid-run (same path, foo->bar). A name-anchored exclusion then holds only the new name while the snapshot entry resolves the same path to the old one — both eras bound again. The PATH is the stable identity shared by the activation and every earlier read: entries at the slash identity's path are now excluded before registry resolution (exclude_paths), independent of what either version calls the skill; the name-based set remains for cross-path same-name shadowing. Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run rename (only the activation-era key binds). Mutations verified red: identity-derived exclusion disabled (M14), path exclusion disabled (M16). * test(authz): pass explicit config to lead middleware wiring test --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
732 lines
30 KiB
Python
732 lines
30 KiB
Python
"""Phase 3 model-level authorization tests.
|
|
|
|
Covers two enforcement layers:
|
|
- Gateway routes (``list_models``, ``get_model``) — request-scoped Principal,
|
|
mirrors Phase 2A's ``resolve_route_permissions``.
|
|
- Runtime model resolution (``_authorize_model_name``) — context-scoped
|
|
Principal with graceful fallback, mirrors Phase 1B's ``apply_tool_authorization``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
from unittest.mock import AsyncMock
|
|
|
|
import pytest
|
|
from fastapi import FastAPI
|
|
from fastapi.testclient import TestClient
|
|
|
|
from app.gateway.routers import models as models_router
|
|
from deerflow.authz.provider import AuthzDecision, AuthzReason
|
|
from deerflow.authz.rbac import RbacAuthorizationProvider
|
|
from deerflow.config.app_config import AppConfig
|
|
from deerflow.config.authorization_config import AuthorizationConfig
|
|
from deerflow.config.model_config import ModelConfig
|
|
from deerflow.config.sandbox_config import SandboxConfig
|
|
from deerflow.config.token_usage_config import TokenUsageConfig
|
|
|
|
# ── Helpers ────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _user(**overrides):
|
|
values = {
|
|
"id": "user-123",
|
|
"system_role": "user",
|
|
"oauth_provider": "github",
|
|
"oauth_id": "oauth-456",
|
|
}
|
|
values.update(overrides)
|
|
return SimpleNamespace(**values)
|
|
|
|
|
|
def _make_app_config(model_names: list[str]) -> AppConfig:
|
|
"""Build a minimal AppConfig with the given model names."""
|
|
return AppConfig(
|
|
models=[ModelConfig(name=n, model=n, use="langchain_openai:ChatOpenAI") for n in model_names],
|
|
sandbox=SandboxConfig(use="deerflow.sandbox.local:LocalSandboxProvider"),
|
|
token_usage=TokenUsageConfig(enabled=False),
|
|
authorization=AuthorizationConfig(),
|
|
)
|
|
|
|
|
|
def _enable_authorization(monkeypatch, provider, *, fail_closed: bool = True, default_role: str = "user") -> None:
|
|
config = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=fail_closed,
|
|
default_role=default_role,
|
|
)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", lambda c: provider)
|
|
|
|
|
|
def _make_models_app(app_config: AppConfig) -> FastAPI:
|
|
"""Build a FastAPI app with the models router and a pinned config."""
|
|
app = FastAPI()
|
|
app.include_router(models_router.router)
|
|
# Pin the config dependency so routes use our test AppConfig.
|
|
app.dependency_overrides[models_router.get_config] = lambda: app_config
|
|
return app
|
|
|
|
|
|
class _RecordingProvider:
|
|
"""Provider that records all requests and can deny/error specific targets."""
|
|
|
|
name = "recording"
|
|
|
|
def __init__(self, *, denied: set[str] | None = None, errors: set[str] | None = None) -> None:
|
|
self.denied = denied or set()
|
|
self.errors = errors or set()
|
|
self.authorize_requests: list = []
|
|
self.filter_requests: list = []
|
|
|
|
def authorize(self, request):
|
|
self.authorize_requests.append(request)
|
|
if request.target in self.errors:
|
|
raise RuntimeError(f"provider failed for {request.target}")
|
|
allowed = request.target not in self.denied
|
|
return AuthzDecision(
|
|
allow=allowed,
|
|
reasons=[AuthzReason(code="authz.allowed" if allowed else "authz.denied")],
|
|
)
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
self.filter_requests.append((resource_type, list(candidates)))
|
|
if resource_type in self.errors:
|
|
raise RuntimeError(f"provider failed for {resource_type}")
|
|
return [c for c in candidates if c not in self.denied]
|
|
|
|
|
|
# ── list_models tests ──────────────────────────────────────────────────
|
|
|
|
|
|
def test_list_models_disabled_returns_all(monkeypatch):
|
|
"""When authorization is disabled, all models are visible."""
|
|
config = AuthorizationConfig(enabled=False)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
cached = AsyncMock(side_effect=AssertionError("disabled must not resolve provider"))
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", cached)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
cached.assert_not_called()
|
|
|
|
|
|
def test_list_models_anonymous_user_returns_all(monkeypatch):
|
|
"""Anonymous requests (user=None) are not filtered."""
|
|
provider = _RecordingProvider()
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=None),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
assert provider.filter_requests == []
|
|
|
|
|
|
def test_list_models_rbac_filters_by_allow(monkeypatch):
|
|
"""Role with allowlist sees only allowed models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4"]
|
|
|
|
|
|
def test_list_models_rbac_filters_by_deny(monkeypatch):
|
|
"""Role with deny hides denied models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": "*", "deny": ["claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "llama-3"]
|
|
|
|
|
|
def test_list_models_wildcard_returns_all(monkeypatch):
|
|
"""Role with allow: '*' sees all models."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": "*"}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
names = [m["name"] for m in response.json()["models"]]
|
|
assert names == ["gpt-4", "claude-3"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_count"),
|
|
[(True, 0), (False, 3)],
|
|
)
|
|
def test_list_models_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_count):
|
|
"""Provider error → empty (fail-closed) or all (fail-open)."""
|
|
provider = _RecordingProvider(errors={"model"})
|
|
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models")
|
|
|
|
assert response.status_code == 200
|
|
assert len(response.json()["models"]) == expected_count
|
|
|
|
|
|
# ── get_model tests ────────────────────────────────────────────────────
|
|
|
|
|
|
def test_get_model_disabled_returns_model(monkeypatch):
|
|
"""When authorization is disabled, get_model works as before."""
|
|
config = AuthorizationConfig(enabled=False)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["name"] == "gpt-4"
|
|
|
|
|
|
def test_get_model_404_when_not_found(monkeypatch):
|
|
"""Non-existent model returns 404 regardless of authorization."""
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": "*"}}})
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/nonexistent")
|
|
|
|
assert response.status_code == 404
|
|
|
|
|
|
def test_get_model_denied_returns_403(monkeypatch):
|
|
"""Role denied model:use → 403 (not 404)."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 403
|
|
|
|
|
|
def test_get_model_allowed_returns_200(monkeypatch):
|
|
"""Role allowed model:use → 200."""
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
|
|
)
|
|
_enable_authorization(monkeypatch, provider)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["name"] == "gpt-4"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_status"),
|
|
[(True, 403), (False, 200)],
|
|
)
|
|
def test_get_model_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
|
|
"""Provider error on model:use → 403 (fail-closed) or 200 (fail-open)."""
|
|
provider = _RecordingProvider(errors={"gpt-4"})
|
|
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == expected_status
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("fail_closed", "expected_status"),
|
|
[(True, 403), (False, 200)],
|
|
)
|
|
def test_get_model_provider_unavailable_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
|
|
"""Provider *resolution* failure → 403 (fail-closed) or 200 (fail-open).
|
|
|
|
Distinct from ``test_get_model_provider_error_fail_closed_vs_open``: that
|
|
test exercises a provider that resolves but errors inside ``authorize``.
|
|
This one exercises ``_AuthorizationUnavailable`` (the provider cannot be
|
|
resolved at all, e.g. misconfigured class path) and pins the fail-open
|
|
path so the docstring's "provider resolution error yields 403 (fail-closed)
|
|
or allows the request (fail-open)" claim is backed by a test.
|
|
"""
|
|
config = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=fail_closed,
|
|
default_role="user",
|
|
)
|
|
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
|
|
|
|
# Force provider resolution to raise → _AuthorizationUnavailable.
|
|
def _boom(_config):
|
|
raise RuntimeError("provider class path invalid")
|
|
|
|
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", _boom)
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
app_config.authorization.fail_closed = fail_closed
|
|
monkeypatch.setattr(
|
|
"app.gateway.routers.models.get_optional_user_from_request",
|
|
AsyncMock(return_value=_user()),
|
|
)
|
|
|
|
with TestClient(_make_models_app(app_config)) as client:
|
|
response = client.get("/api/models/gpt-4")
|
|
|
|
assert response.status_code == expected_status
|
|
|
|
|
|
# ── Runtime model resolution tests (_authorize_model_name) ─────────────
|
|
|
|
|
|
def _rbac_context(**overrides):
|
|
"""Build a minimal run context dict for build_principal_from_context."""
|
|
values = {
|
|
"user_id": "user-123",
|
|
"user_role": "user",
|
|
"oauth_provider": "github",
|
|
"oauth_id": "oauth-456",
|
|
"is_internal": False,
|
|
}
|
|
values.update(overrides)
|
|
return values
|
|
|
|
|
|
def _enable_runtime_authorization(monkeypatch, provider) -> AuthorizationConfig:
|
|
"""Patch resolve_authorization_provider in agent.py to return *provider*.
|
|
|
|
Returns an enabled AuthorizationConfig the caller assigns to app_config.
|
|
"""
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
return AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
|
|
def test_authorize_model_name_disabled_is_noop():
|
|
"""When authorization is disabled, model name is returned unchanged."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
# AuthorizationConfig() defaults to enabled=False.
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_allowed_returns_same(monkeypatch):
|
|
"""Allowed model → returned unchanged."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_denied_falls_back_gracefully(monkeypatch):
|
|
"""Denied model → falls back to first allowed model (RFC §9)."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": ["claude-3"]}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "claude-3"
|
|
|
|
|
|
def test_authorize_model_name_all_denied_fail_closed_raises(monkeypatch):
|
|
"""All models denied + fail_closed → ValueError."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": []}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
|
|
|
|
with pytest.raises(ValueError, match="No models are authorized"):
|
|
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
|
|
|
|
def test_authorize_model_name_all_denied_fail_open_returns_original(monkeypatch):
|
|
"""All models denied + fail_open → returns original model name."""
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
provider = RbacAuthorizationProvider(
|
|
roles={"user": {"models": {"allow": []}}},
|
|
)
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(
|
|
enabled=True,
|
|
fail_closed=False,
|
|
default_role="user",
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_list_vs_use_divergence(monkeypatch):
|
|
"""Custom provider that allows 'list' but denies 'use' → model is denied.
|
|
|
|
Regression for willem-bd's forward-looking note: a custom provider that
|
|
distinguishes ``list`` from ``use`` must not let a model through the
|
|
runtime path just because ``filter_resources`` includes it. The runtime
|
|
path checks ``authorize("model", "use")`` first; only on deny does it
|
|
fall back to ``filter_resources`` to pick a replacement.
|
|
"""
|
|
|
|
class _ListButNotUseProvider:
|
|
"""Allows listing gpt-4 but denies using it."""
|
|
|
|
name = "list-not-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use" and request.target == "gpt-4":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
# gpt-4 is "visible" (listable) but not "usable"
|
|
return list(candidates)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _ListButNotUseProvider(),
|
|
)
|
|
|
|
# gpt-4 is listable but denied for use → falls back to claude-3
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result != "gpt-4"
|
|
assert result == "claude-3"
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_closed(monkeypatch):
|
|
"""All visible models denied for use + fail_closed → ValueError.
|
|
|
|
Regression for willem-bd's edge-case note: when ``filter_resources``
|
|
returns only models that are themselves denied for ``use``, the fallback
|
|
must NOT silently reselect a denied model. With ``fail_closed=True`` it
|
|
must raise; with ``fail_closed=False`` it returns the original name.
|
|
"""
|
|
|
|
class _AllListNoneUseProvider:
|
|
"""Lists all models but denies use for every one of them."""
|
|
|
|
name = "all-list-none-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
return list(candidates) # all visible
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _AllListNoneUseProvider(),
|
|
)
|
|
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
# gpt-4 denied for use; fallback candidates also denied → ValueError
|
|
with pytest.raises(ValueError, match="No models are authorized"):
|
|
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
|
|
|
|
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_open(monkeypatch):
|
|
"""All visible models denied for use + fail_open → returns original name."""
|
|
|
|
class _AllListNoneUseProvider:
|
|
name = "all-list-none-use"
|
|
|
|
def authorize(self, request):
|
|
if request.resource == "model" and request.action == "use":
|
|
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
|
|
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
|
|
|
|
async def aauthorize(self, request):
|
|
return self.authorize(request)
|
|
|
|
def filter_resources(self, principal, resource_type, candidates):
|
|
return list(candidates)
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=False, default_role="user")
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: _AllListNoneUseProvider(),
|
|
)
|
|
|
|
from deerflow.agents.lead_agent.agent import _authorize_model_name
|
|
|
|
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
|
|
assert result == "gpt-4"
|
|
|
|
|
|
# ── DeerFlowClient._ensure_agent path ─────────────────────────────────
|
|
# Regression for willem-bd's Round 4 coverage observation: the embedded/library
|
|
# lead-agent construction path (``DeerFlowClient._ensure_agent``) must enforce
|
|
# ``model:use`` too, not just the Gateway runtime path (``_make_lead_agent``).
|
|
# Otherwise a consumer that enables ``authorization`` with role-scoped model
|
|
# policies gets tools filtered yet can still run a model the role is denied
|
|
# ``use`` for, diverging from the contract this PR establishes.
|
|
|
|
|
|
def test_client_ensure_agent_enforces_model_use_when_authorized(monkeypatch):
|
|
"""``_ensure_agent`` routes the resolved model through ``_authorize_model_name``.
|
|
|
|
Real-path test: we let the genuine ``_authorize_model_name`` run against a
|
|
real RBAC provider (only ``resolve_authorization_provider`` is patched, as
|
|
in the runtime tests above) so the full ``client → authz gate → RBAC →
|
|
fallback → create_chat_model`` chain is exercised — not just "the gate was
|
|
called". The provider allows only ``claude-3`` for ``use``, so the denied
|
|
``gpt-4`` must be swapped for ``claude-3`` before reaching the model factory.
|
|
"""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
|
|
client = _bare_client(app_config)
|
|
config: RunnableConfig = {"configurable": {"model_name": "gpt-4", "user_id": "user-123", "user_role": "user"}}
|
|
client._ensure_agent(config)
|
|
|
|
# Denied ``gpt-4`` was swapped for the authorized fallback ``claude-3``.
|
|
assert captured_name["name"] == "claude-3"
|
|
assert client._effective_model_name == "claude-3"
|
|
|
|
|
|
def test_client_ensure_agent_resolves_none_default_before_authorization(monkeypatch):
|
|
"""A ``None`` model name is resolved to the default before the authz gate.
|
|
|
|
Guards the ``create_chat_model(name=None)`` semantic: when the caller omits
|
|
``model_name`` the implicit default (first configured model) must still pass
|
|
``model:use`` — otherwise the embedded path could run an unauthorized default.
|
|
"""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4", "claude-3"])
|
|
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
|
|
|
|
# Deny the default ``gpt-4``; the gate must fallback to ``claude-3``.
|
|
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
|
|
monkeypatch.setattr(
|
|
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
|
|
lambda config: provider,
|
|
)
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
|
|
client = _bare_client(app_config)
|
|
# No model_name supplied → defaults to ``gpt-4`` (first configured) → denied → fallback.
|
|
config: RunnableConfig = {"configurable": {"user_id": "user-123", "user_role": "user"}}
|
|
client._ensure_agent(config)
|
|
|
|
assert captured_name["name"] == "claude-3"
|
|
assert client._effective_model_name == "claude-3"
|
|
|
|
|
|
def test_client_ensure_agent_noop_when_authorization_disabled(monkeypatch):
|
|
"""When ``authorization.enabled`` is false, ``_ensure_agent`` leaves the model unchanged."""
|
|
from langchain_core.runnables import RunnableConfig
|
|
|
|
app_config = _make_app_config(["gpt-4"])
|
|
# AuthorizationConfig() defaults to enabled=False.
|
|
|
|
captured_name = _stub_client_assembly(monkeypatch)
|
|
client = _bare_client(app_config)
|
|
config: RunnableConfig = {"configurable": {"model_name": "gpt-4"}}
|
|
client._ensure_agent(config)
|
|
|
|
# Disabled → gate is a no-op: original name passed straight through.
|
|
assert captured_name["name"] == "gpt-4"
|
|
|
|
|
|
def _stub_client_assembly(monkeypatch) -> dict[str, str]:
|
|
"""Stub the heavy dependencies ``_ensure_agent`` pulls in after the authz gate.
|
|
|
|
Returns a dict the caller can inspect to see what ``create_chat_model`` got.
|
|
Everything here is downstream of the contract under test, so we replace it
|
|
with no-ops to keep the test focused on the ``_authorize_model_name`` call.
|
|
"""
|
|
captured: dict[str, str] = {}
|
|
monkeypatch.setattr(
|
|
"deerflow.client.create_chat_model",
|
|
lambda **kwargs: captured.__setitem__("name", kwargs.get("name")) or object(),
|
|
)
|
|
monkeypatch.setattr("deerflow.client.create_agent", lambda **kwargs: object())
|
|
monkeypatch.setattr("deerflow.client.build_middlewares", lambda *args, **kwargs: [])
|
|
monkeypatch.setattr("deerflow.client.DeerFlowClient._get_tools", staticmethod(lambda *, model_name, subagent_enabled, mcp_plugins=None: [])) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_enabled_skills_for_config", lambda app_config, **kw: []) # noqa: ARG005
|
|
monkeypatch.setattr(
|
|
"deerflow.client.build_skill_search_setup",
|
|
lambda skills, *, enabled, container_base_path, skill_authorization=None: SimpleNamespace(describe_skill_tool=None, skill_names=frozenset()), # noqa: ARG005
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.client.assemble_deferred_tools",
|
|
lambda tools, *, enabled: ([], SimpleNamespace(deferred_names=frozenset())), # noqa: ARG005
|
|
)
|
|
monkeypatch.setattr("deerflow.client.build_mcp_routing_middleware", lambda *args, **kwargs: None) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_mcp_routing_hints_prompt_section", lambda *args, **kwargs: "") # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.apply_prompt_template", lambda **kwargs: "") # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_thread_state_schema", lambda *args, **kwargs: object()) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.normalize_middleware_state_schemas", lambda schemas, mode, freq: []) # noqa: ARG005
|
|
monkeypatch.setattr("deerflow.client.get_effective_user_id", lambda: "user-123")
|
|
# ``apply_tool_authorization`` (called with the empty tool list above) still
|
|
# resolves a provider via ``tool_filter.resolve_authorization_provider``; route
|
|
# it at an allow-all RBAC provider so the empty list stays empty. The skill
|
|
# filter (added in Phase 3 Skills PR) resolves via ``skill_filter`` namespace.
|
|
monkeypatch.setattr(
|
|
"deerflow.authz.tool_filter.resolve_authorization_provider",
|
|
lambda config: RbacAuthorizationProvider(roles={"user": {"tools": {"allow": "*"}}}),
|
|
)
|
|
monkeypatch.setattr(
|
|
"deerflow.authz.skill_filter.resolve_authorization_provider",
|
|
lambda config: RbacAuthorizationProvider(roles={"user": {"skills": {"allow": "*"}}}),
|
|
)
|
|
return captured
|
|
|
|
|
|
def _bare_client(app_config):
|
|
"""Construct a ``DeerFlowClient`` without running ``__init__``."""
|
|
from deerflow.client import DeerFlowClient
|
|
|
|
client = DeerFlowClient.__new__(DeerFlowClient)
|
|
client._app_config = app_config
|
|
client._agent_name = "default"
|
|
client._available_skills = None
|
|
client._checkpoint_channel_mode = "full"
|
|
client._checkpoint_snapshot_frequency = None
|
|
client._middlewares = []
|
|
client._agent = None
|
|
client._agent_config_key = None
|
|
# Non-None so ``_ensure_agent`` skips the real (postgres/sqlite) checkpointer
|
|
# resolution — the value is never used because ``create_agent`` is stubbed.
|
|
client._checkpointer = object()
|
|
return client
|