deer-flow/backend/tests/test_models_authorization.py
hataa 1cd3eb742b
feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3) (#4541)
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3)

Phase 3 / Skills — the second of three resource-type PRs (Models, Skills,
Sandbox). The RBAC provider already maps "skill" → config key "skills"
(rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed.

Layer 1 (assembly-time, mirrors Phase 1B):
- filter_available_skills_by_authorization() in skill_filter.py filters the
  skill-name allowlist (set[str] | None) by the provider's "skill" policy.
  In lead agent, runs after _available_skill_names() so the catalog
  (describe_skill) and SkillActivationMiddleware share one filtered set.
  In subagent executor _load_skills(), filters config.skills before disk load.
- filter_resources_by_authorization() in enforcement.py — generic version of
  filter_tools_by_authorization for any resource with a name attribute.

Layer 2 (slash-activation):
- No new middleware code needed. SkillActivationMiddleware._resolve_activation
  already checks "reference.name not in self._available_skills". Since Layer 1
  filters the allowlist, denied skills cannot be slash-activated.

authorization.enabled: false is a complete no-op. available_skills=None (no
agent allowlist) + enabled: resolves all skill names from config and filters.
12 new tests + 243 existing authz/skills tests pass.

* test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test

* feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates

* feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes

* chore(authz): align the subagent filter's fallback user with the chain's bucket convention

* feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly

Review round 4 (willem-bd, CHANGES_REQUESTED):

- [P1] Autonomous load path: describe_skill filters its match set on the
  action-scoped skill:activate decision (all-denied is indistinguishable
  from "no match"), and ToolErrorHandlingMiddleware stamps skill_context
  entries only when the action allows it — a denied SKILL.md read gets a
  skill_context_denied marker instead, so durable context, allowed-tools
  policy, and autonomous secret bindings never activate a denied skill.
  File visibility itself stays Layer 1 (sandbox-projection semantics).
- [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now
  receive the authorization-filtered set instead of the raw
  _BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved
  when deferred discovery is on rather than round-tripping through None
  into the legacy full-metadata prompt that re-advertises a denied
  skill (deferred-off keeps the legacy rendering; the filtered empty
  allowlist blanks it on denial).
- [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per
  assembly (_resolve_skill_authorization, memoized per executor) and
  shares it across the Layer 1 filter (authorization=), the activation
  middleware, and both new gates — uncached provider factories can no
  longer hand the layers different instances/policy snapshots.

All gates share one helper (skill_filter.skill_activation_allowed) so
slash, describe, and skill-file-load cannot drift; provider errors keep
the configured fail-closed/fail-open policy. skill_authorization is
threaded through _build_runtime_middlewares into both runtime builders.

Tests: describe gating (deny/fail-closed/fail-open/disabled), read
stamping (denied marker vs entry) + extract_skills skip without warning,
bootstrap three-way param (allow/deny x deferred on/off), lead + subagent
chain wiring for the stamp gate, and the distinct-instance provider
factory repro. Each fix is mutation-verified (reverting it fails its
test). Full A/B suite run: no new failures vs HEAD.

* feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize

Review round 5 (willem-bd):

- [P1] Persisted entries: skill_context survives across runs, so both
  consumers now re-check the action-scoped skill:activate decision before
  reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a
  denied skill (same skip as a disabled/unallowlisted one; all-denied
  keeps the existing fail-closed-to-builtins semantics) and
  _in_context_secret_sources binds no secrets for a denied entry.
  skill_authorization is threaded into SkillToolPolicyMiddleware on both
  chains.
- [P2] Async provider API: skill_activation_allowed_async mirrors the sync
  helper through await provider.aauthorize() (same pattern as
  authorize_sandbox_execution_async). Async hooks precompute an
  activation_decisions map on the event loop (slash target + persisted
  entry names, one batch of awaits) and hand it to the worker-thread
  handler, which consults the map instead of the sync API:
  SkillActivationMiddleware.awrap_model_call (activation + secret
  binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp
  awaits the decision before stamping), and SkillToolPolicyMiddleware's
  awrap_model_call/awrap_tool_call (entry names + slash source name).
  describe_skill becomes a StructuredTool with both a sync function
  (invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async
  agent execution never routes a loop-affine provider through the sync
  API while sync callers keep their semantics.

Tests: allow-read-then-deny-next-run regressions for secret binding and
allowed-tools (partial deny does not poison, all-denied fails closed to
builtins), and an _AsyncOnlyProvider (sync authorize always fails,
aauthorize works) driving four async-path assertions (activation, stamp
allow/deny, describe, tool policy). All six fixes are mutation-verified;
full A/B suite run against HEAD shows no new failures.

* fix(authz): authorize skill activation by declared Skill.name, not directory basename

ShenAC-SAC's P2: bundled skills may declare a name that differs from their
directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy),
but the runtime gates derived authorization targets from the container path.

- New deerflow/skills/container_registry.py: build_container_path_registry()
  + canonical_skill_name() — the one path-to-Skill.name resolution shared by
  the activation, tool-policy, and read-stamping middlewares.
- ToolErrorHandlingMiddleware resolves the read path through the (user-scoped,
  via new user_id kwarg threaded from both runtime builders) registry before
  the skill:activate decision, sync and async (registry lookup in a worker
  thread, aauthorize() on the loop). RBAC allow by the declared name now
  activates; allow by the directory name does not.
- SkillToolPolicyMiddleware async hooks canonicalize the policy paths first
  (three-phase: thread registry -> loop aauthorize -> thread filter reusing
  the registry; the cached per-model-step decision still short-circuits),
  so the decision map is keyed by the names _active_skills_for_paths checks —
  no more map miss falling back to the sync provider API from the thread.
  _entry_names removed: persisted entry["name"] is path-derived by
  construction; entries are resolved by path instead.
- SkillActivationMiddleware collects slash names (already canonical) plus
  persisted entry *paths*, canonicalized off-loop via
  _canonical_names_for_paths().

Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC
allow declared-name/deny directory-name both directions, async stamp via
_AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's
tools with zero sync fallback, persisted-entry secret binding survives the
name mismatch. All three mutations (stamp target, policy map key, entry
canonicalization) fail their tests.

* fix(authz): close the sync-fallback class on skill activation paths

willem-bd's two P2s on the canonical-name fix, plus a fourth instance of
the same class found while sweeping the membership:

- SkillToolPolicyMiddleware: a transient prepass registry failure is now
  preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies
  its own fail-closed treatment instead of silently retrying storage,
  which recovered skills absent from the decision map and fell back to
  the synchronous provider API from the thread (fail-open retaining a
  denied skill's allowed-tools).
- SkillActivationMiddleware: an empty entry-path list no longer pays a
  full skill-tree scan on ordinary authorization-enabled async model
  steps (previously a passive path with zero registry I/O).
- 4th instance (reproduced with the PR fixtures, then fixed): the entry
  prepass failure on the secret-binding path let the consumer-side
  _load_skill_registry_by_path silently recover and resolve entry names
  outside the decision map — the same sync fallback, fail-open binding
  the aauthorize-denied skill's declared secret. The fix goes one step
  further than failure propagation: the prepass hands down the registry
  *snapshot* the decision map was keyed by, and entry sources resolve
  against that snapshot, so a map miss is impossible by construction
  (this also closes the narrower window where both loads succeed but
  disagree because storage changed mid-step). The slash source is
  deliberately unaffected: it does not consult the decision map
  (validated once at activation) and keeps its fresh per-call load.
- Class membership swept: skill_activation_allowed() sync call sites are
  exactly four (stamping sync path and describe's sync implementation are
  legitimately sync); the only two async-reachable fallback points (the
  _activation_allowed helpers) are both construction-guarded now.
  Structural hardening (typed async-batch decision object where a map
  miss is fail-closed by construction) recorded in the plan notes as a
  follow-up, anchored at those two helpers.
- _RegistryArg type alias replaces the degenerate dict|object|None union
  in the policy middleware signatures.

Tests (test_skills_authorization.py 50->54): transient prepass failure ->
builtins-only with exactly one storage attempt and zero provider calls;
ordinary async step -> zero registry scans; two-call scenario -> entry
binding zeroed while the slash binding survives; entries resolve against
the prepass snapshot under a mid-step rename. Mutations verified red:
marker reverted to None, over-broad slash kill, snapshot bypass.

* fix(authz): correct secret-binding reference points and gate the rendered skill reminder

Review round 9 (willem-bd) plus construction-audit follow-ups on the same
class — every async divergence between two loads of the same data now has
an explicit, pinned reference point:

- Slash secret source resolves from a FRESH post-activation registry: the
  prepass snapshot is taken before the aauthorize awaits, so binding the
  slash source from it would activate NEW_KEY content while injecting
  OLD_KEY when a skill's declared secrets change in that window. The
  'one scan per step' bonus that motivated sharing the snapshot is
  withdrawn — consistency wins. The two secret sources resolve
  independently: a transient fresh-load failure zeroes only the slash
  binding; entries keep binding from their own snapshot.
- Same-skill dominance: when a skill is both the slash source (fresh) and
  a persisted entry (snapshot), only the activation-era declaration binds
  — unioning both eras injected a secret the skill no longer declares.
  Aligns with the tool-policy rule (slash activation dominates for the
  run); different-skill entries still stack additively.
- The rendered 'Active skills' reminder (DurableContextMiddleware) no
  longer advertises skills the provider denies — surface consistency with
  describe_skill. Publication/consumption split: the activation
  middleware publishes per-step path-keyed skill:activate decisions into
  the run context (async: reused from the prepass snapshot, zero extra
  I/O; sync: one scan + authorize() only when entries exist — passive
  steps stay scan-free), the renderer consumes them and filters its
  render copy (state untouched; unpublished = permissive, matching the
  other run-context carriers). New carrier key listed in
  REDACTED_CONTEXT_KEYS with a redaction regression.
- Wiring regressions on both chains for everything added above (user_id
  to the stamp gate, skill_authorization + token pairing + activation-
  before-durable ordering to the durable renderer): each mutation
  verified red (M4-M13 across this change set).
- Tests 54->65: slash post-activation lookup, source independence,
  dominance, sync-chain sync-API path, render filter (async/sync/
  disabled with load-count assertions), chain wiring x2, redaction
  listing. Plan notes record rounds 9-2..6 including the reference-point
  post-mortem and the audit axes exhausted.

* fix(authz): anchor same-skill dominance on the slash identity, not binding results

Review round 10 (willem-bd) plus a self-audit extension the new checklist
items caught in the first fix itself:

- R10 (P2): the same-skill exclusion was derived from successfully BOUND
  slash sources. When the prepass snapshots required-secrets [OLD_KEY]
  and the operator strips the declaration during the await window, the
  slash source binds nothing (required_secrets gate) — the exclusion set
  stayed empty and the stale entry view kept injecting OLD_KEY for a
  skill that now declares no secrets. The exclusion now derives from the
  AUTHENTICATED slash activation identity (run-context source path,
  owner-token checked): the identity's declared name resolves from the
  fresh registry first, then the entry snapshot, and joins the exclusion
  set regardless of binding success.
- Self-audit (value->gate->downstream on the fix itself): the declared
  name can also change mid-run (same path, foo->bar). A name-anchored
  exclusion then holds only the new name while the snapshot entry
  resolves the same path to the old one — both eras bound again. The
  PATH is the stable identity shared by the activation and every earlier
  read: entries at the slash identity's path are now excluded before
  registry resolution (exclude_paths), independent of what either
  version calls the skill; the name-based set remains for cross-path
  same-name shadowing.

Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the
OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run
rename (only the activation-era key binds). Mutations verified red:
identity-derived exclusion disabled (M14), path exclusion disabled
(M16).

* test(authz): pass explicit config to lead middleware wiring test

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-30 19:45:07 +08:00

732 lines
30 KiB
Python

"""Phase 3 model-level authorization tests.
Covers two enforcement layers:
- Gateway routes (``list_models``, ``get_model``) — request-scoped Principal,
mirrors Phase 2A's ``resolve_route_permissions``.
- Runtime model resolution (``_authorize_model_name``) — context-scoped
Principal with graceful fallback, mirrors Phase 1B's ``apply_tool_authorization``.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app.gateway.routers import models as models_router
from deerflow.authz.provider import AuthzDecision, AuthzReason
from deerflow.authz.rbac import RbacAuthorizationProvider
from deerflow.config.app_config import AppConfig
from deerflow.config.authorization_config import AuthorizationConfig
from deerflow.config.model_config import ModelConfig
from deerflow.config.sandbox_config import SandboxConfig
from deerflow.config.token_usage_config import TokenUsageConfig
# ── Helpers ────────────────────────────────────────────────────────────
def _user(**overrides):
values = {
"id": "user-123",
"system_role": "user",
"oauth_provider": "github",
"oauth_id": "oauth-456",
}
values.update(overrides)
return SimpleNamespace(**values)
def _make_app_config(model_names: list[str]) -> AppConfig:
"""Build a minimal AppConfig with the given model names."""
return AppConfig(
models=[ModelConfig(name=n, model=n, use="langchain_openai:ChatOpenAI") for n in model_names],
sandbox=SandboxConfig(use="deerflow.sandbox.local:LocalSandboxProvider"),
token_usage=TokenUsageConfig(enabled=False),
authorization=AuthorizationConfig(),
)
def _enable_authorization(monkeypatch, provider, *, fail_closed: bool = True, default_role: str = "user") -> None:
config = AuthorizationConfig(
enabled=True,
fail_closed=fail_closed,
default_role=default_role,
)
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", lambda c: provider)
def _make_models_app(app_config: AppConfig) -> FastAPI:
"""Build a FastAPI app with the models router and a pinned config."""
app = FastAPI()
app.include_router(models_router.router)
# Pin the config dependency so routes use our test AppConfig.
app.dependency_overrides[models_router.get_config] = lambda: app_config
return app
class _RecordingProvider:
"""Provider that records all requests and can deny/error specific targets."""
name = "recording"
def __init__(self, *, denied: set[str] | None = None, errors: set[str] | None = None) -> None:
self.denied = denied or set()
self.errors = errors or set()
self.authorize_requests: list = []
self.filter_requests: list = []
def authorize(self, request):
self.authorize_requests.append(request)
if request.target in self.errors:
raise RuntimeError(f"provider failed for {request.target}")
allowed = request.target not in self.denied
return AuthzDecision(
allow=allowed,
reasons=[AuthzReason(code="authz.allowed" if allowed else "authz.denied")],
)
async def aauthorize(self, request):
return self.authorize(request)
def filter_resources(self, principal, resource_type, candidates):
self.filter_requests.append((resource_type, list(candidates)))
if resource_type in self.errors:
raise RuntimeError(f"provider failed for {resource_type}")
return [c for c in candidates if c not in self.denied]
# ── list_models tests ──────────────────────────────────────────────────
def test_list_models_disabled_returns_all(monkeypatch):
"""When authorization is disabled, all models are visible."""
config = AuthorizationConfig(enabled=False)
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
cached = AsyncMock(side_effect=AssertionError("disabled must not resolve provider"))
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", cached)
app_config = _make_app_config(["gpt-4", "claude-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
assert names == ["gpt-4", "claude-3"]
cached.assert_not_called()
def test_list_models_anonymous_user_returns_all(monkeypatch):
"""Anonymous requests (user=None) are not filtered."""
provider = _RecordingProvider()
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=None),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
assert names == ["gpt-4", "claude-3"]
assert provider.filter_requests == []
def test_list_models_rbac_filters_by_allow(monkeypatch):
"""Role with allowlist sees only allowed models."""
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": ["gpt-4"]}}},
)
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
assert names == ["gpt-4"]
def test_list_models_rbac_filters_by_deny(monkeypatch):
"""Role with deny hides denied models."""
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": "*", "deny": ["claude-3"]}}},
)
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
assert names == ["gpt-4", "llama-3"]
def test_list_models_wildcard_returns_all(monkeypatch):
"""Role with allow: '*' sees all models."""
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": "*"}}},
)
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
assert names == ["gpt-4", "claude-3"]
@pytest.mark.parametrize(
("fail_closed", "expected_count"),
[(True, 0), (False, 3)],
)
def test_list_models_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_count):
"""Provider error → empty (fail-closed) or all (fail-open)."""
provider = _RecordingProvider(errors={"model"})
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
app_config = _make_app_config(["gpt-4", "claude-3", "llama-3"])
app_config.authorization.fail_closed = fail_closed
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models")
assert response.status_code == 200
assert len(response.json()["models"]) == expected_count
# ── get_model tests ────────────────────────────────────────────────────
def test_get_model_disabled_returns_model(monkeypatch):
"""When authorization is disabled, get_model works as before."""
config = AuthorizationConfig(enabled=False)
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
app_config = _make_app_config(["gpt-4"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/gpt-4")
assert response.status_code == 200
assert response.json()["name"] == "gpt-4"
def test_get_model_404_when_not_found(monkeypatch):
"""Non-existent model returns 404 regardless of authorization."""
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": "*"}}})
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/nonexistent")
assert response.status_code == 404
def test_get_model_denied_returns_403(monkeypatch):
"""Role denied model:use → 403 (not 404)."""
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": ["claude-3"]}}},
)
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/gpt-4")
assert response.status_code == 403
def test_get_model_allowed_returns_200(monkeypatch):
"""Role allowed model:use → 200."""
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
)
_enable_authorization(monkeypatch, provider)
app_config = _make_app_config(["gpt-4", "claude-3"])
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/gpt-4")
assert response.status_code == 200
assert response.json()["name"] == "gpt-4"
@pytest.mark.parametrize(
("fail_closed", "expected_status"),
[(True, 403), (False, 200)],
)
def test_get_model_provider_error_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
"""Provider error on model:use → 403 (fail-closed) or 200 (fail-open)."""
provider = _RecordingProvider(errors={"gpt-4"})
_enable_authorization(monkeypatch, provider, fail_closed=fail_closed)
app_config = _make_app_config(["gpt-4"])
app_config.authorization.fail_closed = fail_closed
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/gpt-4")
assert response.status_code == expected_status
@pytest.mark.parametrize(
("fail_closed", "expected_status"),
[(True, 403), (False, 200)],
)
def test_get_model_provider_unavailable_fail_closed_vs_open(monkeypatch, fail_closed, expected_status):
"""Provider *resolution* failure → 403 (fail-closed) or 200 (fail-open).
Distinct from ``test_get_model_provider_error_fail_closed_vs_open``: that
test exercises a provider that resolves but errors inside ``authorize``.
This one exercises ``_AuthorizationUnavailable`` (the provider cannot be
resolved at all, e.g. misconfigured class path) and pins the fail-open
path so the docstring's "provider resolution error yields 403 (fail-closed)
or allows the request (fail-open)" claim is backed by a test.
"""
config = AuthorizationConfig(
enabled=True,
fail_closed=fail_closed,
default_role="user",
)
monkeypatch.setattr("app.gateway.authz._get_route_authorization_config", lambda: config)
# Force provider resolution to raise → _AuthorizationUnavailable.
def _boom(_config):
raise RuntimeError("provider class path invalid")
monkeypatch.setattr("app.gateway.authz._get_cached_route_provider", _boom)
app_config = _make_app_config(["gpt-4"])
app_config.authorization.fail_closed = fail_closed
monkeypatch.setattr(
"app.gateway.routers.models.get_optional_user_from_request",
AsyncMock(return_value=_user()),
)
with TestClient(_make_models_app(app_config)) as client:
response = client.get("/api/models/gpt-4")
assert response.status_code == expected_status
# ── Runtime model resolution tests (_authorize_model_name) ─────────────
def _rbac_context(**overrides):
"""Build a minimal run context dict for build_principal_from_context."""
values = {
"user_id": "user-123",
"user_role": "user",
"oauth_provider": "github",
"oauth_id": "oauth-456",
"is_internal": False,
}
values.update(overrides)
return values
def _enable_runtime_authorization(monkeypatch, provider) -> AuthorizationConfig:
"""Patch resolve_authorization_provider in agent.py to return *provider*.
Returns an enabled AuthorizationConfig the caller assigns to app_config.
"""
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: provider,
)
return AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
def test_authorize_model_name_disabled_is_noop():
"""When authorization is disabled, model name is returned unchanged."""
from deerflow.agents.lead_agent.agent import _authorize_model_name
app_config = _make_app_config(["gpt-4", "claude-3"])
# AuthorizationConfig() defaults to enabled=False.
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result == "gpt-4"
def test_authorize_model_name_allowed_returns_same(monkeypatch):
"""Allowed model → returned unchanged."""
from deerflow.agents.lead_agent.agent import _authorize_model_name
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": ["gpt-4", "claude-3"]}}},
)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result == "gpt-4"
def test_authorize_model_name_denied_falls_back_gracefully(monkeypatch):
"""Denied model → falls back to first allowed model (RFC §9)."""
from deerflow.agents.lead_agent.agent import _authorize_model_name
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": ["claude-3"]}}},
)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result == "claude-3"
def test_authorize_model_name_all_denied_fail_closed_raises(monkeypatch):
"""All models denied + fail_closed → ValueError."""
from deerflow.agents.lead_agent.agent import _authorize_model_name
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": []}}},
)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = _enable_runtime_authorization(monkeypatch, provider)
with pytest.raises(ValueError, match="No models are authorized"):
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
def test_authorize_model_name_all_denied_fail_open_returns_original(monkeypatch):
"""All models denied + fail_open → returns original model name."""
from deerflow.agents.lead_agent.agent import _authorize_model_name
provider = RbacAuthorizationProvider(
roles={"user": {"models": {"allow": []}}},
)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(
enabled=True,
fail_closed=False,
default_role="user",
)
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: provider,
)
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result == "gpt-4"
def test_authorize_model_name_custom_provider_list_vs_use_divergence(monkeypatch):
"""Custom provider that allows 'list' but denies 'use' → model is denied.
Regression for willem-bd's forward-looking note: a custom provider that
distinguishes ``list`` from ``use`` must not let a model through the
runtime path just because ``filter_resources`` includes it. The runtime
path checks ``authorize("model", "use")`` first; only on deny does it
fall back to ``filter_resources`` to pick a replacement.
"""
class _ListButNotUseProvider:
"""Allows listing gpt-4 but denies using it."""
name = "list-not-use"
def authorize(self, request):
if request.resource == "model" and request.action == "use" and request.target == "gpt-4":
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
async def aauthorize(self, request):
return self.authorize(request)
def filter_resources(self, principal, resource_type, candidates):
# gpt-4 is "visible" (listable) but not "usable"
return list(candidates)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: _ListButNotUseProvider(),
)
# gpt-4 is listable but denied for use → falls back to claude-3
from deerflow.agents.lead_agent.agent import _authorize_model_name
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result != "gpt-4"
assert result == "claude-3"
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_closed(monkeypatch):
"""All visible models denied for use + fail_closed → ValueError.
Regression for willem-bd's edge-case note: when ``filter_resources``
returns only models that are themselves denied for ``use``, the fallback
must NOT silently reselect a denied model. With ``fail_closed=True`` it
must raise; with ``fail_closed=False`` it returns the original name.
"""
class _AllListNoneUseProvider:
"""Lists all models but denies use for every one of them."""
name = "all-list-none-use"
def authorize(self, request):
if request.resource == "model" and request.action == "use":
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
async def aauthorize(self, request):
return self.authorize(request)
def filter_resources(self, principal, resource_type, candidates):
return list(candidates) # all visible
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: _AllListNoneUseProvider(),
)
from deerflow.agents.lead_agent.agent import _authorize_model_name
# gpt-4 denied for use; fallback candidates also denied → ValueError
with pytest.raises(ValueError, match="No models are authorized"):
_authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
def test_authorize_model_name_custom_provider_no_usable_fallback_fail_open(monkeypatch):
"""All visible models denied for use + fail_open → returns original name."""
class _AllListNoneUseProvider:
name = "all-list-none-use"
def authorize(self, request):
if request.resource == "model" and request.action == "use":
return AuthzDecision(allow=False, reasons=[AuthzReason(code="authz.denied")])
return AuthzDecision(allow=True, reasons=[AuthzReason(code="authz.allowed")])
async def aauthorize(self, request):
return self.authorize(request)
def filter_resources(self, principal, resource_type, candidates):
return list(candidates)
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=False, default_role="user")
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: _AllListNoneUseProvider(),
)
from deerflow.agents.lead_agent.agent import _authorize_model_name
result = _authorize_model_name("gpt-4", context=_rbac_context(), app_config=app_config)
assert result == "gpt-4"
# ── DeerFlowClient._ensure_agent path ─────────────────────────────────
# Regression for willem-bd's Round 4 coverage observation: the embedded/library
# lead-agent construction path (``DeerFlowClient._ensure_agent``) must enforce
# ``model:use`` too, not just the Gateway runtime path (``_make_lead_agent``).
# Otherwise a consumer that enables ``authorization`` with role-scoped model
# policies gets tools filtered yet can still run a model the role is denied
# ``use`` for, diverging from the contract this PR establishes.
def test_client_ensure_agent_enforces_model_use_when_authorized(monkeypatch):
"""``_ensure_agent`` routes the resolved model through ``_authorize_model_name``.
Real-path test: we let the genuine ``_authorize_model_name`` run against a
real RBAC provider (only ``resolve_authorization_provider`` is patched, as
in the runtime tests above) so the full ``client → authz gate → RBAC →
fallback → create_chat_model`` chain is exercised — not just "the gate was
called". The provider allows only ``claude-3`` for ``use``, so the denied
``gpt-4`` must be swapped for ``claude-3`` before reaching the model factory.
"""
from langchain_core.runnables import RunnableConfig
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: provider,
)
captured_name = _stub_client_assembly(monkeypatch)
client = _bare_client(app_config)
config: RunnableConfig = {"configurable": {"model_name": "gpt-4", "user_id": "user-123", "user_role": "user"}}
client._ensure_agent(config)
# Denied ``gpt-4`` was swapped for the authorized fallback ``claude-3``.
assert captured_name["name"] == "claude-3"
assert client._effective_model_name == "claude-3"
def test_client_ensure_agent_resolves_none_default_before_authorization(monkeypatch):
"""A ``None`` model name is resolved to the default before the authz gate.
Guards the ``create_chat_model(name=None)`` semantic: when the caller omits
``model_name`` the implicit default (first configured model) must still pass
``model:use`` — otherwise the embedded path could run an unauthorized default.
"""
from langchain_core.runnables import RunnableConfig
app_config = _make_app_config(["gpt-4", "claude-3"])
app_config.authorization = AuthorizationConfig(enabled=True, fail_closed=True, default_role="user")
# Deny the default ``gpt-4``; the gate must fallback to ``claude-3``.
provider = RbacAuthorizationProvider(roles={"user": {"models": {"allow": ["claude-3"]}}})
monkeypatch.setattr(
"deerflow.agents.lead_agent.agent.resolve_authorization_provider",
lambda config: provider,
)
captured_name = _stub_client_assembly(monkeypatch)
client = _bare_client(app_config)
# No model_name supplied → defaults to ``gpt-4`` (first configured) → denied → fallback.
config: RunnableConfig = {"configurable": {"user_id": "user-123", "user_role": "user"}}
client._ensure_agent(config)
assert captured_name["name"] == "claude-3"
assert client._effective_model_name == "claude-3"
def test_client_ensure_agent_noop_when_authorization_disabled(monkeypatch):
"""When ``authorization.enabled`` is false, ``_ensure_agent`` leaves the model unchanged."""
from langchain_core.runnables import RunnableConfig
app_config = _make_app_config(["gpt-4"])
# AuthorizationConfig() defaults to enabled=False.
captured_name = _stub_client_assembly(monkeypatch)
client = _bare_client(app_config)
config: RunnableConfig = {"configurable": {"model_name": "gpt-4"}}
client._ensure_agent(config)
# Disabled → gate is a no-op: original name passed straight through.
assert captured_name["name"] == "gpt-4"
def _stub_client_assembly(monkeypatch) -> dict[str, str]:
"""Stub the heavy dependencies ``_ensure_agent`` pulls in after the authz gate.
Returns a dict the caller can inspect to see what ``create_chat_model`` got.
Everything here is downstream of the contract under test, so we replace it
with no-ops to keep the test focused on the ``_authorize_model_name`` call.
"""
captured: dict[str, str] = {}
monkeypatch.setattr(
"deerflow.client.create_chat_model",
lambda **kwargs: captured.__setitem__("name", kwargs.get("name")) or object(),
)
monkeypatch.setattr("deerflow.client.create_agent", lambda **kwargs: object())
monkeypatch.setattr("deerflow.client.build_middlewares", lambda *args, **kwargs: [])
monkeypatch.setattr("deerflow.client.DeerFlowClient._get_tools", staticmethod(lambda *, model_name, subagent_enabled, mcp_plugins=None: [])) # noqa: ARG005
monkeypatch.setattr("deerflow.client.get_enabled_skills_for_config", lambda app_config, **kw: []) # noqa: ARG005
monkeypatch.setattr(
"deerflow.client.build_skill_search_setup",
lambda skills, *, enabled, container_base_path, skill_authorization=None: SimpleNamespace(describe_skill_tool=None, skill_names=frozenset()), # noqa: ARG005
)
monkeypatch.setattr(
"deerflow.client.assemble_deferred_tools",
lambda tools, *, enabled: ([], SimpleNamespace(deferred_names=frozenset())), # noqa: ARG005
)
monkeypatch.setattr("deerflow.client.build_mcp_routing_middleware", lambda *args, **kwargs: None) # noqa: ARG005
monkeypatch.setattr("deerflow.client.get_mcp_routing_hints_prompt_section", lambda *args, **kwargs: "") # noqa: ARG005
monkeypatch.setattr("deerflow.client.apply_prompt_template", lambda **kwargs: "") # noqa: ARG005
monkeypatch.setattr("deerflow.client.get_thread_state_schema", lambda *args, **kwargs: object()) # noqa: ARG005
monkeypatch.setattr("deerflow.client.normalize_middleware_state_schemas", lambda schemas, mode, freq: []) # noqa: ARG005
monkeypatch.setattr("deerflow.client.get_effective_user_id", lambda: "user-123")
# ``apply_tool_authorization`` (called with the empty tool list above) still
# resolves a provider via ``tool_filter.resolve_authorization_provider``; route
# it at an allow-all RBAC provider so the empty list stays empty. The skill
# filter (added in Phase 3 Skills PR) resolves via ``skill_filter`` namespace.
monkeypatch.setattr(
"deerflow.authz.tool_filter.resolve_authorization_provider",
lambda config: RbacAuthorizationProvider(roles={"user": {"tools": {"allow": "*"}}}),
)
monkeypatch.setattr(
"deerflow.authz.skill_filter.resolve_authorization_provider",
lambda config: RbacAuthorizationProvider(roles={"user": {"skills": {"allow": "*"}}}),
)
return captured
def _bare_client(app_config):
"""Construct a ``DeerFlowClient`` without running ``__init__``."""
from deerflow.client import DeerFlowClient
client = DeerFlowClient.__new__(DeerFlowClient)
client._app_config = app_config
client._agent_name = "default"
client._available_skills = None
client._checkpoint_channel_mode = "full"
client._checkpoint_snapshot_frequency = None
client._middlewares = []
client._agent = None
client._agent_config_key = None
# Non-None so ``_ensure_agent`` skips the real (postgres/sqlite) checkpointer
# resolution — the value is never used because ``create_agent`` is stubbed.
client._checkpointer = object()
return client