mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-15 00:19:14 +00:00
* feat(auth): add personal access tokens for programmatic API access (#4849) Backend-first implementation of the PAT contract from #4849: show-once dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT, is_internal=false), digest-only storage (migration 0017), strict credential precedence (invalid Bearer is a 401, never cookie fallback), CSRF double-submit skipped only for Bearer requests while auth-endpoint origin checks still run, scopes intersecting the authz route permissions, session-auth-only PAT management and password changes, and throttled best-effort last_used_at stamps. * fix(auth): harden PAT scope boundary and schema parity from adversarial review Independent review of the initial draft found: (1) scopes only constrained the threads/runs permission axis while admin routes treated a PAT as its (possibly admin) owner — is_admin_user now rejects PAT callers outright since no scope grants admin capability; (2) the model declared a column UNIQUE constraint while migration 0017 created a named unique index, so downgrade failed on create_all-bootstrapped DBs — both now use the named unique index; (3) auth-disabled mode is an operator override and now stays ahead of the Bearer check so a stray Authorization header cannot 401 an E2E sandbox; plus wiring the previously-unused constants, bounding the last_used_at stamp cache, and four new tests (middleware-level expiry, expires_in_days, admin-capability rejection with session control, and the auth-disabled precedence). * docs(api): document personal access tokens for programmatic API access * fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression) P1-1: scope intersection only constrains @require_permission routes, so undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark credential switching, channel config) accepted a PAT holding a single read scope. AuthMiddleware now enforces a default-deny route policy in auth/pat.py: PAT requests are admitted only to the thread/run lifecycle routes the v1 scopes govern; everything else answers 403 regardless of scopes. Session-cookie callers are unaffected. P1-2: the extension principal resolver projected is_admin/roles from the raw system_role, so an admin-owned PAT passed deerflow_extension_api.require_admin on contributed routes despite the documented no-admin guarantee. The projection is now PAT-aware and suppresses every admin signal for PAT callers, mirroring deps.is_admin_user. Both fixes carry regression tests (route outside policy 403 + session control; production resolver admin suppression), and API.md documents the default-deny boundary. * fix(auth): enforce PAT scopes on stateless run entry and harden decorator Follow-up hardening from an independent audit of the P1 fixes: - POST /api/runs/stream and /api/runs/wait were the only allowlisted run entrypoints without @require_permission, so a threads:read-only PAT could still start runs (same bug class as P1-1, now closed): both now carry @require_permission("runs", "create"). POST /api/threads and POST /api/threads/search gain threads:write / threads:read for the same reason. Authorization-disabled deployments see no change (the permission set resolves to all permissions). - require_permission now binds the wrapped signature to locate a positionally-passed request before injecting the test stub, fixing 'got multiple values for argument' on direct positional unit-test calls. - API.md: the intro PAT example used GET /api/models, which the new default-deny policy 403s — replaced with GET /api/threads; the default-deny route list now spells out method sets. Regression test: threads:read-only PAT is 403 on the decorated stateless entry while a runs:create PAT passes. * fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example) - CSRFMiddleware treats an explicitly empty Authorization header as present (is None), so an invalid credential always reaches AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by method/CSRF state. Regression: empty-header request dies at auth. - PATCreateRequest strips the name and rejects whitespace-only values before token generation; created names are stored trimmed. - API.md intro PAT example now uses the implemented POST /api/threads/search endpoint (GET /api/threads does not exist). - AGENTS.md trimmed back under the guidance soft budget after the upstream merge. * fix(auth): tighten PAT route policy to implemented methods only The allowlist admitted GET /api/threads, a method no router implements. Pre-authorizing a dead method weakens the default-deny boundary: a future GET collection route added without a permission decorator would become PAT-reachable without an explicit policy change. Restrict the rule to POST, fix the stale GET description in API.md's PAT constraints, and document the default-deny boundary accurately in the gateway AGENTS.md guidance (only the threads/runs allowlist is PAT-reachable; every other authenticated route 403s PAT callers). Audited every remaining rule against the mounted routers: all other method+path entries map to real routes. Regression: test_pat_policy_does_not_pre_authorize_unimplemented_methods. * test(auth): guarantee the negative digest test mutates the token token[:-1] + "X" is identical to the original whenever the generated token already ends in X (1/62), making the negative digest assertion fail intermittently. Choose the replacement character based on the existing tail so the mutated token always differs. * fix(auth): require runs:cancel for cancel-then-stream requests stream_existing_run is gated at runs:read so action-less stream joins work with read-only credentials, but its ?action=interrupt|rollback branch cancels the run — a separate permission. A runs:read-only PAT passed both the PAT route policy and the route decorator and could interrupt or roll back an active run, bypassing the runs:cancel scope. Decorators cannot express query-parameter-conditional permissions, so the check lives in require_cancel_permission_when_action(), applied at the top of the handler. Regression drives the real helper through the production middleware: runs:read-only PAT + action is 403, the same token joins action-less, runs:read+cancel passes, session control unaffected. * docs(changelog): add the PAT feature entry * docs(readme): add personal access tokens section Repo documentation-update policy requires user-facing features to update README.md in the same changeset; the PAT feature previously touched only backend/docs/API.md and the gateway AGENTS.md. * fix(auth): require runs:cancel for mutating multitask strategies All five run-creation entrypoints were gated only by runs:create, but RunCreateRequest.multitask_strategy accepts interrupt/rollback and start_run forwards it to create_or_reject, which terminates an already-active run. A runs:create-only PAT could therefore kill an existing run through a create request, bypassing runs:cancel. Decorators cannot express body-parameter-conditional permissions, and per-route checks leave the same hole for the next entrypoint, so the gate lives in start_run itself — the single choke point every run-creation path (HTTP routes and internal launchers) flows through. Regenerate launches pass multitask_strategy="reject" and are unaffected; requests without a stamped auth context (internal/test compositions) skip the gate. The check is the shared authz.require_cancel_permission_if primitive; require_cancel_permission_when_action now delegates to it, so every request dimension that carries cancel capability (query action, body strategy) flows through one gate. Regression drives the real middleware stack: runs:create-only PAT + interrupt/rollback is 403 with the exact detail, reject (explicit and default) stays available, runs:create+cancel passes, session control unaffected; a source anchor pins the gate inside start_run. * fix(runs): keep observer joins from applying creator cancel-on-disconnect sse_consumer's finally block applied the record's on_disconnect=cancel policy on ANY consumer's disconnect. The join surfaces (GET /join and the action-less GET/POST stream join) feed it the existing RunRecord, so anyone with thread read access — including a runs:read-only PAT — could cancel a locally-owned running run simply by closing the SSE connection, without runs:cancel. The policy expresses the creator's intent for their own connection; an observer's disconnect must never be read as that intent. sse_consumer gains apply_on_disconnect (default True). The two join surfaces pass False; the creating endpoints (thread-scoped and stateless create-and-stream) keep the creator semantics unchanged. wait_for_run_completion needs no change: its callers are creator-side or post-explicit-cancel paths only. Regression exercises a real generator close — the same machinery Starlette drives on client disconnect — against the production sse_consumer: creator stream disconnect cancels, observer join disconnect does not; a wiring anchor pins both join call sites and the creator defaults. API.md documents the cancel-capability constraint (this fix plus the action/strategy gates) in PAT Constraints. * test(auth): pin the multitask gate behaviorally; state wait invariant Independent adversarial review of the round-5 fixes found the P1-a regression only mirror-pinned: the source anchor could be satisfied by a comment, and deleting the gate from start_run would not fail the suite. This drives the production start_run directly — a create-only auth context gets 403 with the exact detail for interrupt, and a reject request with no cancel permission at all proceeds past the gate (never a permission 403). Also documents wait_for_run_completion's creator-side invariant (every caller is the creating endpoint or post-explicit-cancel) so a future observer wiring thinks twice before reusing it — the one-caller- away variant of the observer-disconnect P1. * docs(changelog): correct the PAT entry's digest and route-policy description The entry said HMAC digests (the implementation stores SHA-256 digests, as documented in API.md and pinned by the repository tests) and claimed the route policy admits 'implemented stateless endpoints' (it admits the thread/run lifecycle routes, narrowing further by scopes). Also notes the cancel-capability gate now covering action and multitask strategies. * fix(auth): enumerate the PAT runs route policy per implemented subroute The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it pre-authorized every current and future subroute under /runs, including methods the router never implemented (e.g. GET /runs/stream), which is the same latent default-deny weakening the threads collection rule was tightened for: a future route added under /runs would become PAT-reachable without an explicit policy change. The wildcard is replaced with six segment-precise rules covering exactly the 14 implemented method+path combinations; the {run_id} slot necessarily matches any single segment, so the POST-only collection names (stream, wait, regenerate, edit-regenerate) are excluded from the GET run-id rule via negative lookahead — no dead method stays pre-authorized. Behavior for implemented routes is unchanged. test_pat_runs_policy_admits_exactly_the_mounted_routes derives the expected set from the mounted thread_runs router instead of a hand-maintained list: every implemented GET/POST route under /runs must be admitted, routes in this router outside the subtree stay denied, and representative unimplemented neighbors are denied — so adding a route under /runs now fails CI until it is explicitly allowlisted, and a removed route leaves a dead rule visible. API.md's PAT constraints list the enumerated routes and drops a feedback mention that belonged to the stateless /api/runs axis. * docs(migration): add the 0017 renumbering coordination note to 0017 The PR's migration-coordination comment states each migration file carries the note; the file did not. Adds it: numbering was generated against main head 0016 alongside #5078 and #4843; whoever merges first keeps the slot, the others renumber on rebase (revision/down_revision plus the bootstrap head assertions). * fix(auth): pad base62 tokens to a fixed 43-char width int.from_bytes discards leading zero bytes, so the unpadded encoder returned a variable-length body — empty for all-zero input, and shorter than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving test_generate_pat_token_format probabilistically flaky and the token body without stable width (review round 6, P3). _base62 now left-pads with "0" to _base62_width(len(data)) — the exact integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The format test asserts the exact fixed width instead of a probabilistic floor, and a new unit test pins the all-zero, leading-zero-byte, and max-value edges deterministically.
283 lines
11 KiB
Python
283 lines
11 KiB
Python
"""CSRF protection middleware for FastAPI.
|
|
|
|
Per RFC-001:
|
|
State-changing operations require CSRF protection.
|
|
"""
|
|
|
|
import os
|
|
import secrets
|
|
from collections.abc import Awaitable, Callable
|
|
from urllib.parse import urlsplit
|
|
|
|
from fastapi import Request, Response
|
|
from starlette.middleware.base import BaseHTTPMiddleware
|
|
from starlette.responses import JSONResponse
|
|
from starlette.types import ASGIApp
|
|
|
|
from app.gateway.auth.config import get_auth_config
|
|
from app.gateway.auth.session_cookie_state import SESSION_COOKIE_ISSUED_STATE_ATTR, SESSION_COOKIE_MAX_AGE_STATE_ATTR, SESSION_COOKIE_SECURE_STATE_ATTR, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR
|
|
from app.gateway.auth_disabled import is_auth_disabled
|
|
from app.gateway.request_path import get_request_route_path
|
|
|
|
CSRF_COOKIE_NAME = "csrf_token"
|
|
CSRF_HEADER_NAME = "X-CSRF-Token"
|
|
CSRF_TOKEN_LENGTH = 64 # bytes
|
|
_CSRF_STATE_CHANGING_METHODS: frozenset[str] = frozenset({"POST", "PUT", "DELETE", "PATCH"})
|
|
_CSRF_EXEMPT_EXACT_PATHS: frozenset[str] = frozenset({"/api/v1/auth/me"})
|
|
|
|
|
|
def is_secure_request(request: Request) -> bool:
|
|
"""Detect whether the original client request was made over HTTPS."""
|
|
return _request_scheme(request) == "https"
|
|
|
|
|
|
def generate_csrf_token() -> str:
|
|
"""Generate a secure random CSRF token."""
|
|
return secrets.token_urlsafe(CSRF_TOKEN_LENGTH)
|
|
|
|
|
|
def should_check_csrf(request: Request) -> bool:
|
|
"""Determine if a request needs CSRF validation.
|
|
|
|
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
|
|
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
|
|
"""
|
|
if request.method not in _CSRF_STATE_CHANGING_METHODS:
|
|
return False
|
|
|
|
if is_auth_disabled():
|
|
return False
|
|
|
|
route_path = get_request_route_path(request)
|
|
path = route_path.rstrip("/")
|
|
# Exempt host-owned endpoints that implement their own request posture.
|
|
if path in _CSRF_EXEMPT_EXACT_PATHS:
|
|
return False
|
|
# Inbound webhooks authenticate themselves via provider-specific signatures
|
|
# (e.g. GitHub's X-Hub-Signature-256), not the CSRF double-submit cookie.
|
|
if route_path.startswith("/api/webhooks/"):
|
|
return False
|
|
return True
|
|
|
|
|
|
_AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
|
|
{
|
|
"/api/v1/auth/login/local",
|
|
"/api/v1/auth/logout",
|
|
"/api/v1/auth/register",
|
|
"/api/v1/auth/initialize",
|
|
}
|
|
)
|
|
|
|
|
|
def is_auth_endpoint(request: Request) -> bool:
|
|
"""Check if the request is to an auth endpoint.
|
|
|
|
Auth endpoints don't need CSRF validation on first call (no token).
|
|
"""
|
|
return get_request_route_path(request).rstrip("/") in _AUTH_EXEMPT_PATHS
|
|
|
|
|
|
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:
|
|
"""Return normalized host[:port], omitting default ports."""
|
|
host = hostname.lower()
|
|
if ":" in host and not host.startswith("["):
|
|
host = f"[{host}]"
|
|
|
|
if port is None or (scheme == "http" and port == 80) or (scheme == "https" and port == 443):
|
|
return host
|
|
return f"{host}:{port}"
|
|
|
|
|
|
def _normalize_origin(origin: str) -> str | None:
|
|
"""Return a normalized scheme://host[:port] origin, or None for invalid input."""
|
|
try:
|
|
parsed = urlsplit(origin.strip())
|
|
port = parsed.port
|
|
except ValueError:
|
|
return None
|
|
|
|
scheme = parsed.scheme.lower()
|
|
if scheme not in {"http", "https"} or not parsed.hostname:
|
|
return None
|
|
|
|
# Browser Origin is only scheme/host/port. Reject URL-shaped or credentialed values.
|
|
if parsed.username or parsed.password or parsed.path or parsed.query or parsed.fragment:
|
|
return None
|
|
|
|
return f"{scheme}://{_host_with_optional_port(parsed.hostname, port, scheme)}"
|
|
|
|
|
|
def _configured_cors_origins() -> set[str]:
|
|
"""Return explicit configured browser origins that may call auth routes."""
|
|
origins = set()
|
|
for raw_origin in os.environ.get("GATEWAY_CORS_ORIGINS", "").split(","):
|
|
origin = raw_origin.strip()
|
|
if not origin or origin == "*":
|
|
continue
|
|
normalized = _normalize_origin(origin)
|
|
if normalized:
|
|
origins.add(normalized)
|
|
return origins
|
|
|
|
|
|
def get_configured_cors_origins() -> set[str]:
|
|
"""Return normalized explicit browser origins from GATEWAY_CORS_ORIGINS."""
|
|
return _configured_cors_origins()
|
|
|
|
|
|
# Response headers a split-origin browser client must be able to read. Only the
|
|
# CORS-safelisted set is visible to JS by default, and the created run's id
|
|
# travels in `Content-Location` — the LangGraph SDK resolves run metadata from
|
|
# it, so withholding it leaves such a client unable to learn its own run id.
|
|
CORS_EXPOSED_HEADERS: tuple[str, ...] = ("Content-Location",)
|
|
|
|
|
|
def _first_header_value(value: str | None) -> str | None:
|
|
"""Return the first value from a comma-separated proxy header."""
|
|
if not value:
|
|
return None
|
|
first = value.split(",", 1)[0].strip()
|
|
return first or None
|
|
|
|
|
|
def _forwarded_param(request: Request, name: str) -> str | None:
|
|
"""Extract a parameter from the first RFC 7239 Forwarded header entry."""
|
|
forwarded = _first_header_value(request.headers.get("forwarded"))
|
|
if not forwarded:
|
|
return None
|
|
|
|
for part in forwarded.split(";"):
|
|
key, sep, value = part.strip().partition("=")
|
|
if sep and key.lower() == name:
|
|
return value.strip().strip('"') or None
|
|
return None
|
|
|
|
|
|
def _request_scheme(request: Request) -> str:
|
|
"""Resolve the original request scheme from trusted proxy headers."""
|
|
scheme = _forwarded_param(request, "proto") or _first_header_value(request.headers.get("x-forwarded-proto")) or request.url.scheme
|
|
return scheme.lower()
|
|
|
|
|
|
def _request_origin(request: Request) -> str | None:
|
|
"""Build the origin for the URL the browser is targeting."""
|
|
scheme = _request_scheme(request)
|
|
host = _forwarded_param(request, "host") or _first_header_value(request.headers.get("x-forwarded-host")) or request.headers.get("host") or request.url.netloc
|
|
|
|
forwarded_port = _first_header_value(request.headers.get("x-forwarded-port"))
|
|
if forwarded_port and ":" not in host.rsplit("]", 1)[-1]:
|
|
host = f"{host}:{forwarded_port}"
|
|
|
|
return _normalize_origin(f"{scheme}://{host}")
|
|
|
|
|
|
def is_allowed_auth_origin(request: Request) -> bool:
|
|
"""Allow auth POSTs only from the same origin or explicit configured origins.
|
|
|
|
Login/register/initialize are exempt from the double-submit token because
|
|
first-time browser clients do not have a CSRF token yet. They still create
|
|
a session cookie, so browser requests with a hostile Origin header must be
|
|
rejected to prevent login CSRF / session fixation. Requests without Origin
|
|
are allowed for non-browser clients such as curl and mobile integrations.
|
|
"""
|
|
origin = request.headers.get("origin")
|
|
if not origin:
|
|
return True
|
|
|
|
normalized_origin = _normalize_origin(origin)
|
|
if normalized_origin is None:
|
|
return False
|
|
|
|
request_origin = _request_origin(request)
|
|
return normalized_origin in _configured_cors_origins() or (request_origin is not None and normalized_origin == request_origin)
|
|
|
|
|
|
def auth_csrf_cookie_settings(request: Request) -> tuple[bool, int | None]:
|
|
"""Return ``(secure, max_age)`` for auth-created CSRF cookies."""
|
|
session_cookie_issued = getattr(request.state, SESSION_COOKIE_ISSUED_STATE_ATTR, False)
|
|
if session_cookie_issued:
|
|
return (
|
|
bool(getattr(request.state, SESSION_COOKIE_SECURE_STATE_ATTR, is_secure_request(request))),
|
|
getattr(request.state, SESSION_COOKIE_MAX_AGE_STATE_ATTR, None),
|
|
)
|
|
|
|
secure = is_secure_request(request)
|
|
max_age = get_auth_config().token_expiry_days * 24 * 3600 if secure else None
|
|
return secure, max_age
|
|
|
|
|
|
class CSRFMiddleware(BaseHTTPMiddleware):
|
|
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
|
|
|
|
def __init__(self, app: ASGIApp) -> None:
|
|
super().__init__(app)
|
|
|
|
async def dispatch(self, request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
|
|
_is_auth = is_auth_endpoint(request)
|
|
|
|
if should_check_csrf(request) and _is_auth and not is_allowed_auth_origin(request):
|
|
return JSONResponse(
|
|
status_code=403,
|
|
content={"detail": "Cross-site auth request denied."},
|
|
)
|
|
|
|
if should_check_csrf(request) and not _is_auth and request.headers.get("authorization") is None:
|
|
# Bearer-authenticated requests (PAT, #4849) are exempt from the
|
|
# cookie double-submit check only — the cross-site origin check on
|
|
# auth endpoints above still runs for every request. Safety rests
|
|
# on AuthMiddleware's strict Bearer precedence: an invalid Bearer
|
|
# header is a 401 there, so a cross-site attacker cannot ride a
|
|
# victim's cookie by padding a garbage Authorization header, and a
|
|
# cross-site request carrying a custom Authorization header at all
|
|
# requires a CORS preflight the attacker cannot obtain.
|
|
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
|
|
header_token = request.headers.get(CSRF_HEADER_NAME)
|
|
|
|
if not cookie_token or not header_token:
|
|
return JSONResponse(
|
|
status_code=403,
|
|
content={"detail": "CSRF token missing. Include X-CSRF-Token header."},
|
|
)
|
|
|
|
if not secrets.compare_digest(cookie_token, header_token):
|
|
return JSONResponse(
|
|
status_code=403,
|
|
content={"detail": "CSRF token mismatch."},
|
|
)
|
|
|
|
response = await call_next(request)
|
|
|
|
# For auth endpoints that set up session, also set CSRF cookie.
|
|
# Session-creating handlers may stamp the final access-token max_age on
|
|
# request.state; mirroring it here keeps the double-submit cookie pair
|
|
# from diverging across HTTPS, localhost, and sandbox deployments.
|
|
if _is_auth and request.method == "POST" and not getattr(request.state, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR, False):
|
|
# Generate a new CSRF token for the session
|
|
csrf_token = generate_csrf_token()
|
|
secure, max_age = auth_csrf_cookie_settings(request)
|
|
response.set_cookie(
|
|
key=CSRF_COOKIE_NAME,
|
|
value=csrf_token,
|
|
httponly=False, # Must be JS-readable for Double Submit Cookie pattern
|
|
secure=secure,
|
|
samesite="strict",
|
|
# Match the access_token cookie's lifetime (auth.py::_set_session_cookie)
|
|
# so the double-submit pair never diverges. A session-only csrf_token is
|
|
# evicted when iOS Safari terminates a home-screen PWA while the persistent
|
|
# access_token survives — leaving the user "logged in" but unable to make
|
|
# any state-changing request (403 "CSRF token missing").
|
|
max_age=max_age,
|
|
)
|
|
|
|
return response
|
|
|
|
|
|
def get_csrf_token(request: Request) -> str | None:
|
|
"""Get the CSRF token from the current request's cookies.
|
|
|
|
This is useful for server-side rendering where you need to embed
|
|
token in forms or headers.
|
|
"""
|
|
return request.cookies.get(CSRF_COOKIE_NAME)
|