deer-flow/backend/app/gateway/csrf_middleware.py
Sunshine bf740ffa90
feat(auth): add personal access tokens for programmatic API access (#5041)
* feat(auth): add personal access tokens for programmatic API access (#4849)

Backend-first implementation of the PAT contract from #4849: show-once
dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT,
is_internal=false), digest-only storage (migration 0017), strict
credential precedence (invalid Bearer is a 401, never cookie fallback),
CSRF double-submit skipped only for Bearer requests while
auth-endpoint origin checks still run, scopes intersecting the authz
route permissions, session-auth-only PAT management and password
changes, and throttled best-effort last_used_at stamps.

* fix(auth): harden PAT scope boundary and schema parity from adversarial review

Independent review of the initial draft found: (1) scopes only constrained
the threads/runs permission axis while admin routes treated a PAT as its
(possibly admin) owner — is_admin_user now rejects PAT callers outright
since no scope grants admin capability; (2) the model declared a column
UNIQUE constraint while migration 0017 created a named unique index, so
downgrade failed on create_all-bootstrapped DBs — both now use the named
unique index; (3) auth-disabled mode is an operator override and now stays
ahead of the Bearer check so a stray Authorization header cannot 401 an
E2E sandbox; plus wiring the previously-unused constants, bounding the
last_used_at stamp cache, and four new tests (middleware-level expiry,
expires_in_days, admin-capability rejection with session control, and the
auth-disabled precedence).

* docs(api): document personal access tokens for programmatic API access

* fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression)

P1-1: scope intersection only constrains @require_permission routes, so
undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark
credential switching, channel config) accepted a PAT holding a single read
scope. AuthMiddleware now enforces a default-deny route policy in
auth/pat.py: PAT requests are admitted only to the thread/run lifecycle
routes the v1 scopes govern; everything else answers 403 regardless of
scopes. Session-cookie callers are unaffected.

P1-2: the extension principal resolver projected is_admin/roles from the
raw system_role, so an admin-owned PAT passed
deerflow_extension_api.require_admin on contributed routes despite the
documented no-admin guarantee. The projection is now PAT-aware and
suppresses every admin signal for PAT callers, mirroring
deps.is_admin_user.

Both fixes carry regression tests (route outside policy 403 + session
control; production resolver admin suppression), and API.md documents the
default-deny boundary.

* fix(auth): enforce PAT scopes on stateless run entry and harden decorator

Follow-up hardening from an independent audit of the P1 fixes:

- POST /api/runs/stream and /api/runs/wait were the only allowlisted run
  entrypoints without @require_permission, so a threads:read-only PAT
  could still start runs (same bug class as P1-1, now closed): both now
  carry @require_permission("runs", "create"). POST /api/threads and
  POST /api/threads/search gain threads:write / threads:read for the
  same reason. Authorization-disabled deployments see no change (the
  permission set resolves to all permissions).
- require_permission now binds the wrapped signature to locate a
  positionally-passed request before injecting the test stub, fixing
  'got multiple values for argument' on direct positional unit-test
  calls.
- API.md: the intro PAT example used GET /api/models, which the new
  default-deny policy 403s — replaced with GET /api/threads; the
  default-deny route list now spells out method sets.

Regression test: threads:read-only PAT is 403 on the decorated stateless
entry while a runs:create PAT passes.

* fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example)

- CSRFMiddleware treats an explicitly empty Authorization header as
  present (is None), so an invalid credential always reaches
  AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by
  method/CSRF state. Regression: empty-header request dies at auth.
- PATCreateRequest strips the name and rejects whitespace-only values
  before token generation; created names are stored trimmed.
- API.md intro PAT example now uses the implemented
  POST /api/threads/search endpoint (GET /api/threads does not exist).
- AGENTS.md trimmed back under the guidance soft budget after the
  upstream merge.

* fix(auth): tighten PAT route policy to implemented methods only

The allowlist admitted GET /api/threads, a method no router implements.
Pre-authorizing a dead method weakens the default-deny boundary: a
future GET collection route added without a permission decorator would
become PAT-reachable without an explicit policy change. Restrict the
rule to POST, fix the stale GET description in API.md's PAT
constraints, and document the default-deny boundary accurately in the
gateway AGENTS.md guidance (only the threads/runs allowlist is
PAT-reachable; every other authenticated route 403s PAT callers).

Audited every remaining rule against the mounted routers: all other
method+path entries map to real routes. Regression:
test_pat_policy_does_not_pre_authorize_unimplemented_methods.

* test(auth): guarantee the negative digest test mutates the token

token[:-1] + "X" is identical to the original whenever the generated
token already ends in X (1/62), making the negative digest assertion
fail intermittently. Choose the replacement character based on the
existing tail so the mutated token always differs.

* fix(auth): require runs:cancel for cancel-then-stream requests

stream_existing_run is gated at runs:read so action-less stream joins
work with read-only credentials, but its ?action=interrupt|rollback
branch cancels the run — a separate permission. A runs:read-only PAT
passed both the PAT route policy and the route decorator and could
interrupt or roll back an active run, bypassing the runs:cancel scope.

Decorators cannot express query-parameter-conditional permissions, so
the check lives in require_cancel_permission_when_action(), applied at
the top of the handler. Regression drives the real helper through the
production middleware: runs:read-only PAT + action is 403, the same
token joins action-less, runs:read+cancel passes, session control
unaffected.

* docs(changelog): add the PAT feature entry

* docs(readme): add personal access tokens section

Repo documentation-update policy requires user-facing features to
update README.md in the same changeset; the PAT feature previously
touched only backend/docs/API.md and the gateway AGENTS.md.

* fix(auth): require runs:cancel for mutating multitask strategies

All five run-creation entrypoints were gated only by runs:create, but
RunCreateRequest.multitask_strategy accepts interrupt/rollback and
start_run forwards it to create_or_reject, which terminates an
already-active run. A runs:create-only PAT could therefore kill an
existing run through a create request, bypassing runs:cancel.

Decorators cannot express body-parameter-conditional permissions, and
per-route checks leave the same hole for the next entrypoint, so the
gate lives in start_run itself — the single choke point every
run-creation path (HTTP routes and internal launchers) flows through.
Regenerate launches pass multitask_strategy="reject" and are
unaffected; requests without a stamped auth context (internal/test
compositions) skip the gate.

The check is the shared authz.require_cancel_permission_if primitive;
require_cancel_permission_when_action now delegates to it, so every
request dimension that carries cancel capability (query action, body
strategy) flows through one gate.

Regression drives the real middleware stack: runs:create-only PAT +
interrupt/rollback is 403 with the exact detail, reject (explicit and
default) stays available, runs:create+cancel passes, session control
unaffected; a source anchor pins the gate inside start_run.

* fix(runs): keep observer joins from applying creator cancel-on-disconnect

sse_consumer's finally block applied the record's on_disconnect=cancel
policy on ANY consumer's disconnect. The join surfaces (GET /join and
the action-less GET/POST stream join) feed it the existing RunRecord,
so anyone with thread read access — including a runs:read-only PAT —
could cancel a locally-owned running run simply by closing the SSE
connection, without runs:cancel. The policy expresses the creator's
intent for their own connection; an observer's disconnect must never
be read as that intent.

sse_consumer gains apply_on_disconnect (default True). The two join
surfaces pass False; the creating endpoints (thread-scoped and
stateless create-and-stream) keep the creator semantics unchanged.
wait_for_run_completion needs no change: its callers are creator-side
or post-explicit-cancel paths only.

Regression exercises a real generator close — the same machinery
Starlette drives on client disconnect — against the production
sse_consumer: creator stream disconnect cancels, observer join
disconnect does not; a wiring anchor pins both join call sites and the
creator defaults. API.md documents the cancel-capability constraint
(this fix plus the action/strategy gates) in PAT Constraints.

* test(auth): pin the multitask gate behaviorally; state wait invariant

Independent adversarial review of the round-5 fixes found the P1-a
regression only mirror-pinned: the source anchor could be satisfied by
a comment, and deleting the gate from start_run would not fail the
suite. This drives the production start_run directly — a create-only
auth context gets 403 with the exact detail for interrupt, and a
reject request with no cancel permission at all proceeds past the gate
(never a permission 403).

Also documents wait_for_run_completion's creator-side invariant
(every caller is the creating endpoint or post-explicit-cancel) so a
future observer wiring thinks twice before reusing it — the one-caller-
away variant of the observer-disconnect P1.

* docs(changelog): correct the PAT entry's digest and route-policy description

The entry said HMAC digests (the implementation stores SHA-256 digests,
as documented in API.md and pinned by the repository tests) and claimed
the route policy admits 'implemented stateless endpoints' (it admits
the thread/run lifecycle routes, narrowing further by scopes). Also
notes the cancel-capability gate now covering action and multitask
strategies.

* fix(auth): enumerate the PAT runs route policy per implemented subroute

The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it
pre-authorized every current and future subroute under /runs, including
methods the router never implemented (e.g. GET /runs/stream), which is
the same latent default-deny weakening the threads collection rule was
tightened for: a future route added under /runs would become
PAT-reachable without an explicit policy change.

The wildcard is replaced with six segment-precise rules covering exactly
the 14 implemented method+path combinations; the {run_id} slot
necessarily matches any single segment, so the POST-only collection
names (stream, wait, regenerate, edit-regenerate) are excluded from the
GET run-id rule via negative lookahead — no dead method stays
pre-authorized. Behavior for implemented routes is unchanged.

test_pat_runs_policy_admits_exactly_the_mounted_routes derives the
expected set from the mounted thread_runs router instead of a
hand-maintained list: every implemented GET/POST route under /runs must
be admitted, routes in this router outside the subtree stay denied, and
representative unimplemented neighbors are denied — so adding a route
under /runs now fails CI until it is explicitly allowlisted, and a
removed route leaves a dead rule visible. API.md's PAT constraints list
the enumerated routes and drops a feedback mention that belonged to the
stateless /api/runs axis.

* docs(migration): add the 0017 renumbering coordination note to 0017

The PR's migration-coordination comment states each migration file
carries the note; the file did not. Adds it: numbering was generated
against main head 0016 alongside #5078 and #4843; whoever merges first
keeps the slot, the others renumber on rebase (revision/down_revision
plus the bootstrap head assertions).

* fix(auth): pad base62 tokens to a fixed 43-char width

int.from_bytes discards leading zero bytes, so the unpadded encoder
returned a variable-length body — empty for all-zero input, and shorter
than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving
test_generate_pat_token_format probabilistically flaky and the token
body without stable width (review round 6, P3).

_base62 now left-pads with "0" to _base62_width(len(data)) — the exact
integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The
format test asserts the exact fixed width instead of a probabilistic
floor, and a new unit test pins the all-zero, leading-zero-byte, and
max-value edges deterministically.
2026-08-29 23:50:45 +08:00

283 lines
11 KiB
Python

"""CSRF protection middleware for FastAPI.
Per RFC-001:
State-changing operations require CSRF protection.
"""
import os
import secrets
from collections.abc import Awaitable, Callable
from urllib.parse import urlsplit
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.session_cookie_state import SESSION_COOKIE_ISSUED_STATE_ATTR, SESSION_COOKIE_MAX_AGE_STATE_ATTR, SESSION_COOKIE_SECURE_STATE_ATTR, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR
from app.gateway.auth_disabled import is_auth_disabled
from app.gateway.request_path import get_request_route_path
CSRF_COOKIE_NAME = "csrf_token"
CSRF_HEADER_NAME = "X-CSRF-Token"
CSRF_TOKEN_LENGTH = 64 # bytes
_CSRF_STATE_CHANGING_METHODS: frozenset[str] = frozenset({"POST", "PUT", "DELETE", "PATCH"})
_CSRF_EXEMPT_EXACT_PATHS: frozenset[str] = frozenset({"/api/v1/auth/me"})
def is_secure_request(request: Request) -> bool:
"""Detect whether the original client request was made over HTTPS."""
return _request_scheme(request) == "https"
def generate_csrf_token() -> str:
"""Generate a secure random CSRF token."""
return secrets.token_urlsafe(CSRF_TOKEN_LENGTH)
def should_check_csrf(request: Request) -> bool:
"""Determine if a request needs CSRF validation.
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
"""
if request.method not in _CSRF_STATE_CHANGING_METHODS:
return False
if is_auth_disabled():
return False
route_path = get_request_route_path(request)
path = route_path.rstrip("/")
# Exempt host-owned endpoints that implement their own request posture.
if path in _CSRF_EXEMPT_EXACT_PATHS:
return False
# Inbound webhooks authenticate themselves via provider-specific signatures
# (e.g. GitHub's X-Hub-Signature-256), not the CSRF double-submit cookie.
if route_path.startswith("/api/webhooks/"):
return False
return True
_AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
{
"/api/v1/auth/login/local",
"/api/v1/auth/logout",
"/api/v1/auth/register",
"/api/v1/auth/initialize",
}
)
def is_auth_endpoint(request: Request) -> bool:
"""Check if the request is to an auth endpoint.
Auth endpoints don't need CSRF validation on first call (no token).
"""
return get_request_route_path(request).rstrip("/") in _AUTH_EXEMPT_PATHS
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:
"""Return normalized host[:port], omitting default ports."""
host = hostname.lower()
if ":" in host and not host.startswith("["):
host = f"[{host}]"
if port is None or (scheme == "http" and port == 80) or (scheme == "https" and port == 443):
return host
return f"{host}:{port}"
def _normalize_origin(origin: str) -> str | None:
"""Return a normalized scheme://host[:port] origin, or None for invalid input."""
try:
parsed = urlsplit(origin.strip())
port = parsed.port
except ValueError:
return None
scheme = parsed.scheme.lower()
if scheme not in {"http", "https"} or not parsed.hostname:
return None
# Browser Origin is only scheme/host/port. Reject URL-shaped or credentialed values.
if parsed.username or parsed.password or parsed.path or parsed.query or parsed.fragment:
return None
return f"{scheme}://{_host_with_optional_port(parsed.hostname, port, scheme)}"
def _configured_cors_origins() -> set[str]:
"""Return explicit configured browser origins that may call auth routes."""
origins = set()
for raw_origin in os.environ.get("GATEWAY_CORS_ORIGINS", "").split(","):
origin = raw_origin.strip()
if not origin or origin == "*":
continue
normalized = _normalize_origin(origin)
if normalized:
origins.add(normalized)
return origins
def get_configured_cors_origins() -> set[str]:
"""Return normalized explicit browser origins from GATEWAY_CORS_ORIGINS."""
return _configured_cors_origins()
# Response headers a split-origin browser client must be able to read. Only the
# CORS-safelisted set is visible to JS by default, and the created run's id
# travels in `Content-Location` — the LangGraph SDK resolves run metadata from
# it, so withholding it leaves such a client unable to learn its own run id.
CORS_EXPOSED_HEADERS: tuple[str, ...] = ("Content-Location",)
def _first_header_value(value: str | None) -> str | None:
"""Return the first value from a comma-separated proxy header."""
if not value:
return None
first = value.split(",", 1)[0].strip()
return first or None
def _forwarded_param(request: Request, name: str) -> str | None:
"""Extract a parameter from the first RFC 7239 Forwarded header entry."""
forwarded = _first_header_value(request.headers.get("forwarded"))
if not forwarded:
return None
for part in forwarded.split(";"):
key, sep, value = part.strip().partition("=")
if sep and key.lower() == name:
return value.strip().strip('"') or None
return None
def _request_scheme(request: Request) -> str:
"""Resolve the original request scheme from trusted proxy headers."""
scheme = _forwarded_param(request, "proto") or _first_header_value(request.headers.get("x-forwarded-proto")) or request.url.scheme
return scheme.lower()
def _request_origin(request: Request) -> str | None:
"""Build the origin for the URL the browser is targeting."""
scheme = _request_scheme(request)
host = _forwarded_param(request, "host") or _first_header_value(request.headers.get("x-forwarded-host")) or request.headers.get("host") or request.url.netloc
forwarded_port = _first_header_value(request.headers.get("x-forwarded-port"))
if forwarded_port and ":" not in host.rsplit("]", 1)[-1]:
host = f"{host}:{forwarded_port}"
return _normalize_origin(f"{scheme}://{host}")
def is_allowed_auth_origin(request: Request) -> bool:
"""Allow auth POSTs only from the same origin or explicit configured origins.
Login/register/initialize are exempt from the double-submit token because
first-time browser clients do not have a CSRF token yet. They still create
a session cookie, so browser requests with a hostile Origin header must be
rejected to prevent login CSRF / session fixation. Requests without Origin
are allowed for non-browser clients such as curl and mobile integrations.
"""
origin = request.headers.get("origin")
if not origin:
return True
normalized_origin = _normalize_origin(origin)
if normalized_origin is None:
return False
request_origin = _request_origin(request)
return normalized_origin in _configured_cors_origins() or (request_origin is not None and normalized_origin == request_origin)
def auth_csrf_cookie_settings(request: Request) -> tuple[bool, int | None]:
"""Return ``(secure, max_age)`` for auth-created CSRF cookies."""
session_cookie_issued = getattr(request.state, SESSION_COOKIE_ISSUED_STATE_ATTR, False)
if session_cookie_issued:
return (
bool(getattr(request.state, SESSION_COOKIE_SECURE_STATE_ATTR, is_secure_request(request))),
getattr(request.state, SESSION_COOKIE_MAX_AGE_STATE_ATTR, None),
)
secure = is_secure_request(request)
max_age = get_auth_config().token_expiry_days * 24 * 3600 if secure else None
return secure, max_age
class CSRFMiddleware(BaseHTTPMiddleware):
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
_is_auth = is_auth_endpoint(request)
if should_check_csrf(request) and _is_auth and not is_allowed_auth_origin(request):
return JSONResponse(
status_code=403,
content={"detail": "Cross-site auth request denied."},
)
if should_check_csrf(request) and not _is_auth and request.headers.get("authorization") is None:
# Bearer-authenticated requests (PAT, #4849) are exempt from the
# cookie double-submit check only — the cross-site origin check on
# auth endpoints above still runs for every request. Safety rests
# on AuthMiddleware's strict Bearer precedence: an invalid Bearer
# header is a 401 there, so a cross-site attacker cannot ride a
# victim's cookie by padding a garbage Authorization header, and a
# cross-site request carrying a custom Authorization header at all
# requires a CORS preflight the attacker cannot obtain.
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
header_token = request.headers.get(CSRF_HEADER_NAME)
if not cookie_token or not header_token:
return JSONResponse(
status_code=403,
content={"detail": "CSRF token missing. Include X-CSRF-Token header."},
)
if not secrets.compare_digest(cookie_token, header_token):
return JSONResponse(
status_code=403,
content={"detail": "CSRF token mismatch."},
)
response = await call_next(request)
# For auth endpoints that set up session, also set CSRF cookie.
# Session-creating handlers may stamp the final access-token max_age on
# request.state; mirroring it here keeps the double-submit cookie pair
# from diverging across HTTPS, localhost, and sandbox deployments.
if _is_auth and request.method == "POST" and not getattr(request.state, SKIP_AUTH_CSRF_COOKIE_STATE_ATTR, False):
# Generate a new CSRF token for the session
csrf_token = generate_csrf_token()
secure, max_age = auth_csrf_cookie_settings(request)
response.set_cookie(
key=CSRF_COOKIE_NAME,
value=csrf_token,
httponly=False, # Must be JS-readable for Double Submit Cookie pattern
secure=secure,
samesite="strict",
# Match the access_token cookie's lifetime (auth.py::_set_session_cookie)
# so the double-submit pair never diverges. A session-only csrf_token is
# evicted when iOS Safari terminates a home-screen PWA while the persistent
# access_token survives — leaving the user "logged in" but unable to make
# any state-changing request (403 "CSRF token missing").
max_age=max_age,
)
return response
def get_csrf_token(request: Request) -> str | None:
"""Get the CSRF token from the current request's cookies.
This is useful for server-side rendering where you need to embed
token in forms or headers.
"""
return request.cookies.get(CSRF_COOKIE_NAME)