deer-flow/backend/app/gateway/auth_middleware.py
Sunshine bf740ffa90
feat(auth): add personal access tokens for programmatic API access (#5041)
* feat(auth): add personal access tokens for programmatic API access (#4849)

Backend-first implementation of the PAT contract from #4849: show-once
dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT,
is_internal=false), digest-only storage (migration 0017), strict
credential precedence (invalid Bearer is a 401, never cookie fallback),
CSRF double-submit skipped only for Bearer requests while
auth-endpoint origin checks still run, scopes intersecting the authz
route permissions, session-auth-only PAT management and password
changes, and throttled best-effort last_used_at stamps.

* fix(auth): harden PAT scope boundary and schema parity from adversarial review

Independent review of the initial draft found: (1) scopes only constrained
the threads/runs permission axis while admin routes treated a PAT as its
(possibly admin) owner — is_admin_user now rejects PAT callers outright
since no scope grants admin capability; (2) the model declared a column
UNIQUE constraint while migration 0017 created a named unique index, so
downgrade failed on create_all-bootstrapped DBs — both now use the named
unique index; (3) auth-disabled mode is an operator override and now stays
ahead of the Bearer check so a stray Authorization header cannot 401 an
E2E sandbox; plus wiring the previously-unused constants, bounding the
last_used_at stamp cache, and four new tests (middleware-level expiry,
expires_in_days, admin-capability rejection with session control, and the
auth-disabled precedence).

* docs(api): document personal access tokens for programmatic API access

* fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression)

P1-1: scope intersection only constrains @require_permission routes, so
undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark
credential switching, channel config) accepted a PAT holding a single read
scope. AuthMiddleware now enforces a default-deny route policy in
auth/pat.py: PAT requests are admitted only to the thread/run lifecycle
routes the v1 scopes govern; everything else answers 403 regardless of
scopes. Session-cookie callers are unaffected.

P1-2: the extension principal resolver projected is_admin/roles from the
raw system_role, so an admin-owned PAT passed
deerflow_extension_api.require_admin on contributed routes despite the
documented no-admin guarantee. The projection is now PAT-aware and
suppresses every admin signal for PAT callers, mirroring
deps.is_admin_user.

Both fixes carry regression tests (route outside policy 403 + session
control; production resolver admin suppression), and API.md documents the
default-deny boundary.

* fix(auth): enforce PAT scopes on stateless run entry and harden decorator

Follow-up hardening from an independent audit of the P1 fixes:

- POST /api/runs/stream and /api/runs/wait were the only allowlisted run
  entrypoints without @require_permission, so a threads:read-only PAT
  could still start runs (same bug class as P1-1, now closed): both now
  carry @require_permission("runs", "create"). POST /api/threads and
  POST /api/threads/search gain threads:write / threads:read for the
  same reason. Authorization-disabled deployments see no change (the
  permission set resolves to all permissions).
- require_permission now binds the wrapped signature to locate a
  positionally-passed request before injecting the test stub, fixing
  'got multiple values for argument' on direct positional unit-test
  calls.
- API.md: the intro PAT example used GET /api/models, which the new
  default-deny policy 403s — replaced with GET /api/threads; the
  default-deny route list now spells out method sets.

Regression test: threads:read-only PAT is 403 on the decorated stateless
entry while a runs:create PAT passes.

* fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example)

- CSRFMiddleware treats an explicitly empty Authorization header as
  present (is None), so an invalid credential always reaches
  AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by
  method/CSRF state. Regression: empty-header request dies at auth.
- PATCreateRequest strips the name and rejects whitespace-only values
  before token generation; created names are stored trimmed.
- API.md intro PAT example now uses the implemented
  POST /api/threads/search endpoint (GET /api/threads does not exist).
- AGENTS.md trimmed back under the guidance soft budget after the
  upstream merge.

* fix(auth): tighten PAT route policy to implemented methods only

The allowlist admitted GET /api/threads, a method no router implements.
Pre-authorizing a dead method weakens the default-deny boundary: a
future GET collection route added without a permission decorator would
become PAT-reachable without an explicit policy change. Restrict the
rule to POST, fix the stale GET description in API.md's PAT
constraints, and document the default-deny boundary accurately in the
gateway AGENTS.md guidance (only the threads/runs allowlist is
PAT-reachable; every other authenticated route 403s PAT callers).

Audited every remaining rule against the mounted routers: all other
method+path entries map to real routes. Regression:
test_pat_policy_does_not_pre_authorize_unimplemented_methods.

* test(auth): guarantee the negative digest test mutates the token

token[:-1] + "X" is identical to the original whenever the generated
token already ends in X (1/62), making the negative digest assertion
fail intermittently. Choose the replacement character based on the
existing tail so the mutated token always differs.

* fix(auth): require runs:cancel for cancel-then-stream requests

stream_existing_run is gated at runs:read so action-less stream joins
work with read-only credentials, but its ?action=interrupt|rollback
branch cancels the run — a separate permission. A runs:read-only PAT
passed both the PAT route policy and the route decorator and could
interrupt or roll back an active run, bypassing the runs:cancel scope.

Decorators cannot express query-parameter-conditional permissions, so
the check lives in require_cancel_permission_when_action(), applied at
the top of the handler. Regression drives the real helper through the
production middleware: runs:read-only PAT + action is 403, the same
token joins action-less, runs:read+cancel passes, session control
unaffected.

* docs(changelog): add the PAT feature entry

* docs(readme): add personal access tokens section

Repo documentation-update policy requires user-facing features to
update README.md in the same changeset; the PAT feature previously
touched only backend/docs/API.md and the gateway AGENTS.md.

* fix(auth): require runs:cancel for mutating multitask strategies

All five run-creation entrypoints were gated only by runs:create, but
RunCreateRequest.multitask_strategy accepts interrupt/rollback and
start_run forwards it to create_or_reject, which terminates an
already-active run. A runs:create-only PAT could therefore kill an
existing run through a create request, bypassing runs:cancel.

Decorators cannot express body-parameter-conditional permissions, and
per-route checks leave the same hole for the next entrypoint, so the
gate lives in start_run itself — the single choke point every
run-creation path (HTTP routes and internal launchers) flows through.
Regenerate launches pass multitask_strategy="reject" and are
unaffected; requests without a stamped auth context (internal/test
compositions) skip the gate.

The check is the shared authz.require_cancel_permission_if primitive;
require_cancel_permission_when_action now delegates to it, so every
request dimension that carries cancel capability (query action, body
strategy) flows through one gate.

Regression drives the real middleware stack: runs:create-only PAT +
interrupt/rollback is 403 with the exact detail, reject (explicit and
default) stays available, runs:create+cancel passes, session control
unaffected; a source anchor pins the gate inside start_run.

* fix(runs): keep observer joins from applying creator cancel-on-disconnect

sse_consumer's finally block applied the record's on_disconnect=cancel
policy on ANY consumer's disconnect. The join surfaces (GET /join and
the action-less GET/POST stream join) feed it the existing RunRecord,
so anyone with thread read access — including a runs:read-only PAT —
could cancel a locally-owned running run simply by closing the SSE
connection, without runs:cancel. The policy expresses the creator's
intent for their own connection; an observer's disconnect must never
be read as that intent.

sse_consumer gains apply_on_disconnect (default True). The two join
surfaces pass False; the creating endpoints (thread-scoped and
stateless create-and-stream) keep the creator semantics unchanged.
wait_for_run_completion needs no change: its callers are creator-side
or post-explicit-cancel paths only.

Regression exercises a real generator close — the same machinery
Starlette drives on client disconnect — against the production
sse_consumer: creator stream disconnect cancels, observer join
disconnect does not; a wiring anchor pins both join call sites and the
creator defaults. API.md documents the cancel-capability constraint
(this fix plus the action/strategy gates) in PAT Constraints.

* test(auth): pin the multitask gate behaviorally; state wait invariant

Independent adversarial review of the round-5 fixes found the P1-a
regression only mirror-pinned: the source anchor could be satisfied by
a comment, and deleting the gate from start_run would not fail the
suite. This drives the production start_run directly — a create-only
auth context gets 403 with the exact detail for interrupt, and a
reject request with no cancel permission at all proceeds past the gate
(never a permission 403).

Also documents wait_for_run_completion's creator-side invariant
(every caller is the creating endpoint or post-explicit-cancel) so a
future observer wiring thinks twice before reusing it — the one-caller-
away variant of the observer-disconnect P1.

* docs(changelog): correct the PAT entry's digest and route-policy description

The entry said HMAC digests (the implementation stores SHA-256 digests,
as documented in API.md and pinned by the repository tests) and claimed
the route policy admits 'implemented stateless endpoints' (it admits
the thread/run lifecycle routes, narrowing further by scopes). Also
notes the cancel-capability gate now covering action and multitask
strategies.

* fix(auth): enumerate the PAT runs route policy per implemented subroute

The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it
pre-authorized every current and future subroute under /runs, including
methods the router never implemented (e.g. GET /runs/stream), which is
the same latent default-deny weakening the threads collection rule was
tightened for: a future route added under /runs would become
PAT-reachable without an explicit policy change.

The wildcard is replaced with six segment-precise rules covering exactly
the 14 implemented method+path combinations; the {run_id} slot
necessarily matches any single segment, so the POST-only collection
names (stream, wait, regenerate, edit-regenerate) are excluded from the
GET run-id rule via negative lookahead — no dead method stays
pre-authorized. Behavior for implemented routes is unchanged.

test_pat_runs_policy_admits_exactly_the_mounted_routes derives the
expected set from the mounted thread_runs router instead of a
hand-maintained list: every implemented GET/POST route under /runs must
be admitted, routes in this router outside the subtree stay denied, and
representative unimplemented neighbors are denied — so adding a route
under /runs now fails CI until it is explicitly allowlisted, and a
removed route leaves a dead rule visible. API.md's PAT constraints list
the enumerated routes and drops a feedback mention that belonged to the
stateless /api/runs axis.

* docs(migration): add the 0017 renumbering coordination note to 0017

The PR's migration-coordination comment states each migration file
carries the note; the file did not. Adds it: numbering was generated
against main head 0016 alongside #5078 and #4843; whoever merges first
keeps the slot, the others renumber on rebase (revision/down_revision
plus the bootstrap head assertions).

* fix(auth): pad base62 tokens to a fixed 43-char width

int.from_bytes discards leading zero bytes, so the unpadded encoder
returned a variable-length body — empty for all-zero input, and shorter
than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving
test_generate_pat_token_format probabilistically flaky and the token
body without stable width (review round 6, P3).

_base62 now left-pads with "0" to _base62_width(len(data)) — the exact
integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The
format test asserts the exact fixed width instead of a probabilistic
floor, and a new unit test pins the all-zero, leading-zero-byte, and
max-value edges deterministically.
2026-08-29 23:50:45 +08:00

200 lines
9.0 KiB
Python

"""Global authentication middleware — fail-closed safety net.
Rejects unauthenticated requests to non-public paths with 401. When a
request passes the cookie check, resolves the JWT payload to a real
``User`` object and stamps it into both ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filtering works automatically via the sentinel pattern.
Fine-grained permission checks remain in authz.py decorators.
"""
from collections.abc import Callable
from fastapi import HTTPException, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.auth_disabled import (
AUTH_SOURCE_AUTH_DISABLED,
AUTH_SOURCE_INTERNAL,
AUTH_SOURCE_PAT,
AUTH_SOURCE_SESSION,
get_auth_disabled_user,
is_auth_disabled,
)
from app.gateway.authz import AuthContext, resolve_route_permissions
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
from app.gateway.request_path import get_request_route_path
from deerflow.runtime.user_context import reset_current_user, set_current_user
# Paths that never require authentication.
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
"/health",
"/docs",
"/redoc",
"/openapi.json",
"/api/v1/auth/oauth/",
"/api/v1/auth/callback/",
# Inbound webhooks authenticate themselves via provider-specific signatures
# (e.g. GitHub's X-Hub-Signature-256), not session cookies.
"/api/webhooks/",
)
# Exact auth paths that are public (login/register/status check).
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
{
"/api/v1/auth/login/local",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
"/api/v1/auth/providers",
}
)
def _is_public(path: str) -> bool:
stripped = path.rstrip("/")
if stripped in _PUBLIC_EXACT_PATHS:
return True
return any(path.startswith(prefix) for prefix in _PUBLIC_PATH_PREFIXES)
class AuthMiddleware(BaseHTTPMiddleware):
"""Strict auth gate: reject requests without a valid session.
Two-stage check for non-public paths:
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
2. JWT validation via ``get_optional_user_from_request`` — return 401
TOKEN_INVALID if the token is absent, malformed, expired, or the
signed user does not exist / is stale
On success, stamps ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filters work downstream without every route needing a
``@require_auth`` decorator. Routes that need per-resource
authorization (e.g. "user A cannot read user B's thread by guessing
the URL") should additionally use ``@require_permission(...,
owner_check=True)`` for explicit enforcement — but authentication
itself is fully handled here.
"""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
if _is_public(get_request_route_path(request)):
return await call_next(request)
internal_user = None
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
# Extract the channel owner user ID from the trusted header.
# When present, the synthetic internal user carries the actual
# owner identity so that get_effective_user_id() and per-user
# filesystem paths (custom skills, memory, thread data) resolve
# to the IM channel user instead of falling back to "default".
from app.gateway.internal_auth import INTERNAL_OWNER_USER_ID_HEADER_NAME
owner_user_id = request.headers.get(INTERNAL_OWNER_USER_ID_HEADER_NAME)
if owner_user_id:
owner_user_id = owner_user_id.strip()
internal_user = get_internal_user(owner_user_id=owner_user_id or None)
auth_source = AUTH_SOURCE_SESSION
access_token = request.cookies.get("access_token")
authorization = request.headers.get("authorization")
pat_scopes: frozenset[str] = frozenset()
# Non-public path: require session cookie
if internal_user is not None:
user = internal_user
auth_source = AUTH_SOURCE_INTERNAL
elif authorization is not None and not is_auth_disabled():
# Bearer (PAT) credential precedence (#4849): a present-but-invalid
# Authorization header is a hard 401 and never silently falls back
# to the session cookie. This is also what makes the CSRF
# middleware's Bearer skip safe — a cross-site attacker cannot ride
# a victim's cookie by padding the request with a garbage Bearer
# header, because the request dies here before any route runs.
# Auth-disabled mode is an operator override of all authentication,
# so it stays ahead of the Bearer check (a stray Authorization
# header from a proxy must not 401 an E2E sandbox).
from app.gateway.auth.pat import authenticate_pat, is_pat_allowed_route
try:
user, pat_scopes = await authenticate_pat(request.app, authorization)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
# Default-deny route boundary (#5041 review P1-1): scopes only
# constrain @require_permission routes, so any route outside the
# explicit PAT policy is closed to PAT callers outright — an
# all-scopes token must not reach undecorated mutation routes.
if not is_pat_allowed_route(request.method, get_request_route_path(request)):
return JSONResponse(
status_code=403,
content={"detail": "PAT credentials are not permitted on this route"},
)
auth_source = AUTH_SOURCE_PAT
elif access_token:
# Strict JWT validation: reject junk/expired tokens with 401
# right here instead of silently passing through. This closes
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
# without this, non-isolation routes like /api/models would
# accept any cookie-shaped string as authentication.
#
# We call the *strict* resolver so that fine-grained error
# codes (token_expired, token_invalid, user_not_found, …)
# propagate from AuthErrorCode, not get flattened into one
# generic code. BaseHTTPMiddleware doesn't let HTTPException
# bubble up, so we catch and render it as JSONResponse here.
from app.gateway.deps import get_current_user_from_request
try:
user = await get_current_user_from_request(request)
except HTTPException as exc:
if not is_auth_disabled():
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
user = get_auth_disabled_user()
auth_source = AUTH_SOURCE_AUTH_DISABLED
elif is_auth_disabled():
user = get_auth_disabled_user()
auth_source = AUTH_SOURCE_AUTH_DISABLED
else:
return JSONResponse(
status_code=401,
content={
"detail": AuthErrorResponse(
code=AuthErrorCode.NOT_AUTHENTICATED,
message="Authentication required",
).model_dump()
},
)
# Stamp both request.state.user (for the contextvar pattern)
# and request.state.auth (so @require_permission's "auth is
# None" branch short-circuits instead of running the entire
# JWT-decode + DB-lookup pipeline a second time per request).
request.state.user = user
request.state.auth_source = auth_source
permissions = await resolve_route_permissions(
user,
is_internal=auth_source == AUTH_SOURCE_INTERNAL,
)
if auth_source == AUTH_SOURCE_PAT:
# A PAT can only narrow its owning user's permissions: the stored
# scopes intersect the resolved route permissions, never widen
# them, and role changes / authorization policy stay authoritative
# because they were resolved fresh from the owning user above.
permissions = [permission for permission in permissions if permission in pat_scopes]
request.state.auth = AuthContext(user=user, permissions=permissions)
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)