mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-16 17:46:20 +00:00
* feat(auth): add personal access tokens for programmatic API access (#4849) Backend-first implementation of the PAT contract from #4849: show-once dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT, is_internal=false), digest-only storage (migration 0017), strict credential precedence (invalid Bearer is a 401, never cookie fallback), CSRF double-submit skipped only for Bearer requests while auth-endpoint origin checks still run, scopes intersecting the authz route permissions, session-auth-only PAT management and password changes, and throttled best-effort last_used_at stamps. * fix(auth): harden PAT scope boundary and schema parity from adversarial review Independent review of the initial draft found: (1) scopes only constrained the threads/runs permission axis while admin routes treated a PAT as its (possibly admin) owner — is_admin_user now rejects PAT callers outright since no scope grants admin capability; (2) the model declared a column UNIQUE constraint while migration 0017 created a named unique index, so downgrade failed on create_all-bootstrapped DBs — both now use the named unique index; (3) auth-disabled mode is an operator override and now stays ahead of the Bearer check so a stray Authorization header cannot 401 an E2E sandbox; plus wiring the previously-unused constants, bounding the last_used_at stamp cache, and four new tests (middleware-level expiry, expires_in_days, admin-capability rejection with session control, and the auth-disabled precedence). * docs(api): document personal access tokens for programmatic API access * fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression) P1-1: scope intersection only constrains @require_permission routes, so undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark credential switching, channel config) accepted a PAT holding a single read scope. AuthMiddleware now enforces a default-deny route policy in auth/pat.py: PAT requests are admitted only to the thread/run lifecycle routes the v1 scopes govern; everything else answers 403 regardless of scopes. Session-cookie callers are unaffected. P1-2: the extension principal resolver projected is_admin/roles from the raw system_role, so an admin-owned PAT passed deerflow_extension_api.require_admin on contributed routes despite the documented no-admin guarantee. The projection is now PAT-aware and suppresses every admin signal for PAT callers, mirroring deps.is_admin_user. Both fixes carry regression tests (route outside policy 403 + session control; production resolver admin suppression), and API.md documents the default-deny boundary. * fix(auth): enforce PAT scopes on stateless run entry and harden decorator Follow-up hardening from an independent audit of the P1 fixes: - POST /api/runs/stream and /api/runs/wait were the only allowlisted run entrypoints without @require_permission, so a threads:read-only PAT could still start runs (same bug class as P1-1, now closed): both now carry @require_permission("runs", "create"). POST /api/threads and POST /api/threads/search gain threads:write / threads:read for the same reason. Authorization-disabled deployments see no change (the permission set resolves to all permissions). - require_permission now binds the wrapped signature to locate a positionally-passed request before injecting the test stub, fixing 'got multiple values for argument' on direct positional unit-test calls. - API.md: the intro PAT example used GET /api/models, which the new default-deny policy 403s — replaced with GET /api/threads; the default-deny route list now spells out method sets. Regression test: threads:read-only PAT is 403 on the decorated stateless entry while a runs:create PAT passes. * fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example) - CSRFMiddleware treats an explicitly empty Authorization header as present (is None), so an invalid credential always reaches AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by method/CSRF state. Regression: empty-header request dies at auth. - PATCreateRequest strips the name and rejects whitespace-only values before token generation; created names are stored trimmed. - API.md intro PAT example now uses the implemented POST /api/threads/search endpoint (GET /api/threads does not exist). - AGENTS.md trimmed back under the guidance soft budget after the upstream merge. * fix(auth): tighten PAT route policy to implemented methods only The allowlist admitted GET /api/threads, a method no router implements. Pre-authorizing a dead method weakens the default-deny boundary: a future GET collection route added without a permission decorator would become PAT-reachable without an explicit policy change. Restrict the rule to POST, fix the stale GET description in API.md's PAT constraints, and document the default-deny boundary accurately in the gateway AGENTS.md guidance (only the threads/runs allowlist is PAT-reachable; every other authenticated route 403s PAT callers). Audited every remaining rule against the mounted routers: all other method+path entries map to real routes. Regression: test_pat_policy_does_not_pre_authorize_unimplemented_methods. * test(auth): guarantee the negative digest test mutates the token token[:-1] + "X" is identical to the original whenever the generated token already ends in X (1/62), making the negative digest assertion fail intermittently. Choose the replacement character based on the existing tail so the mutated token always differs. * fix(auth): require runs:cancel for cancel-then-stream requests stream_existing_run is gated at runs:read so action-less stream joins work with read-only credentials, but its ?action=interrupt|rollback branch cancels the run — a separate permission. A runs:read-only PAT passed both the PAT route policy and the route decorator and could interrupt or roll back an active run, bypassing the runs:cancel scope. Decorators cannot express query-parameter-conditional permissions, so the check lives in require_cancel_permission_when_action(), applied at the top of the handler. Regression drives the real helper through the production middleware: runs:read-only PAT + action is 403, the same token joins action-less, runs:read+cancel passes, session control unaffected. * docs(changelog): add the PAT feature entry * docs(readme): add personal access tokens section Repo documentation-update policy requires user-facing features to update README.md in the same changeset; the PAT feature previously touched only backend/docs/API.md and the gateway AGENTS.md. * fix(auth): require runs:cancel for mutating multitask strategies All five run-creation entrypoints were gated only by runs:create, but RunCreateRequest.multitask_strategy accepts interrupt/rollback and start_run forwards it to create_or_reject, which terminates an already-active run. A runs:create-only PAT could therefore kill an existing run through a create request, bypassing runs:cancel. Decorators cannot express body-parameter-conditional permissions, and per-route checks leave the same hole for the next entrypoint, so the gate lives in start_run itself — the single choke point every run-creation path (HTTP routes and internal launchers) flows through. Regenerate launches pass multitask_strategy="reject" and are unaffected; requests without a stamped auth context (internal/test compositions) skip the gate. The check is the shared authz.require_cancel_permission_if primitive; require_cancel_permission_when_action now delegates to it, so every request dimension that carries cancel capability (query action, body strategy) flows through one gate. Regression drives the real middleware stack: runs:create-only PAT + interrupt/rollback is 403 with the exact detail, reject (explicit and default) stays available, runs:create+cancel passes, session control unaffected; a source anchor pins the gate inside start_run. * fix(runs): keep observer joins from applying creator cancel-on-disconnect sse_consumer's finally block applied the record's on_disconnect=cancel policy on ANY consumer's disconnect. The join surfaces (GET /join and the action-less GET/POST stream join) feed it the existing RunRecord, so anyone with thread read access — including a runs:read-only PAT — could cancel a locally-owned running run simply by closing the SSE connection, without runs:cancel. The policy expresses the creator's intent for their own connection; an observer's disconnect must never be read as that intent. sse_consumer gains apply_on_disconnect (default True). The two join surfaces pass False; the creating endpoints (thread-scoped and stateless create-and-stream) keep the creator semantics unchanged. wait_for_run_completion needs no change: its callers are creator-side or post-explicit-cancel paths only. Regression exercises a real generator close — the same machinery Starlette drives on client disconnect — against the production sse_consumer: creator stream disconnect cancels, observer join disconnect does not; a wiring anchor pins both join call sites and the creator defaults. API.md documents the cancel-capability constraint (this fix plus the action/strategy gates) in PAT Constraints. * test(auth): pin the multitask gate behaviorally; state wait invariant Independent adversarial review of the round-5 fixes found the P1-a regression only mirror-pinned: the source anchor could be satisfied by a comment, and deleting the gate from start_run would not fail the suite. This drives the production start_run directly — a create-only auth context gets 403 with the exact detail for interrupt, and a reject request with no cancel permission at all proceeds past the gate (never a permission 403). Also documents wait_for_run_completion's creator-side invariant (every caller is the creating endpoint or post-explicit-cancel) so a future observer wiring thinks twice before reusing it — the one-caller- away variant of the observer-disconnect P1. * docs(changelog): correct the PAT entry's digest and route-policy description The entry said HMAC digests (the implementation stores SHA-256 digests, as documented in API.md and pinned by the repository tests) and claimed the route policy admits 'implemented stateless endpoints' (it admits the thread/run lifecycle routes, narrowing further by scopes). Also notes the cancel-capability gate now covering action and multitask strategies. * fix(auth): enumerate the PAT runs route policy per implemented subroute The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it pre-authorized every current and future subroute under /runs, including methods the router never implemented (e.g. GET /runs/stream), which is the same latent default-deny weakening the threads collection rule was tightened for: a future route added under /runs would become PAT-reachable without an explicit policy change. The wildcard is replaced with six segment-precise rules covering exactly the 14 implemented method+path combinations; the {run_id} slot necessarily matches any single segment, so the POST-only collection names (stream, wait, regenerate, edit-regenerate) are excluded from the GET run-id rule via negative lookahead — no dead method stays pre-authorized. Behavior for implemented routes is unchanged. test_pat_runs_policy_admits_exactly_the_mounted_routes derives the expected set from the mounted thread_runs router instead of a hand-maintained list: every implemented GET/POST route under /runs must be admitted, routes in this router outside the subtree stay denied, and representative unimplemented neighbors are denied — so adding a route under /runs now fails CI until it is explicitly allowlisted, and a removed route leaves a dead rule visible. API.md's PAT constraints list the enumerated routes and drops a feedback mention that belonged to the stateless /api/runs axis. * docs(migration): add the 0017 renumbering coordination note to 0017 The PR's migration-coordination comment states each migration file carries the note; the file did not. Adds it: numbering was generated against main head 0016 alongside #5078 and #4843; whoever merges first keeps the slot, the others renumber on rebase (revision/down_revision plus the bootstrap head assertions). * fix(auth): pad base62 tokens to a fixed 43-char width int.from_bytes discards leading zero bytes, so the unpadded encoder returned a variable-length body — empty for all-zero input, and shorter than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving test_generate_pat_token_format probabilistically flaky and the token body without stable width (review round 6, P3). _base62 now left-pads with "0" to _base62_width(len(data)) — the exact integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The format test asserts the exact fixed width instead of a probabilistic floor, and a new unit test pins the all-zero, leading-zero-byte, and max-value edges deterministically.
197 lines
8.2 KiB
Python
197 lines
8.2 KiB
Python
"""Personal Access Token (PAT) credentials for programmatic API access.
|
|
|
|
Tokens are ``dfp_`` + base62(32 CSPRNG bytes), shown exactly once in the
|
|
create response and persisted only as a SHA-256 digest. Validation is a
|
|
digest-indexed lookup plus a constant-time re-comparison, with a single
|
|
generic failure surface so a 401 never reveals which check failed.
|
|
|
|
v1 scopes are exactly the route-permission strings owned by
|
|
``app.gateway.authz`` — a PAT can only narrow its owning user's
|
|
permissions, never widen them.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import functools
|
|
import hashlib
|
|
import hmac
|
|
import re
|
|
import secrets
|
|
from typing import Any
|
|
|
|
PAT_TOKEN_PREFIX = "dfp_"
|
|
PAT_RANDOM_BYTES = 32
|
|
# Best-effort ``last_used_at`` writes are throttled per token so high-volume
|
|
# automation does not turn every request into a database write.
|
|
PAT_LAST_USED_WRITE_INTERVAL_SECONDS = 300.0
|
|
|
|
PAT_ALLOWED_SCOPES: frozenset[str] = frozenset(
|
|
{
|
|
"threads:read",
|
|
"threads:write",
|
|
"threads:delete",
|
|
"runs:create",
|
|
"runs:read",
|
|
"runs:cancel",
|
|
}
|
|
)
|
|
|
|
PAT_MAX_NAME_LENGTH = 128
|
|
|
|
# Default-deny route boundary for PAT callers (#5041 review P1-1): scope
|
|
# intersection in AuthMiddleware only constrains routes that consult
|
|
# ``request.state.auth.permissions`` (``@require_permission``). Authenticated
|
|
# mutation routes without that decorator — memory deletion, agent creation,
|
|
# credential switching, channel configuration — would otherwise accept a
|
|
# PAT holding a single read scope. A route is reachable by PAT only when it
|
|
# is explicitly listed here, and only together with the thread/run lifecycle
|
|
# the v1 scopes govern; everything else answers 403 regardless of scopes.
|
|
_PAT_ROUTE_RULES: tuple[tuple[frozenset[str], re.Pattern[str]], ...] = (
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/search$")),
|
|
(frozenset({"GET", "PATCH", "DELETE"}), re.compile(r"^/api/threads/[^/]+$")),
|
|
(frozenset({"GET", "PUT", "DELETE"}), re.compile(r"^/api/threads/[^/]+/goal$")),
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/state$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/[^/]+/(compact|history|branches)$")),
|
|
# Runs subtree: enumerated per implemented subroute instead of a
|
|
# ``runs(/.*)?`` wildcard, so a route added under /runs is default-denied
|
|
# until explicitly listed — the same no-dead-methods precision the
|
|
# threads collection rule enforces. The ``{run_id}`` slot necessarily
|
|
# matches any single segment; the POST-only collection endpoints sharing
|
|
# that depth (stream, wait, regenerate, edit-regenerate) are excluded
|
|
# from the GET run-id rule so no unimplemented method is pre-authorized.
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/runs$")),
|
|
(
|
|
frozenset({"POST"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/(stream|wait|regenerate/prepare|edit-regenerate/prepare)$"),
|
|
),
|
|
(
|
|
frozenset({"GET"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/(?!stream$|wait$|regenerate$|edit-regenerate$)[^/]+$"),
|
|
),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/[^/]+/runs/[^/]+/cancel$")),
|
|
(
|
|
frozenset({"GET"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/[^/]+/(join|messages|events|workspace-changes)$"),
|
|
),
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/runs/[^/]+/stream$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/runs/(stream|wait)$")),
|
|
(frozenset({"GET"}), re.compile(r"^/api/runs/[^/]+/(messages|feedback)$")),
|
|
)
|
|
|
|
_BASE62_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
def is_pat_allowed_route(method: str, path: str) -> bool:
|
|
"""Return whether the PAT route policy admits *method* + *path*.
|
|
|
|
Trailing slashes are normalized away so the mounted route and its
|
|
redirect-style twin resolve identically.
|
|
"""
|
|
normalized = path.rstrip("/") or "/"
|
|
return any(method in methods and pattern.match(normalized) for methods, pattern in _PAT_ROUTE_RULES)
|
|
|
|
|
|
@functools.cache
|
|
def _base62_width(byte_length: int) -> int:
|
|
"""Digits sufficient for any *byte_length*-byte value (exact integer math)."""
|
|
width = 1
|
|
limit = 1 << (8 * byte_length)
|
|
while 62**width < limit:
|
|
width += 1
|
|
return width
|
|
|
|
|
|
def _base62(data: bytes) -> str:
|
|
"""Fixed-width big-endian base62 of *data*, ``0``-padded on the left.
|
|
|
|
``int.from_bytes`` discards leading zero bytes, so an unpadded encoding
|
|
would be variable-length (and empty for all-zero input) — any draw below
|
|
62**39 would have produced a shorter-than-expected token. The fixed width
|
|
keeps every token body exactly ``_base62_width(len(data))`` characters
|
|
and makes the format test deterministic.
|
|
"""
|
|
value = int.from_bytes(data, "big")
|
|
digits: list[str] = []
|
|
while value:
|
|
value, remainder = divmod(value, 62)
|
|
digits.append(_BASE62_ALPHABET[remainder])
|
|
body = "".join(reversed(digits))
|
|
return body.rjust(_base62_width(len(data)), "0")
|
|
|
|
|
|
def generate_pat_token() -> str:
|
|
"""Generate a show-once raw token: ``dfp_`` + base62(CSPRNG bytes)."""
|
|
return PAT_TOKEN_PREFIX + _base62(secrets.token_bytes(PAT_RANDOM_BYTES))
|
|
|
|
|
|
def pat_token_digest(token: str) -> str:
|
|
"""Return the hex SHA-256 digest persisted for *token*."""
|
|
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
|
|
|
|
|
def digest_matches(stored_digest: str | None, token: str) -> bool:
|
|
"""Constant-time comparison of *token* against a stored digest."""
|
|
if not isinstance(stored_digest, str) or not stored_digest:
|
|
return False
|
|
return hmac.compare_digest(stored_digest, pat_token_digest(token))
|
|
|
|
|
|
def extract_bearer_token(authorization: str | None) -> str | None:
|
|
"""Return the Bearer credential from an Authorization header value.
|
|
|
|
``None`` means the request carries no Authorization header at all, so the
|
|
caller should fall through to the session-cookie path. Any other unusable
|
|
value (non-Bearer scheme, empty credential) returns ``""`` so callers
|
|
treat it as an invalid credential rather than an absent one.
|
|
"""
|
|
if authorization is None:
|
|
return None
|
|
scheme, _, value = authorization.partition(" ")
|
|
if scheme.lower() != "bearer":
|
|
return ""
|
|
return value.strip()
|
|
|
|
|
|
async def authenticate_pat(app: Any, authorization: str | None) -> tuple[Any, frozenset[str]]:
|
|
"""Validate the Bearer credential and resolve its owning user.
|
|
|
|
Returns ``(user, scopes)``. Every token-verdict failure mode — malformed
|
|
token, unknown/revoked/expired token, PAT store not configured, missing
|
|
owning user — raises the same generic 401 so responses cannot serve as an
|
|
oracle on which check failed. Infrastructure errors (store I/O failures)
|
|
propagate and fail closed; they are not part of the token verdict.
|
|
"""
|
|
from fastapi import HTTPException
|
|
|
|
token = extract_bearer_token(authorization)
|
|
if not token or not token.startswith(PAT_TOKEN_PREFIX):
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
pat_repo = getattr(app.state, "pat_repo", None)
|
|
if pat_repo is None:
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
record = await pat_repo.get_active_by_digest(pat_token_digest(token))
|
|
if record is None or not digest_matches(record.get("token_digest"), token):
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
from app.gateway.deps import get_local_provider
|
|
|
|
user = await get_local_provider().get_user(str(record["user_id"]))
|
|
if user is None:
|
|
# The owning user was deleted or became unresolvable; the token is
|
|
# dead even though its row survives (deleting a user revokes their
|
|
# PATs, without needing a FK cascade).
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
await pat_repo.touch_last_used(str(record["id"]))
|
|
return user, frozenset(record.get("scopes") or ())
|
|
|
|
|
|
def validate_scopes(scopes: list[str]) -> list[str]:
|
|
"""Validate a creation-time scope list; returns the deduplicated order."""
|
|
unknown = sorted(set(scopes) - PAT_ALLOWED_SCOPES)
|
|
if unknown:
|
|
raise ValueError(f"Unknown PAT scopes: {', '.join(unknown)}")
|
|
deduplicated = sorted(set(scopes))
|
|
if not deduplicated:
|
|
raise ValueError("A PAT must request at least one scope")
|
|
return deduplicated
|