deer-flow/backend/tests/test_personal_access_tokens_repository.py
Sunshine bf740ffa90
feat(auth): add personal access tokens for programmatic API access (#5041)
* feat(auth): add personal access tokens for programmatic API access (#4849)

Backend-first implementation of the PAT contract from #4849: show-once
dfp_ tokens bound to their owning user (AUTH_SOURCE_PAT,
is_internal=false), digest-only storage (migration 0017), strict
credential precedence (invalid Bearer is a 401, never cookie fallback),
CSRF double-submit skipped only for Bearer requests while
auth-endpoint origin checks still run, scopes intersecting the authz
route permissions, session-auth-only PAT management and password
changes, and throttled best-effort last_used_at stamps.

* fix(auth): harden PAT scope boundary and schema parity from adversarial review

Independent review of the initial draft found: (1) scopes only constrained
the threads/runs permission axis while admin routes treated a PAT as its
(possibly admin) owner — is_admin_user now rejects PAT callers outright
since no scope grants admin capability; (2) the model declared a column
UNIQUE constraint while migration 0017 created a named unique index, so
downgrade failed on create_all-bootstrapped DBs — both now use the named
unique index; (3) auth-disabled mode is an operator override and now stays
ahead of the Bearer check so a stray Authorization header cannot 401 an
E2E sandbox; plus wiring the previously-unused constants, bounding the
last_used_at stamp cache, and four new tests (middleware-level expiry,
expires_in_days, admin-capability rejection with session control, and the
auth-disabled precedence).

* docs(api): document personal access tokens for programmatic API access

* fix(auth): close PAT security boundaries from review (default-deny routes, extension admin suppression)

P1-1: scope intersection only constrains @require_permission routes, so
undecorated mutation routes (DELETE /api/memory, POST /api/agents, Lark
credential switching, channel config) accepted a PAT holding a single read
scope. AuthMiddleware now enforces a default-deny route policy in
auth/pat.py: PAT requests are admitted only to the thread/run lifecycle
routes the v1 scopes govern; everything else answers 403 regardless of
scopes. Session-cookie callers are unaffected.

P1-2: the extension principal resolver projected is_admin/roles from the
raw system_role, so an admin-owned PAT passed
deerflow_extension_api.require_admin on contributed routes despite the
documented no-admin guarantee. The projection is now PAT-aware and
suppresses every admin signal for PAT callers, mirroring
deps.is_admin_user.

Both fixes carry regression tests (route outside policy 403 + session
control; production resolver admin suppression), and API.md documents the
default-deny boundary.

* fix(auth): enforce PAT scopes on stateless run entry and harden decorator

Follow-up hardening from an independent audit of the P1 fixes:

- POST /api/runs/stream and /api/runs/wait were the only allowlisted run
  entrypoints without @require_permission, so a threads:read-only PAT
  could still start runs (same bug class as P1-1, now closed): both now
  carry @require_permission("runs", "create"). POST /api/threads and
  POST /api/threads/search gain threads:write / threads:read for the
  same reason. Authorization-disabled deployments see no change (the
  permission set resolves to all permissions).
- require_permission now binds the wrapped signature to locate a
  positionally-passed request before injecting the test stub, fixing
  'got multiple values for argument' on direct positional unit-test
  calls.
- API.md: the intro PAT example used GET /api/models, which the new
  default-deny policy 403s — replaced with GET /api/threads; the
  default-deny route list now spells out method sets.

Regression test: threads:read-only PAT is 403 on the decorated stateless
entry while a runs:create PAT passes.

* fix(auth): address review P2s (empty Authorization header, PAT name trimming, API example)

- CSRFMiddleware treats an explicitly empty Authorization header as
  present (is None), so an invalid credential always reaches
  AuthMiddleware's uniform 401 instead of a CSRF 403 that varies by
  method/CSRF state. Regression: empty-header request dies at auth.
- PATCreateRequest strips the name and rejects whitespace-only values
  before token generation; created names are stored trimmed.
- API.md intro PAT example now uses the implemented
  POST /api/threads/search endpoint (GET /api/threads does not exist).
- AGENTS.md trimmed back under the guidance soft budget after the
  upstream merge.

* fix(auth): tighten PAT route policy to implemented methods only

The allowlist admitted GET /api/threads, a method no router implements.
Pre-authorizing a dead method weakens the default-deny boundary: a
future GET collection route added without a permission decorator would
become PAT-reachable without an explicit policy change. Restrict the
rule to POST, fix the stale GET description in API.md's PAT
constraints, and document the default-deny boundary accurately in the
gateway AGENTS.md guidance (only the threads/runs allowlist is
PAT-reachable; every other authenticated route 403s PAT callers).

Audited every remaining rule against the mounted routers: all other
method+path entries map to real routes. Regression:
test_pat_policy_does_not_pre_authorize_unimplemented_methods.

* test(auth): guarantee the negative digest test mutates the token

token[:-1] + "X" is identical to the original whenever the generated
token already ends in X (1/62), making the negative digest assertion
fail intermittently. Choose the replacement character based on the
existing tail so the mutated token always differs.

* fix(auth): require runs:cancel for cancel-then-stream requests

stream_existing_run is gated at runs:read so action-less stream joins
work with read-only credentials, but its ?action=interrupt|rollback
branch cancels the run — a separate permission. A runs:read-only PAT
passed both the PAT route policy and the route decorator and could
interrupt or roll back an active run, bypassing the runs:cancel scope.

Decorators cannot express query-parameter-conditional permissions, so
the check lives in require_cancel_permission_when_action(), applied at
the top of the handler. Regression drives the real helper through the
production middleware: runs:read-only PAT + action is 403, the same
token joins action-less, runs:read+cancel passes, session control
unaffected.

* docs(changelog): add the PAT feature entry

* docs(readme): add personal access tokens section

Repo documentation-update policy requires user-facing features to
update README.md in the same changeset; the PAT feature previously
touched only backend/docs/API.md and the gateway AGENTS.md.

* fix(auth): require runs:cancel for mutating multitask strategies

All five run-creation entrypoints were gated only by runs:create, but
RunCreateRequest.multitask_strategy accepts interrupt/rollback and
start_run forwards it to create_or_reject, which terminates an
already-active run. A runs:create-only PAT could therefore kill an
existing run through a create request, bypassing runs:cancel.

Decorators cannot express body-parameter-conditional permissions, and
per-route checks leave the same hole for the next entrypoint, so the
gate lives in start_run itself — the single choke point every
run-creation path (HTTP routes and internal launchers) flows through.
Regenerate launches pass multitask_strategy="reject" and are
unaffected; requests without a stamped auth context (internal/test
compositions) skip the gate.

The check is the shared authz.require_cancel_permission_if primitive;
require_cancel_permission_when_action now delegates to it, so every
request dimension that carries cancel capability (query action, body
strategy) flows through one gate.

Regression drives the real middleware stack: runs:create-only PAT +
interrupt/rollback is 403 with the exact detail, reject (explicit and
default) stays available, runs:create+cancel passes, session control
unaffected; a source anchor pins the gate inside start_run.

* fix(runs): keep observer joins from applying creator cancel-on-disconnect

sse_consumer's finally block applied the record's on_disconnect=cancel
policy on ANY consumer's disconnect. The join surfaces (GET /join and
the action-less GET/POST stream join) feed it the existing RunRecord,
so anyone with thread read access — including a runs:read-only PAT —
could cancel a locally-owned running run simply by closing the SSE
connection, without runs:cancel. The policy expresses the creator's
intent for their own connection; an observer's disconnect must never
be read as that intent.

sse_consumer gains apply_on_disconnect (default True). The two join
surfaces pass False; the creating endpoints (thread-scoped and
stateless create-and-stream) keep the creator semantics unchanged.
wait_for_run_completion needs no change: its callers are creator-side
or post-explicit-cancel paths only.

Regression exercises a real generator close — the same machinery
Starlette drives on client disconnect — against the production
sse_consumer: creator stream disconnect cancels, observer join
disconnect does not; a wiring anchor pins both join call sites and the
creator defaults. API.md documents the cancel-capability constraint
(this fix plus the action/strategy gates) in PAT Constraints.

* test(auth): pin the multitask gate behaviorally; state wait invariant

Independent adversarial review of the round-5 fixes found the P1-a
regression only mirror-pinned: the source anchor could be satisfied by
a comment, and deleting the gate from start_run would not fail the
suite. This drives the production start_run directly — a create-only
auth context gets 403 with the exact detail for interrupt, and a
reject request with no cancel permission at all proceeds past the gate
(never a permission 403).

Also documents wait_for_run_completion's creator-side invariant
(every caller is the creating endpoint or post-explicit-cancel) so a
future observer wiring thinks twice before reusing it — the one-caller-
away variant of the observer-disconnect P1.

* docs(changelog): correct the PAT entry's digest and route-policy description

The entry said HMAC digests (the implementation stores SHA-256 digests,
as documented in API.md and pinned by the repository tests) and claimed
the route policy admits 'implemented stateless endpoints' (it admits
the thread/run lifecycle routes, narrowing further by scopes). Also
notes the cancel-capability gate now covering action and multitask
strategies.

* fix(auth): enumerate the PAT runs route policy per implemented subroute

The runs subtree rule was a GET|POST /runs(/.*)? wildcard — it
pre-authorized every current and future subroute under /runs, including
methods the router never implemented (e.g. GET /runs/stream), which is
the same latent default-deny weakening the threads collection rule was
tightened for: a future route added under /runs would become
PAT-reachable without an explicit policy change.

The wildcard is replaced with six segment-precise rules covering exactly
the 14 implemented method+path combinations; the {run_id} slot
necessarily matches any single segment, so the POST-only collection
names (stream, wait, regenerate, edit-regenerate) are excluded from the
GET run-id rule via negative lookahead — no dead method stays
pre-authorized. Behavior for implemented routes is unchanged.

test_pat_runs_policy_admits_exactly_the_mounted_routes derives the
expected set from the mounted thread_runs router instead of a
hand-maintained list: every implemented GET/POST route under /runs must
be admitted, routes in this router outside the subtree stay denied, and
representative unimplemented neighbors are denied — so adding a route
under /runs now fails CI until it is explicitly allowlisted, and a
removed route leaves a dead rule visible. API.md's PAT constraints list
the enumerated routes and drops a feedback mention that belonged to the
stateless /api/runs axis.

* docs(migration): add the 0017 renumbering coordination note to 0017

The PR's migration-coordination comment states each migration file
carries the note; the file did not. Adds it: numbering was generated
against main head 0016 alongside #5078 and #4843; whoever merges first
keeps the slot, the others renumber on rebase (revision/down_revision
plus the bootstrap head assertions).

* fix(auth): pad base62 tokens to a fixed 43-char width

int.from_bytes discards leading zero bytes, so the unpadded encoder
returned a variable-length body — empty for all-zero input, and shorter
than 40 characters for any draw below 62**39 (~1 in 14.5M), leaving
test_generate_pat_token_format probabilistically flaky and the token
body without stable width (review round 6, P3).

_base62 now left-pads with "0" to _base62_width(len(data)) — the exact
integer digit count (62^43 > 2^256 > 62^42, so 43 for 32 bytes). The
format test asserts the exact fixed width instead of a probabilistic
floor, and a new unit test pins the all-zero, leading-zero-byte, and
max-value edges deterministically.
2026-08-29 23:50:45 +08:00

210 lines
8.6 KiB
Python

"""Tests for PAT token utilities and the personal access token repository."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import pytest
import pytest_asyncio
from app.gateway.auth.pat import (
PAT_ALLOWED_SCOPES,
PAT_TOKEN_PREFIX,
digest_matches,
extract_bearer_token,
generate_pat_token,
pat_token_digest,
validate_scopes,
)
from deerflow.config.database_config import DatabaseConfig
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
from deerflow.persistence.personal_access_tokens import PersonalAccessTokenRepository
@pytest_asyncio.fixture(autouse=True)
async def _close_persistence_engine():
yield
await close_engine()
async def _make_repo(tmp_path) -> PersonalAccessTokenRepository:
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
session_factory = get_session_factory()
assert session_factory is not None
return PersonalAccessTokenRepository(session_factory)
# ── Token utilities ───────────────────────────────────────────────────────
def test_generate_pat_token_format():
token = generate_pat_token()
assert token.startswith(PAT_TOKEN_PREFIX)
body = token[len(PAT_TOKEN_PREFIX) :]
assert len(body) == 43 # fixed width: 62^43 > 2^256 > 62^42
assert body.isalnum()
# Two draws must differ: CSPRNG, not a counter.
assert token != generate_pat_token()
def test_base62_pads_to_fixed_width_for_leading_zero_and_all_zero_input():
"""``int.from_bytes`` discards leading zero bytes; the fixed-width pad
keeps the token body exactly 43 chars for every draw, including the
all-zero and single-leading-byte edges (review round 6, P3)."""
from app.gateway.auth.pat import PAT_RANDOM_BYTES, _base62
assert _base62(b"\x00" * PAT_RANDOM_BYTES) == "0" * 43
assert _base62(b"\x00" * (PAT_RANDOM_BYTES - 1) + b"\x01") == "0" * 42 + "1"
assert len(_base62(b"\xff" * PAT_RANDOM_BYTES)) == 43
def test_pat_token_digest_is_deterministic_and_constant_time_comparable():
token = generate_pat_token()
assert pat_token_digest(token) == pat_token_digest(token)
assert len(pat_token_digest(token)) == 64
assert digest_matches(pat_token_digest(token), token) is True
# The mutated token must differ from the original even when the CSPRNG
# tail already ends in "X" (1/62), or this assertion fails intermittently.
mutated_tail = "X" if token[-1] != "X" else "Y"
assert digest_matches(pat_token_digest(token), token[:-1] + mutated_tail) is False
assert digest_matches(None, token) is False
assert digest_matches("", token) is False
def test_extract_bearer_token_classifies_absent_scheme_and_credential():
assert extract_bearer_token(None) is None
assert extract_bearer_token("Bearer dfp_abc") == "dfp_abc"
assert extract_bearer_token("bearer dfp_abc") == "dfp_abc" # scheme is case-insensitive
assert extract_bearer_token("Basic dXNlcg==") == "" # present but unusable
assert extract_bearer_token("Bearer ") == ""
assert extract_bearer_token("Bearer") == ""
def test_validate_scopes_deduplicates_and_rejects_unknown():
assert validate_scopes(["runs:read", "threads:read", "runs:read"]) == ["runs:read", "threads:read"]
with pytest.raises(ValueError, match="Unknown PAT scopes"):
validate_scopes(["runs:write"]) # not a route permission
with pytest.raises(ValueError, match="at least one scope"):
validate_scopes([])
def test_pat_scopes_stay_aligned_with_route_permissions():
"""PAT scopes are exactly the authz route permissions — fail on drift."""
from app.gateway.authz import _ALL_PERMISSIONS
assert PAT_ALLOWED_SCOPES == frozenset(_ALL_PERMISSIONS)
# ── Repository ────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_create_and_resolve_by_digest_roundtrip(tmp_path):
repo = await _make_repo(tmp_path)
token = generate_pat_token()
record = await repo.create(
user_id="user-1",
name="ci-runner",
scopes=["runs:read"],
token_digest=pat_token_digest(token),
)
assert record["user_id"] == "user-1"
assert record["scopes"] == ["runs:read"]
assert record["revoked_at"] is None
resolved = await repo.get_active_by_digest(pat_token_digest(token))
assert resolved is not None
assert resolved["id"] == record["id"]
assert resolved["token_digest"] == pat_token_digest(token)
# Digest lookup never matches a different token.
assert await repo.get_active_by_digest(pat_token_digest(generate_pat_token())) is None
@pytest.mark.asyncio
async def test_revoked_token_no_longer_resolves(tmp_path):
repo = await _make_repo(tmp_path)
token = generate_pat_token()
record = await repo.create(user_id="user-1", name="temp", scopes=["runs:read"], token_digest=pat_token_digest(token))
assert await repo.revoke(record["id"], "user-1") is True
# Revoking twice is a no-op.
assert await repo.revoke(record["id"], "user-1") is False
assert await repo.get_active_by_digest(pat_token_digest(token)) is None
@pytest.mark.asyncio
async def test_revoke_is_scoped_to_the_owning_user(tmp_path):
repo = await _make_repo(tmp_path)
token = generate_pat_token()
record = await repo.create(user_id="user-1", name="mine", scopes=["runs:read"], token_digest=pat_token_digest(token))
assert await repo.revoke(record["id"], "user-2") is False # not the owner
assert await repo.get_active_by_digest(pat_token_digest(token)) is not None
@pytest.mark.asyncio
async def test_expired_token_no_longer_resolves(tmp_path):
repo = await _make_repo(tmp_path)
token = generate_pat_token()
await repo.create(
user_id="user-1",
name="short-lived",
scopes=["runs:read"],
token_digest=pat_token_digest(token),
expires_at=datetime.now(UTC) - timedelta(seconds=1),
)
assert await repo.get_active_by_digest(pat_token_digest(token)) is None
@pytest.mark.asyncio
async def test_list_for_user_is_isolated_and_never_returns_raw_tokens(tmp_path):
repo = await _make_repo(tmp_path)
token = generate_pat_token()
created = await repo.create(user_id="user-1", name="a", scopes=["runs:read"], token_digest=pat_token_digest(token))
await repo.create(user_id="user-2", name="b", scopes=["threads:read"], token_digest=pat_token_digest(generate_pat_token()))
listed = await repo.list_for_user("user-1")
assert [item["id"] for item in listed] == [created["id"]]
assert listed[0]["token_digest"] == pat_token_digest(token) # digest only; raw token never persisted
@pytest.mark.asyncio
async def test_token_digest_unique_constraint(tmp_path):
repo = await _make_repo(tmp_path)
digest = pat_token_digest(generate_pat_token())
await repo.create(user_id="user-1", name="a", scopes=["runs:read"], token_digest=digest)
from sqlalchemy.exc import IntegrityError
with pytest.raises(IntegrityError):
await repo.create(user_id="user-1", name="dup", scopes=["runs:read"], token_digest=digest)
@pytest.mark.asyncio
async def test_touch_last_used_is_throttled(tmp_path):
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
session_factory = get_session_factory()
repo = PersonalAccessTokenRepository(session_factory, last_used_write_interval_seconds=300.0)
record = await repo.create(user_id="user-1", name="t", scopes=["runs:read"], token_digest=pat_token_digest(generate_pat_token()))
await repo.touch_last_used(record["id"])
first = (await repo.list_for_user("user-1"))[0]["last_used_at"]
assert first is not None
# A second touch inside the throttle window must not produce a write.
await repo.touch_last_used(record["id"])
second = (await repo.list_for_user("user-1"))[0]["last_used_at"]
assert second == first
# After the window elapses the next touch writes again.
repo._last_used_written_at.clear()
await repo.touch_last_used(record["id"])
third = (await repo.list_for_user("user-1"))[0]["last_used_at"]
assert third != first
@pytest.mark.asyncio
async def test_touch_last_used_never_raises_on_unknown_id(tmp_path):
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
repo = PersonalAccessTokenRepository(get_session_factory())
await repo.touch_last_used("no-such-pat") # update affects 0 rows; still a commit