mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-14 16:08:41 +00:00
* feat(projects): project workspaces with scoped chats and thread membership
Backend:
- projects table model and migration; fail-closed ProjectRepository with
ownership checks, CRUD/archive/restore/delete router, and atomic thread
move between projects
- threads_meta.project_id column exposed as reserved deerflow_project_id
metadata; project-aware thread create/search with pagination bounds and
membership echoed in create responses
- first-run admission assigns the project only at genuine first run, seeded
at write time and dropped when invalid; serialized against project
deletion and thread assignment
- branch creation inherits the source thread's project membership (an
archived/deleted project degrades the branch to unassigned instead of
failing the request)
Frontend:
- projects data layer, thread move API, and sidebar projects section with
flat/grouped modes, archived-project threads, and stable virtual-list
offsets
- project detail page with project-scoped new chat
(/workspace/chats/new?project=) and paginated thread list
- move-to-project thread menu, new-project dialog, archived-project gates
- project-scoped new chats pre-create the thread with membership before the
first submit or /goal set, so runs never proceed outside the project
- goal-set preparation is fenced against conversation switches: a stale
continuation is dropped instead of saving the goal or launching the
abandoned submission on the newly opened conversation
- project thread lists join thread lifecycle invalidations (stop, pin) so
an open project page never keeps stale titles, recency, or pagination
* fix(chats): keep archive undo toast when the sidebar row unmounts
The archive success toast was fired from per-mutate callbacks passed to
mutation.mutate. React Query drops those handlers when the observer
component unmounts before the mutation settles; archiving the open chat
removes its sidebar row mid-flight, so the undo toast never appeared and
the e2e archive-undo test timed out waiting for it.
Move the success/error handlers to the mutation level (useArchiveThread
options, same pattern as useMoveThreadToProject) where callbacks are
delivered even after the originating row unmounts.
* fix(projects): pin project thread listing contract and exclude archived chats
GET /api/projects/{id}/threads returned the thread store row verbatim
(list[dict], no response_model): user_id/assistant_id leaked, any future
ThreadMetaRow column would auto-leak, and the OpenAPI schema was empty.
Return a narrow ProjectThreadResponse (the exact fields ProjectThread
declares) with the same metadata secret redaction the surrounding thread
endpoints get from _MetadataRedactingResponse.
The listing also ran search() without the archived filter, so a retired
chat rendered as a normal row on the project page while the sidebar hid
it. Search archived=False to mirror the sidebar's archived:false lists;
restore stays on the global Archived tab.
Both regressions pinned by new router tests: wire-shape allowlist and
archived-member exclusion.
* docs(migrations): record the 0019/0020 chain against the bootstrap reservation
The tree now chains 0018 -> 0019_projects -> 0020_threads_meta_project_id,
so migrations/AGENTS.md was stale twice over: the revision index stopped at
0018 and the rolling-forward section still claimed the tree 'deliberately
remains at 0018'.
Document the new head and record the intentional numeric-prefix reuse of
0019: 0019_projects is in-chain while 0019_thread_incarnations stays the
reserved, allowlisted out-of-tree rollout id. The owning rollout revision
must re-parent onto this tree's head when it merges so alembic never sees
two heads off 0018; bootstrap.py now cross-references that note next to
_FORWARD_COMPATIBLE_REVISION.
* fix(chats): invalidate project thread lists on archive/restore
useArchiveThread refreshed the infinite sidebar cache, threads/search and
the per-thread metadata cache but not the project-scoped list
([...PROJECTS_QUERY_KEY, 'threads', id]) this PR adds — the one thread
mutation not wired to that key, after usePinThread, useRenameThread,
useDeleteThread, useMoveThreadToProject and invalidateStoppedThreadCaches.
An archive from a sidebar row while a project page is open therefore left
the archived chat rendered as a normal row until remount (and undo left it
missing). Invalidate the prefix in the mutation-level success handler.
Regression test asserts the project-list prefix is invalidated on success.
* fix(projects): fetch project discovery only in grouped sidebar mode
RecentChatList mounted two useProjects queries per sidebar render, but
knownProjectIds is consumed only by the grouped-mode exclusion filter; in
the default flat mode every page load paid two GET /api/projects?status=
round trips for data nothing read. Gate both queries on grouped mode —
GroupedProjectList fetches the same keys when the toggle is on and
TanStack dedupes the observers.
Also set retry: false on useProject: a deleted or foreign project 404s
deterministically, and the page renders a dedicated not-found state for
it, so the default 1s/2s/4s retry backoff kept deep links in 'loading'
for ~7s before that state appeared. Matches useThreadMetadata /
useThreadTokenUsage.
* fix(threads): fail closed on project-scoped create in memory mode
MemoryThreadMetaStore.create accepted project_id and silently ignored it,
making memory mode the one membership path that fails open: POST
/api/threads with a project id returned 200 and the run started
unassigned, violating the invariant that a run never proceeds outside the
selected project (the SQL store raises ProjectNotAssignableError inside
the insert transaction for the same request).
Raise ProjectNotAssignableError whenever project_id is present so the
router's existing 404 mapping applies, the frontend keeps the composer
text for a retry, and memory mode behaves exactly like SQL mode.
set_project already reports rejection; create now matches it.
Store-level test (raises, nothing persisted, project filter stays empty,
unscoped creates still work) plus a router-level test asserting the 404
and that no row is left behind.
* fix(projects): window the project page thread list
ProjectThreadsSection rendered every loaded page as a plain Link row, so a
long-lived project accumulated unbounded DOM on the page's scroll surface:
each load-more appended another 100 rows and every formatTimeAgo tick
re-rendered the whole list.
Reuse VirtualThreadList (now generic over any row shape with a
thread_id), pointing its scroll parent at this page's ScrollArea viewport
via the shared [data-slot="scroll-area-viewport"] selector used by
/workspace/chats; under the 60-row threshold it falls back to the plain
render, so small projects are unchanged.
* fix(projects): restore row dividers and pin them with a render test
The row class template literal concatenated transition-colors directly
with the conditional border-b token, so non-final rows rendered the
invalid class 'transition-colorsborder-b' and lost both the divider and
the transition. Compose the row classes with cn() and a boolean guard
instead.
The section moved out of page.tsx into a testable component so the row
markup finally has coverage: a DOM test asserts every row except the
final data row carries border-b (index-based, not last: — correct under
virtualization where the last mounted row is not the last data row), and
the untitled fallback plus load-more button render for a partial page.
* fix(projects): validate forward schemas and fence membership reads
211 lines
9.1 KiB
Python
211 lines
9.1 KiB
Python
"""Personal Access Token (PAT) credentials for programmatic API access.
|
|
|
|
Tokens are ``dfp_`` + base62(32 CSPRNG bytes), shown exactly once in the
|
|
create response and persisted only as a SHA-256 digest. Validation is a
|
|
digest-indexed lookup plus a constant-time re-comparison, with a single
|
|
generic failure surface so a 401 never reveals which check failed.
|
|
|
|
v1 scopes are exactly the route-permission strings owned by
|
|
``app.gateway.authz`` — a PAT can only narrow its owning user's
|
|
permissions, never widen them.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import functools
|
|
import hashlib
|
|
import hmac
|
|
import re
|
|
import secrets
|
|
from typing import Any
|
|
|
|
PAT_TOKEN_PREFIX = "dfp_"
|
|
PAT_RANDOM_BYTES = 32
|
|
# Best-effort ``last_used_at`` writes are throttled per token so high-volume
|
|
# automation does not turn every request into a database write.
|
|
PAT_LAST_USED_WRITE_INTERVAL_SECONDS = 300.0
|
|
|
|
PAT_ALLOWED_SCOPES: frozenset[str] = frozenset(
|
|
{
|
|
"threads:read",
|
|
"threads:write",
|
|
"threads:delete",
|
|
"runs:create",
|
|
"runs:read",
|
|
"runs:cancel",
|
|
"projects:read",
|
|
"projects:write",
|
|
"projects:delete",
|
|
}
|
|
)
|
|
|
|
PAT_MAX_NAME_LENGTH = 128
|
|
|
|
# Default-deny route boundary for PAT callers (#5041 review P1-1): scope
|
|
# intersection in AuthMiddleware only constrains routes that consult
|
|
# ``request.state.auth.permissions`` (``@require_permission``). Authenticated
|
|
# mutation routes without that decorator — memory deletion, agent creation,
|
|
# credential switching, channel configuration — would otherwise accept a
|
|
# PAT holding a single read scope. A route is reachable by PAT only when it
|
|
# is explicitly listed here, and only together with the thread/run lifecycle
|
|
# the v1 scopes govern; everything else answers 403 regardless of scopes.
|
|
_PAT_ROUTE_RULES: tuple[tuple[frozenset[str], re.Pattern[str]], ...] = (
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/search$")),
|
|
(frozenset({"GET", "PATCH", "DELETE"}), re.compile(r"^/api/threads/[^/]+$")),
|
|
(frozenset({"GET", "PUT", "DELETE"}), re.compile(r"^/api/threads/[^/]+/goal$")),
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/state$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/[^/]+/(compact|history|branches)$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/[^/]+/move$")),
|
|
# Projects subtree: same enumerated-no-dead-methods precision as the
|
|
# threads/runs rules — only the methods the projects router implements
|
|
# are admitted, so a future projects route is default-denied until
|
|
# explicitly listed. Scope narrowing (projects:read|write|delete and
|
|
# threads:write for move) stays enforced by ``@require_permission``.
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/projects$")),
|
|
(frozenset({"GET", "PATCH", "DELETE"}), re.compile(r"^/api/projects/[^/]+$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/projects/[^/]+/(archive|restore)$")),
|
|
(frozenset({"GET"}), re.compile(r"^/api/projects/[^/]+/threads$")),
|
|
# Runs subtree: enumerated per implemented subroute instead of a
|
|
# ``runs(/.*)?`` wildcard, so a route added under /runs is default-denied
|
|
# until explicitly listed — the same no-dead-methods precision the
|
|
# threads collection rule enforces. The ``{run_id}`` slot necessarily
|
|
# matches any single segment; the POST-only collection endpoints sharing
|
|
# that depth (stream, wait, regenerate, edit-regenerate) are excluded
|
|
# from the GET run-id rule so no unimplemented method is pre-authorized.
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/runs$")),
|
|
(
|
|
frozenset({"POST"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/(stream|wait|regenerate/prepare|edit-regenerate/prepare)$"),
|
|
),
|
|
(
|
|
frozenset({"GET"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/(?!stream$|wait$|regenerate$|edit-regenerate$)[^/]+$"),
|
|
),
|
|
(frozenset({"POST"}), re.compile(r"^/api/threads/[^/]+/runs/[^/]+/cancel$")),
|
|
(
|
|
frozenset({"GET"}),
|
|
re.compile(r"^/api/threads/[^/]+/runs/[^/]+/(join|messages|events|workspace-changes)$"),
|
|
),
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/runs/[^/]+/artifacts/archive$")),
|
|
(frozenset({"GET", "POST"}), re.compile(r"^/api/threads/[^/]+/runs/[^/]+/stream$")),
|
|
(frozenset({"POST"}), re.compile(r"^/api/runs/(stream|wait)$")),
|
|
(frozenset({"GET"}), re.compile(r"^/api/runs/[^/]+/(messages|feedback)$")),
|
|
)
|
|
|
|
_BASE62_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
def is_pat_allowed_route(method: str, path: str) -> bool:
|
|
"""Return whether the PAT route policy admits *method* + *path*.
|
|
|
|
Trailing slashes are normalized away so the mounted route and its
|
|
redirect-style twin resolve identically.
|
|
"""
|
|
normalized = path.rstrip("/") or "/"
|
|
return any(method in methods and pattern.match(normalized) for methods, pattern in _PAT_ROUTE_RULES)
|
|
|
|
|
|
@functools.cache
|
|
def _base62_width(byte_length: int) -> int:
|
|
"""Digits sufficient for any *byte_length*-byte value (exact integer math)."""
|
|
width = 1
|
|
limit = 1 << (8 * byte_length)
|
|
while 62**width < limit:
|
|
width += 1
|
|
return width
|
|
|
|
|
|
def _base62(data: bytes) -> str:
|
|
"""Fixed-width big-endian base62 of *data*, ``0``-padded on the left.
|
|
|
|
``int.from_bytes`` discards leading zero bytes, so an unpadded encoding
|
|
would be variable-length (and empty for all-zero input) — any draw below
|
|
62**39 would have produced a shorter-than-expected token. The fixed width
|
|
keeps every token body exactly ``_base62_width(len(data))`` characters
|
|
and makes the format test deterministic.
|
|
"""
|
|
value = int.from_bytes(data, "big")
|
|
digits: list[str] = []
|
|
while value:
|
|
value, remainder = divmod(value, 62)
|
|
digits.append(_BASE62_ALPHABET[remainder])
|
|
body = "".join(reversed(digits))
|
|
return body.rjust(_base62_width(len(data)), "0")
|
|
|
|
|
|
def generate_pat_token() -> str:
|
|
"""Generate a show-once raw token: ``dfp_`` + base62(CSPRNG bytes)."""
|
|
return PAT_TOKEN_PREFIX + _base62(secrets.token_bytes(PAT_RANDOM_BYTES))
|
|
|
|
|
|
def pat_token_digest(token: str) -> str:
|
|
"""Return the hex SHA-256 digest persisted for *token*."""
|
|
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
|
|
|
|
|
def digest_matches(stored_digest: str | None, token: str) -> bool:
|
|
"""Constant-time comparison of *token* against a stored digest."""
|
|
if not isinstance(stored_digest, str) or not stored_digest:
|
|
return False
|
|
return hmac.compare_digest(stored_digest, pat_token_digest(token))
|
|
|
|
|
|
def extract_bearer_token(authorization: str | None) -> str | None:
|
|
"""Return the Bearer credential from an Authorization header value.
|
|
|
|
``None`` means the request carries no Authorization header at all, so the
|
|
caller should fall through to the session-cookie path. Any other unusable
|
|
value (non-Bearer scheme, empty credential) returns ``""`` so callers
|
|
treat it as an invalid credential rather than an absent one.
|
|
"""
|
|
if authorization is None:
|
|
return None
|
|
scheme, _, value = authorization.partition(" ")
|
|
if scheme.lower() != "bearer":
|
|
return ""
|
|
return value.strip()
|
|
|
|
|
|
async def authenticate_pat(app: Any, authorization: str | None) -> tuple[Any, frozenset[str]]:
|
|
"""Validate the Bearer credential and resolve its owning user.
|
|
|
|
Returns ``(user, scopes)``. Every token-verdict failure mode — malformed
|
|
token, unknown/revoked/expired token, PAT store not configured, missing
|
|
owning user — raises the same generic 401 so responses cannot serve as an
|
|
oracle on which check failed. Infrastructure errors (store I/O failures)
|
|
propagate and fail closed; they are not part of the token verdict.
|
|
"""
|
|
from fastapi import HTTPException
|
|
|
|
token = extract_bearer_token(authorization)
|
|
if not token or not token.startswith(PAT_TOKEN_PREFIX):
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
pat_repo = getattr(app.state, "pat_repo", None)
|
|
if pat_repo is None:
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
record = await pat_repo.get_active_by_digest(pat_token_digest(token))
|
|
if record is None or not digest_matches(record.get("token_digest"), token):
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
from app.gateway.deps import get_local_provider
|
|
|
|
user = await get_local_provider().get_user(str(record["user_id"]))
|
|
if user is None:
|
|
# The owning user was deleted or became unresolvable; the token is
|
|
# dead even though its row survives (deleting a user revokes their
|
|
# PATs, without needing a FK cascade).
|
|
raise HTTPException(status_code=401, detail="Invalid token")
|
|
await pat_repo.touch_last_used(str(record["id"]))
|
|
return user, frozenset(record.get("scopes") or ())
|
|
|
|
|
|
def validate_scopes(scopes: list[str]) -> list[str]:
|
|
"""Validate a creation-time scope list; returns the deduplicated order."""
|
|
unknown = sorted(set(scopes) - PAT_ALLOWED_SCOPES)
|
|
if unknown:
|
|
raise ValueError(f"Unknown PAT scopes: {', '.join(unknown)}")
|
|
deduplicated = sorted(set(scopes))
|
|
if not deduplicated:
|
|
raise ValueError("A PAT must request at least one scope")
|
|
return deduplicated
|