Ryker_Feng 063d62c3c3
feat(persistence): support custom postgres schema (#3442)
* feat(persistence): support custom postgres schema

* fix(persistence): address CI lint/test failures and review feedback

- Map missing psycopg import to actionable POSTGRES_INSTALL guidance in
  sync/async schema-creation helpers
- Accept SQLAlchemy compound DSN schemes (postgresql+asyncpg) when
  injecting search_path, normalizing to a libpq-consumable DSN
- Guard keyword-DSN tests with importorskip so they skip without psycopg
- Set database=None in sync checkpointer none-fix test to avoid MagicMock
  backend resolution
- Apply ruff import sort and format

* fix(persistence): address pg-schema review feedback

- Restrict postgres_schema regex to lowercase-only so the quoted CREATE
  SCHEMA matches the unquoted search_path (PG case-folds it), fixing the
  mixed-case bug where tables silently fell back to public.
- Replace shlex.join/split with libpq-correct backslash escaping for the
  options parameter so values containing spaces survive intact.
- Add normalize_libpq_dsn() and route the async checkpointer pool through
  dsn_with_search_path() so a +asyncpg suffix is stripped and existing DSN
  options (e.g. statement_timeout) are merged instead of overridden.
- Extract shared ensure_postgres_schema()/ensure_postgres_schema_async()
  helpers (mapping missing psycopg to the install hint) used by all four
  provider sites.
- Tests: reject mixed-case schemas, preserve space-containing libpq option,
  cover normalize_libpq_dsn, and assert pool search_path via DSN.

* fix(persistence): align pg-schema test with merged store API

The main merge moved the sync Store factory to the single-path
_resolve_store_config/_sync_store_cm design, dropping the PR's
_sync_store_from_database helper. The integration test still imported
the removed symbol, breaking test collection (backend-unit-tests).
Resolve the store config from a DatabaseConfig and drive it through
_sync_store_cm instead.

* fix(persistence): address pg-schema review feedback

- reject trailing/leading whitespace in postgres_schema via re.fullmatch
  (a $-anchored re.match let "deerflow\n" through, silently landing tables
  in public)
- re-escape all whitespace (TAB/CR/LF) when re-joining libpq options so a
  caller's pre-existing options value round-trips losslessly
- re-validate the identifier inside create_schema_sql as defense-in-depth
  at the SQL-emitting boundary
- accept the postgres:// short scheme in the alembic search_path injection
- close the sync psycopg connection explicitly (psycopg3 __exit__ does not
  close()), mirroring the async path
- drop the partial checkpointer/store reset on a database config change;
  database is restart-required and the ORM engine is not rebuilt, so a
  partial reset would half-migrate the deployment

* docs(config): complete the postgres_schema migration checklist

Address PR review (P1): the documented `public`->schema migration only
moved runs, run_events, threads_meta, feedback, and users. That strands
every other DeerFlow-owned table -- the four channel_* tables, both
scheduled_* tables, agents, and (critically) alembic_version -- in
`public`. On restart bootstrap treats the partially-populated target
schema as unversioned, re-baselines it, and replays migrations while the
real rows stay invisible in `public`.

List the full owned set explicitly, call out alembic_version as required,
and keep the "discover the rest" query for version-drift safety.

* refactor(checkpointer): drop test-only _sync_checkpointer_from_database

Address PR review: the helper was only reached by the env-gated
integration test and re-implemented the DatabaseConfig->CheckpointerConfig
backend resolution that _resolve_checkpointer_config already owns, so a
future backend added there would silently miss this path. Mirror the store
side of the same test, which reuses the production path directly:
_resolve_checkpointer_config(...) + _sync_checkpointer_cm(...).
2026-07-30 13:51:17 +08:00

125 lines
4.5 KiB
Python

"""Async Store factory — backend mirrors runtime persistence configuration.
The deprecated ``checkpointer`` section takes precedence when present;
otherwise Store follows the unified ``database`` section in *config.yaml*:
- ``memory`` → :class:`langgraph.store.memory.InMemoryStore`
- ``sqlite`` → :class:`langgraph.store.sqlite.aio.AsyncSqliteStore`
- ``postgres`` → :class:`langgraph.store.postgres.aio.AsyncPostgresStore`
Usage (e.g. FastAPI lifespan)::
from deerflow.runtime.store import make_store
async with make_store() as store:
app.state.store = store
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from collections.abc import AsyncIterator
from langgraph.store.base import BaseStore
from deerflow.config.app_config import AppConfig, get_app_config
from deerflow.persistence.postgres_schema import dsn_with_search_path, ensure_postgres_schema_async
from deerflow.runtime.store.provider import (
POSTGRES_CONN_REQUIRED,
POSTGRES_STORE_INSTALL,
SQLITE_STORE_INSTALL,
_resolve_store_config,
ensure_sqlite_parent_dir,
resolve_sqlite_conn_str,
)
logger = logging.getLogger(__name__)
async def _ensure_postgres_schema(conn_string: str, schema: str) -> None:
"""Create the configured schema before LangGraph creates its store tables."""
await ensure_postgres_schema_async(conn_string, schema, install_hint=POSTGRES_STORE_INSTALL)
# ---------------------------------------------------------------------------
# Internal backend factory
# ---------------------------------------------------------------------------
@contextlib.asynccontextmanager
async def _async_store(config) -> AsyncIterator[BaseStore]:
"""Async context manager that constructs and tears down a Store.
The ``config`` argument is a :class:`deerflow.config.checkpointer_config.CheckpointerConfig`
instance — the same object used by the checkpointer factory.
"""
if config.type == "memory":
from langgraph.store.memory import InMemoryStore
logger.info("Store: using InMemoryStore (in-process, not persistent)")
yield InMemoryStore()
return
if config.type == "sqlite":
try:
from langgraph.store.sqlite.aio import AsyncSqliteStore
except ImportError as exc:
raise ImportError(SQLITE_STORE_INSTALL) from exc
conn_str = resolve_sqlite_conn_str(config.connection_string or "store.db")
await asyncio.to_thread(ensure_sqlite_parent_dir, conn_str)
async with AsyncSqliteStore.from_conn_string(conn_str) as store:
await store.setup()
logger.info("Store: using AsyncSqliteStore (%s)", conn_str)
yield store
return
if config.type == "postgres":
try:
from langgraph.store.postgres.aio import AsyncPostgresStore # type: ignore[import]
except ImportError as exc:
raise ImportError(POSTGRES_STORE_INSTALL) from exc
if not config.connection_string:
raise ValueError(POSTGRES_CONN_REQUIRED)
await _ensure_postgres_schema(config.connection_string, config.postgres_schema)
conn_string = dsn_with_search_path(config.connection_string, config.postgres_schema)
async with AsyncPostgresStore.from_conn_string(conn_string) as store:
await store.setup()
logger.info("Store: using AsyncPostgresStore")
yield store
return
raise ValueError(f"Unknown store backend type: {config.type!r}")
# ---------------------------------------------------------------------------
# Public async context manager
# ---------------------------------------------------------------------------
@contextlib.asynccontextmanager
async def make_store(app_config: AppConfig | None = None) -> AsyncIterator[BaseStore]:
"""Yield a Store selected from legacy or unified persistence config.
The legacy ``checkpointer`` section takes precedence when configured;
otherwise the unified ``database`` section selects the backend, matching
:func:`deerflow.runtime.checkpointer.async_provider.make_checkpointer`::
async with make_store(app_config) as store:
app.state.store = store
An :class:`~langgraph.store.memory.InMemoryStore` is returned only when the
resolved backend is explicitly ``memory``.
"""
if app_config is None:
app_config = get_app_config()
config = _resolve_store_config(app_config)
async with _async_store(config) as store:
yield store