deer-flow/backend/packages/harness/deerflow/persistence/migrations/AGENTS.md

Schema Migrations (packages/harness/deerflow/persistence/migrations/)

DeerFlow's application tables (runs, threads_meta, feedback, users, run_events, plus the four channel_* tables) are owned by alembic via a hybrid bootstrap strategy. LangGraph's checkpointer tables (checkpoints, checkpoint_blobs, checkpoint_writes, checkpoint_migrations) live in the same database but are owned by LangGraph and excluded from alembic's view via migrations/_env_filters.py::include_object.

Convention: every ORM model change (new column, new table, new index) MUST ship as an alembic revision under migrations/versions/. The Gateway runs alembic upgrade head automatically on startup; users do not run alembic manually in production.

Hybrid bootstrap (persistence/bootstrap.py::bootstrap_schema, invoked from persistence/engine.py::init_engine):

DB state	Action
empty (no DeerFlow tables)	create_all + alembic stamp head
legacy (DeerFlow tables, no alembic_version)	create_all (baseline tables only, backfill) + alembic stamp 0001_baseline + upgrade head
versioned (alembic_version row exists)	alembic upgrade head

The legacy branch handles pre-alembic databases that already have at least one DeerFlow-owned table. create_all runs first because stamping at 0001_baseline makes alembic skip the baseline's own create_table DDL on the subsequent upgrade — so any baseline table introduced into Base.metadata after the user's DB was first provisioned (e.g. the channel_* tables from PR #1930 for users upgrading across multiple releases) would otherwise never be created, and the first request hitting that table would 500 with no such table. The backfill is restricted to _BASELINE_TABLE_NAMES so it does not also create tables that future revisions introduce — those revisions' own op.create_table would otherwise fail with relation already exists. A guard test pins _BASELINE_TABLE_NAMES against 0001_baseline.upgrade()'s actual output, so editing 0001 to add or remove a table forces a matching update to the constant. Column-level shape (pre-#3658 vs post-#3658 vs manual-ALTER for token_usage_by_model) is answered by each versions/*.py revision via the idempotent helpers in migrations/_helpers.py (safe_add_column / safe_drop_column) which no-op when the change is already present and logger.warning on shape drift. Adding a new ORM column / table only requires a new revision file — no edit to bootstrap.py is needed unless the new revision adds a new baseline table (rare; only happens when a new model is part of the baseline rather than introduced by its own revision).

The empty-DB path keeps using create_all because Base.metadata is the only authoritative schema source — create_all renders both SQLite (JSON, type affinity) and Postgres (JSONB, partial indexes) correctly without anyone having to keep a hand-written baseline in lockstep. 0001_baseline.upgrade() is therefore almost never executed in practice; it exists as a stamp target + chain root.

Concurrency safety: Postgres uses pg_advisory_lock to serialise concurrent Gateway instances. SQLite uses a per-engine asyncio.Lock for same-process startup and is best-effort across processes via SQLite's file-level write lock + PRAGMA busy_timeout; multi-instance deployments should use Postgres. Column revisions in versions/ additionally use idempotent helpers (_helpers.py::safe_add_column, safe_drop_column) so repeated post-baseline changes and retries are no-ops when the change is already present.

Authoring a new revision:

cd backend && make migrate-rev MSG="add foo column to runs"

This invokes alembic revision --autogenerate against the live ORM models. Review the generated file under migrations/versions/ and switch raw op.add_column / op.drop_column calls to the idempotent helpers from _helpers.py before committing. There is no make migrate / make migrate-stamp target on purpose — the only execution path is Gateway startup, which keeps operational mistakes off the table.

Where things live:

• migrations/env.py — alembic env, delegates filter to _env_filters.py, sets render_as_batch=True for SQLite ALTER support
• migrations/_env_filters.py::include_object — drops LangGraph checkpointer tables from alembic's view
• migrations/_helpers.py — safe_add_column / safe_drop_column
• migrations/versions/0001_baseline.py — chain root, matches the schema create_all produces from Base.metadata
• migrations/versions/0002_runs_token_usage.py — fixes issue #3682
• migrations/versions/0004_run_ownership.py — runs multi-worker ownership + the uq_runs_thread_active partial unique index, with a _dedupe_active_runs_per_thread() pre-step so CREATE UNIQUE INDEX cannot fail on a field DB that already has duplicate active rows per thread
• migrations/versions/0007_scheduled_run_active_index.py — the uq_scheduled_task_run_active partial unique index (at most one queued/running scheduled_task_runs row per task_id), with a _dedupe_active_scheduled_runs_per_task() pre-step (keeps the newest active row per task, supersedes the rest to interrupted with an explanatory error + finished_at) mirroring 0004; chains after 0006_agents
• migrations/versions/0008_thread_operation_kind.py — adds runs.operation_kind for durable non-run thread reservations; chains after 0007_scheduled_run_active_index
• migrations/versions/0010_run_cancel_request.py — adds the nullable runs.cancel_action / cancel_requested_at handoff used by non-owning workers; chains after 0009_webhook_dedupe
• persistence/bootstrap.py — bootstrap_schema(engine, backend=...), the three-branch decision + locking
• Tests: tests/test_persistence_bootstrap.py (branches), tests/test_persistence_bootstrap_concurrency.py (concurrency), tests/test_persistence_bootstrap_regression.py (issue #3682), tests/test_persistence_migrations_env.py (filter), tests/blocking_io/test_persistence_bootstrap.py (asyncio.to_thread anchor), tests/test_migration_0004_run_ownership_dedupe.py + tests/test_migration_0007_scheduled_run_active_dedupe.py (dedupe-before-unique-index pre-steps)