mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-08-15 01:08:53 +00:00
* feat(runs): cross-process run ownership with lease + reconciliation (#3948) Implements work items 2 and 3 of the multi-worker P0 plan (docs/multi_worker.md). Work item 1 (Postgres startup gate, #3960) already landed; this PR makes run creation race-safe across worker processes and lets Postgres deployments recover orphaned inflight runs from crashed workers without mis-marking live runs as orphans. Work item 2 — cross-process atomic create_or_reject - Alembic revision 0004_run_ownership adds runs.owner_worker_id, runs.lease_expires_at, idx_runs_lease, and a partial unique index uq_runs_thread_active (one pending/running run per thread). The index is declared on RunRow.__table_args__ with sqlite_where + postgresql_where (mirroring uq_channel_connection_active_identity) so the empty-DB bootstrap path — which runs Base.metadata.create_all + alembic stamp head without executing any revision's upgrade() — also lands it on fresh deployments. Migration 0004 additionally creates it idempotently for legacy/versioned upgrades. - RunRepository.create_run_atomic is the new atomic primitive: - reject: INSERT directly; the partial unique index catches duplicate active runs; the manager surfaces the result as ConflictError. - interrupt/rollback: SELECT FOR UPDATE the conflicting rows, skip rows whose lease is still valid AND owned by another live worker (raise ConflictError — the INSERT would have failed on the index anyway, and a retry loop cannot make progress), cancel the rest in the same transaction, then INSERT the new row. Rows owned by this worker are interruptible regardless of lease state. - RunManager.create_or_reject dispatches to the store under the existing local lock; same-worker in-memory cancellation runs after the store commit succeeds. MemoryRunStore mirrors the same semantics for tests and database.backend=memory. Work item 3 — lease heartbeat + Postgres reconciliation - RunOwnershipConfig (lease_seconds=30, grace_seconds=10, heartbeat_enabled=false by default), registered as startup-only in reload_boundary.STARTUP_ONLY_FIELDS because the heartbeat background task is created once in langgraph_runtime() and is not rebuilt on config.yaml edits. - When heartbeat_enabled, each worker renewes leases on its own active runs with interval = lease_seconds / 3. The loop is bounded and stop-event-cancellable so shutdown is prompt. - reconcile_orphaned_inflight_runs now runs on every backend — the sqlite-only gate in app/gateway/deps.py is dropped in the same commit so there is no window where Postgres would mis-mark live Worker A runs as orphans. Reconciliation errors only runs whose lease is NULL (legacy pre-ownership rows) or older than grace_seconds. In single-worker mode (heartbeat off, NULL leases) all inflight rows reclaim immediately, preserving the pre-ownership recovery latency. - Heartbeat starts AFTER startup reconciliation and stops BEFORE the in-flight run drain on shutdown so the two cannot race. GATEWAY_WORKERS=1 with heartbeat_enabled=false keeps current behavior. Verified: 170 related tests + full backend suite (minus Docker-gated live tests) green; ruff check + ruff format clean. * fix(runs): tighten unique-violation handling and document clock-sync budget Three follow-up fixes to the cross-process run ownership work in #3948, surfacing during review. 1. _is_unique_violation: detect by driver-native signal, not message text The previous substring heuristic ("unique" + "violat", or "duplicate") missed SQLite's actual phrasing "UNIQUE constraint failed: <table>.<index>" — SQLite says "failed", not "violates", and never "duplicate". On SQLite the detector returned False, the reject path re-raised the raw IntegrityError, and clients saw HTTP 500 instead of ConflictError 409. The conversion is the load-bearing piece of the "store is source of truth" design but was untested — every atomic test used MemoryRunStore, which raises ConflictError directly and never reached this branch. Now prefers driver-native signals: psycopg pgcode/sqlcode "23505" and sqlite3 sqlite_errorcode SQLITE_CONSTRAINT_UNIQUE (reachable through SQLAlchemy IntegrityError.orig). Message matching stays as a fallback with SQLite's exact "unique constraint failed" phrase added. 2. interrupt/rollback: convert exhausted-retry IntegrityError to ConflictError The reject branch converts unique violations to ConflictError. The interrupt/rollback retry loop did not — on the 3rd attempt it re-raised the raw IntegrityError, leaking HTTP 500 for the same race condition that reject surfaces as 409. Symmetric conversion added after the loop; callers now see a consistent ConflictError regardless of strategy. 3. Document clock-sync requirement for multi-worker lease reconciliation reconcile_orphaned_inflight_runs compares another worker's UTC lease_expires_at against this worker's datetime.now(UTC). The only skew budget is grace_seconds (default 10s) — worst case, with the owning worker's heartbeat just about to fire, a peer whose clock is more than ~grace_seconds ahead can mis-reclaim a still-live run as an orphan. Documented in RunOwnershipConfig's docstring (with the math) and in config.example.yaml (with operational guidance), so operators in NTP-poor environments know to raise grace_seconds. Default unchanged: 10s is reasonable for NTP-synced K8s/cloud, and bumping it would slow recovery of genuinely dead workers (lease_seconds + grace_seconds from last heartbeat to reclaim). Tests: - test_create_run_atomic_reject_propagates_conflict_on_unique_violation: end-to-end against a real SQLite-backed RunRepository, pre-inserts an active run, asserts reject-strategy create surfaces as ConflictError rather than raw IntegrityError. - test_is_unique_violation_detects_real_sqlite_integrity_error: unit test for the detector against a real SQLite-raised IntegrityError; asserts driver-level sqlite_errorcode is SQLITE_CONSTRAINT_UNIQUE. - test_interrupt_exhausted_retries_surface_as_conflict_error: pins the symmetric 409 behavior after the retry loop exhausts. Verified: ruff check + ruff format clean; multi-worker + run_repository + owner_isolation + reload_boundary suites green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): close multi-worker ownership gaps in lease heartbeat and unique-violation detection Five code-review fixes from docs/multi_worker.md: 1. Drop unused ``claim_inflight_runs`` primitive — no caller anywhere. ``create_run_atomic`` does its own inline claim (SELECT FOR UPDATE + cancel) inside the INSERT transaction; a separate claim primitive would split that into two transactions and open a claim→INSERT race. Removes ~40 lines across base.py / memory.py / sql.py plus the unused ``now_iso`` parameter, freeing future RunStore implementations from providing it. 2. Broaden ``_renew_leases`` filter to renew pending/running runs owned by this worker even when ``record.task is None``. The previous ``task is not None`` requirement skipped the brief window between ``create_run_atomic`` inserting the row and the worker spawning the agent task; under event-loop load that window can approach ``lease_seconds``, after which peer reconciliation marks the run ``error`` (visible) or a peer's ``create_or_reject("interrupt")`` silently kills the queued run. Filter now: ``task is None or not task.done()``. 3. Document the unsynchronised ``record.lease_expires_at = new_expiry`` write. ``lease_expires_at`` is the only field on an existing record this path mutates; ``set_status`` / ``_persist_status`` touch other fields, so there is no concurrent writer to race against. Re-acquiring ``self._lock`` would serialise unrelated run mutations for no gain. 4. Gate ``_is_unique_violation`` message fallbacks on ``isinstance(current, (SAIntegrityError, sqlite3.IntegrityError))``. The driver-code path (pgcode/sqlite_errorcode) remains load-bearing; substring fallbacks are now belt-and-suspenders only for cases where the driver attribute isn't reachable through the cause chain. Without the gate, any application exception whose ``str()`` happens to contain "duplicate key" / "unique" + "violat" (CHECK constraint, validation error) would silently surface as HTTP 409 instead of 500. 5. Route ``update_lease`` through ``_call_store_with_retry`` for consistency with every other store call, and wrap ``await self._renew_leases()`` in ``_heartbeat_loop`` with ``except Exception: logger.warning(...)``. Previously a transient error from the snapshot path or an unexpected exception would kill the heartbeat task silently — after which no lease is ever renewed again and every active run eventually looks orphaned. ``except Exception`` lets ``CancelledError`` (BaseException since 3.8) propagate so shutdown cancellation still works. Regression tests: - ``test_heartbeat_renews_pending_run_before_task_is_spawned`` - ``test_is_unique_violation_does_not_misclassify_application_exception`` Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): harden multi-worker migration, memory atomicity, and tz-naive lease comparison Three follow-up fixes to the multi-worker run ownership work: - migration 0004 dedupe pass: cancel superseded duplicate active rows per thread before creating the partial UNIQUE index ``uq_runs_thread_active`` so dirty DBs (Postgres deployments that had reconciliation skipped by the old sqlite-only gate, or any env that ran GATEWAY_WORKERS>1 before this PR) do not abort the alembic upgrade and block gateway startup. Keeps the newest active row per thread, marks the rest as error with an explanatory message. - MemoryRunStore.create_run_atomic interrupt/rollback path: split the single- pass loop into two passes (collect candidates, validate, then mutate) so a ConflictError raised on a later candidate does not leave earlier candidates half-interrupted. Mirrors the SQL store's transactional rollback semantics; the entire test_multi_worker_run_ownership.py suite runs against memory so this divergence was giving false confidence. - RunRepository.create_run_atomic interrupt path: coerce tz-naive ``row.lease_expires_at`` to UTC before comparing against the aware ``cutoff``. SQLite drops tzinfo on read despite ``DateTime(timezone=True)`` (this file's own comment acknowledges it), so the Python-side comparison raised ``TypeError: can't compare offset-naive and offset-aware datetimes`` whenever heartbeat was enabled on SQLite and a lease was non-NULL. Defaults (heartbeat off -> leases always NULL) masked it, but there was no guard against the combination. Follows the existing "naive is UTC" convention from ``coerce_iso``. Each fix ships with a regression test pinning the behavior. Co-Authored-By: heart-scalpel <heart-scalpel@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): enforce heartbeat for multi-worker, fix memory-store datetime comparison, lazy-import ConflictError in store layer Three fixes from code review: 1. Extend the startup gate (GATEWAY_WORKERS>1) to also require run_ownership.heartbeat_enabled=true. Without heartbeat every run has a NULL lease, so reconciliation treats all inflight rows as orphans and Worker B would kill Worker A's live runs on every rolling update or scale-up. 2. Fix MemoryRunStore.list_inflight_with_expired_lease to parse created_at as datetime instead of ISO string lexical comparison, and handle tz-naive lease values uniformly with the SQL store. 3. Store layer (sql.py, memory.py) now lazy-imports ConflictError inside create_run_atomic instead of importing from the higher RunManager layer at module level. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): add owner check to update_lease, document create() assumption, restore deleted comment - update_lease (SQL + memory) now requires owner_worker_id match in WHERE clause so the primitive is safe by construction against misuse - create() docstring notes it bypasses atomic create_run_atomic and assumes no active run exists for the thread - restore explanatory comment in MemoryRunStore.aggregate_tokens_by_thread that was dropped in an earlier commit Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): add psycopg3 sqlstate detection and periodic orphan reconciliation - _is_unique_violation now checks sqlstate attribute (psycopg3 uses this instead of pgcode). On Postgres, the only supported multi-worker backend, detection was falling through to the message-substring fallback. - _heartbeat_loop now runs reconcile_orphaned_inflight_runs every 3rd cycle (every lease_seconds) to catch orphans whose lease expires between pod restarts. Single-worker deployments are unaffected (heartbeat off). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: heart-scalpel <heart-scalpel@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
665 lines
30 KiB
Python
665 lines
30 KiB
Python
import hashlib
|
|
import logging
|
|
import os
|
|
from collections.abc import Mapping
|
|
from contextvars import ContextVar
|
|
from pathlib import Path
|
|
from typing import Any, Literal, Self
|
|
|
|
import yaml
|
|
from dotenv import load_dotenv
|
|
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, model_validator
|
|
|
|
from deerflow.config.acp_config import ACPAgentConfig, load_acp_config_from_dict
|
|
from deerflow.config.agents_api_config import AgentsApiConfig, load_agents_api_config_from_dict
|
|
from deerflow.config.auth_config import AuthAppConfig
|
|
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
|
|
from deerflow.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
|
|
from deerflow.config.database_config import DatabaseConfig
|
|
from deerflow.config.extensions_config import ExtensionsConfig
|
|
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
|
from deerflow.config.input_polish_config import InputPolishConfig
|
|
from deerflow.config.loop_detection_config import LoopDetectionConfig
|
|
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
|
from deerflow.config.model_config import ModelConfig
|
|
from deerflow.config.read_before_write_config import ReadBeforeWriteConfig
|
|
from deerflow.config.reload_boundary import format_field_description
|
|
from deerflow.config.run_events_config import RunEventsConfig
|
|
from deerflow.config.run_ownership_config import RunOwnershipConfig
|
|
from deerflow.config.runtime_paths import existing_project_file
|
|
from deerflow.config.safety_finish_reason_config import SafetyFinishReasonConfig
|
|
from deerflow.config.sandbox_config import SandboxConfig
|
|
from deerflow.config.scheduler_config import SchedulerConfig
|
|
from deerflow.config.skill_evolution_config import SkillEvolutionConfig
|
|
from deerflow.config.skill_scan_config import SkillScanConfig
|
|
from deerflow.config.skills_config import SkillsConfig
|
|
from deerflow.config.stream_bridge_config import StreamBridgeConfig, load_stream_bridge_config_from_dict
|
|
from deerflow.config.subagents_config import SubagentsAppConfig, load_subagents_config_from_dict
|
|
from deerflow.config.suggestions_config import SuggestionsConfig
|
|
from deerflow.config.summarization_config import SummarizationConfig, load_summarization_config_from_dict
|
|
from deerflow.config.title_config import TitleConfig, load_title_config_from_dict
|
|
from deerflow.config.token_budget_config import TokenBudgetConfig
|
|
from deerflow.config.token_usage_config import TokenUsageConfig
|
|
from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
|
|
from deerflow.config.tool_output_config import ToolOutputConfig
|
|
from deerflow.config.tool_progress_config import ToolProgressConfig
|
|
from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict
|
|
|
|
load_dotenv()
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
CONFIG_FILE_DATABASE_DEFAULTS = {
|
|
"backend": "sqlite",
|
|
"sqlite_dir": ".deer-flow/data",
|
|
}
|
|
|
|
|
|
class CircuitBreakerConfig(BaseModel):
|
|
"""Configuration for the LLM Circuit Breaker."""
|
|
|
|
failure_threshold: int = Field(default=5, description="Number of consecutive failures before tripping the circuit")
|
|
recovery_timeout_sec: int = Field(default=60, description="Time in seconds before attempting to recover the circuit")
|
|
|
|
|
|
class LoggingEnhanceConfig(BaseModel):
|
|
"""Request trace logging enhancement settings."""
|
|
|
|
enabled: bool = Field(default=False, description="Enable request-level trace ids in Gateway response headers and log records.")
|
|
format: Literal["text", "json"] = Field(default="text", description="Enhanced log output format.")
|
|
|
|
|
|
class LoggingConfig(BaseModel):
|
|
"""Logging configuration."""
|
|
|
|
enhance: LoggingEnhanceConfig = Field(default_factory=LoggingEnhanceConfig, description="Request trace correlation logging settings.")
|
|
|
|
|
|
def is_trace_correlation_enabled(config: Any) -> bool:
|
|
"""Return ``True`` when ``logging.enhance.enabled`` is set on *config*.
|
|
|
|
Single source of truth for the request-trace-correlation gate, shared by
|
|
the Gateway ``TraceMiddleware`` and the embedded ``DeerFlowClient`` so
|
|
the two entry points cannot drift on when ``deerflow_trace_id`` is
|
|
emitted (Langfuse metadata) and when a request-level trace id is bound
|
|
at all. Accepts any object exposing ``logging.enhance.enabled`` via
|
|
``getattr`` chains (``AppConfig``, ``SimpleNamespace`` fixtures, etc.);
|
|
missing intermediate attributes silently degrade to ``False``.
|
|
"""
|
|
logging_config = getattr(config, "logging", None)
|
|
enhance = getattr(logging_config, "enhance", None)
|
|
return bool(getattr(enhance, "enabled", False))
|
|
|
|
|
|
def _legacy_config_candidates() -> tuple[Path, ...]:
|
|
"""Return source-tree config.yaml locations for monorepo compatibility."""
|
|
backend_dir = Path(__file__).resolve().parents[4]
|
|
repo_root = backend_dir.parent
|
|
return (backend_dir / "config.yaml", repo_root / "config.yaml")
|
|
|
|
|
|
def logging_level_from_config(name: str | None) -> int:
|
|
"""Map ``config.yaml`` ``log_level`` string to a :mod:`logging` level constant."""
|
|
mapping = logging.getLevelNamesMapping()
|
|
return mapping.get((name or "info").strip().upper(), logging.INFO)
|
|
|
|
|
|
def apply_logging_level(name: str | None) -> None:
|
|
"""Resolve *name* to a logging level and apply it to the ``deerflow``/``app`` logger hierarchies.
|
|
|
|
Only the ``deerflow`` and ``app`` logger levels are changed so that
|
|
third-party library verbosity (e.g. uvicorn, sqlalchemy) is not
|
|
affected. Root handler levels are lowered (never raised) so that
|
|
messages from the configured loggers can propagate through without
|
|
being filtered, while preserving handler thresholds that may be
|
|
intentionally restrictive for third-party log output.
|
|
"""
|
|
level = logging_level_from_config(name)
|
|
for logger_name in ("deerflow", "app"):
|
|
logging.getLogger(logger_name).setLevel(level)
|
|
for handler in logging.root.handlers:
|
|
if level < handler.level:
|
|
handler.setLevel(level)
|
|
|
|
|
|
class AppConfig(BaseModel):
|
|
"""Config for the DeerFlow application"""
|
|
|
|
log_level: str = Field(
|
|
default="info",
|
|
description=format_field_description(
|
|
"log_level",
|
|
field_doc="Logging level for deerflow and app modules (debug/info/warning/error); third-party libraries are not affected.",
|
|
),
|
|
)
|
|
logging: LoggingConfig = Field(
|
|
default_factory=LoggingConfig,
|
|
description=format_field_description(
|
|
"logging",
|
|
field_doc="Structured logging and request trace correlation settings.",
|
|
),
|
|
)
|
|
token_usage: TokenUsageConfig = Field(default_factory=TokenUsageConfig, description="Token usage tracking configuration")
|
|
token_budget: TokenBudgetConfig = Field(default_factory=TokenBudgetConfig, description="Token Budget tracking and limits configuration.")
|
|
max_recursion_limit: int = Field(
|
|
default=1000,
|
|
ge=1,
|
|
description="Hard server-side ceiling for a client-supplied run recursion_limit. Client values above this are clamped; prevents runaway LangGraph super-steps (LLM cost / DoS).",
|
|
)
|
|
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
|
|
sandbox: SandboxConfig = Field(
|
|
description=format_field_description(
|
|
"sandbox",
|
|
field_doc="Sandbox provider configuration (local filesystem or Docker-based aio sandbox).",
|
|
),
|
|
)
|
|
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
|
|
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
|
|
skills: SkillsConfig = Field(default_factory=SkillsConfig, description="Skills configuration")
|
|
skill_scan: SkillScanConfig = Field(default_factory=SkillScanConfig, description="Native deterministic skill safety scanning configuration")
|
|
skill_evolution: SkillEvolutionConfig = Field(default_factory=SkillEvolutionConfig, description="Agent-managed skill evolution configuration")
|
|
extensions: ExtensionsConfig = Field(default_factory=ExtensionsConfig, description="Extensions configuration (MCP servers and skills state)")
|
|
tool_output: ToolOutputConfig = Field(default_factory=ToolOutputConfig, description="Tool output budget protection configuration")
|
|
tool_search: ToolSearchConfig = Field(default_factory=ToolSearchConfig, description="Tool search / deferred loading configuration")
|
|
title: TitleConfig = Field(default_factory=TitleConfig, description="Automatic title generation configuration")
|
|
summarization: SummarizationConfig = Field(default_factory=SummarizationConfig, description="Conversation summarization configuration")
|
|
memory: MemoryConfig = Field(default_factory=MemoryConfig, description="Memory subsystem configuration")
|
|
agents_api: AgentsApiConfig = Field(default_factory=AgentsApiConfig, description="Custom-agent management API configuration")
|
|
acp_agents: dict[str, ACPAgentConfig] = Field(default_factory=dict, description="ACP-compatible agent configuration")
|
|
subagents: SubagentsAppConfig = Field(default_factory=SubagentsAppConfig, description="Subagent runtime configuration")
|
|
guardrails: GuardrailsConfig = Field(default_factory=GuardrailsConfig, description="Guardrail middleware configuration")
|
|
input_polish: InputPolishConfig = Field(default_factory=InputPolishConfig, description="Pre-send input polishing configuration.")
|
|
suggestions: SuggestionsConfig = Field(default_factory=SuggestionsConfig, description="Follow-up suggestions configuration.")
|
|
circuit_breaker: CircuitBreakerConfig = Field(default_factory=CircuitBreakerConfig, description="LLM circuit breaker configuration")
|
|
channel_connections: ChannelConnectionsConfig = Field(
|
|
default_factory=ChannelConnectionsConfig,
|
|
description=format_field_description(
|
|
"channel_connections",
|
|
field_doc="User-facing IM channel connection configuration.",
|
|
),
|
|
)
|
|
loop_detection: LoopDetectionConfig = Field(default_factory=LoopDetectionConfig, description="Loop detection middleware configuration")
|
|
tool_progress: ToolProgressConfig = Field(default_factory=ToolProgressConfig, description="Tool progress state machine middleware configuration")
|
|
read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration")
|
|
safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration")
|
|
auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)")
|
|
model_config = ConfigDict(extra="allow")
|
|
database: DatabaseConfig = Field(
|
|
default_factory=DatabaseConfig,
|
|
description=format_field_description(
|
|
"database",
|
|
field_doc="Unified database backend for run/feedback metadata (memory, sqlite, or postgres).",
|
|
),
|
|
)
|
|
run_events: RunEventsConfig = Field(
|
|
default_factory=RunEventsConfig,
|
|
description=format_field_description(
|
|
"run_events",
|
|
field_doc="Run-event store backend (memory for dev, db for production queries, jsonl for lightweight single-node persistence).",
|
|
),
|
|
)
|
|
scheduler: SchedulerConfig = Field(
|
|
default_factory=SchedulerConfig,
|
|
description=format_field_description(
|
|
"scheduler",
|
|
field_doc="Scheduled task runtime configuration (background poller for one-time and cron agent runs).",
|
|
),
|
|
)
|
|
checkpointer: CheckpointerConfig | None = Field(
|
|
default=None,
|
|
description=format_field_description(
|
|
"checkpointer",
|
|
field_doc="LangGraph state-persistence checkpointer configuration.",
|
|
),
|
|
)
|
|
stream_bridge: StreamBridgeConfig | None = Field(
|
|
default=None,
|
|
description=format_field_description(
|
|
"stream_bridge",
|
|
field_doc="Stream bridge connecting agent workers to SSE endpoints.",
|
|
),
|
|
)
|
|
run_ownership: RunOwnershipConfig = Field(
|
|
default_factory=RunOwnershipConfig,
|
|
description=format_field_description(
|
|
"run_ownership",
|
|
field_doc="Run ownership and lease configuration for multi-worker deployments.",
|
|
),
|
|
)
|
|
|
|
# Name -> config lookup tables, (re)built after validation by
|
|
# ``_build_name_indexes``. They make ``get_model_config`` / ``get_tool_config``
|
|
# / ``get_tool_group_config`` O(1) instead of an O(n) ``next(...)`` scan per
|
|
# call. Private attrs are excluded from serialization.
|
|
_models_by_name: dict[str, ModelConfig] = PrivateAttr(default_factory=dict)
|
|
_tools_by_name: dict[str, ToolConfig] = PrivateAttr(default_factory=dict)
|
|
_tool_groups_by_name: dict[str, ToolGroupConfig] = PrivateAttr(default_factory=dict)
|
|
|
|
@model_validator(mode="before")
|
|
@classmethod
|
|
def _drop_null_config_sections(cls, data: Any) -> Any:
|
|
"""Treat a present-but-null config section as absent so its default applies.
|
|
|
|
Commenting out every entry under a top-level YAML key — e.g. ``models:``
|
|
(a list) or ``memory:`` (an object), with only comments beneath it as
|
|
shipped throughout ``config.example.yaml`` — makes PyYAML parse the value
|
|
as ``None``. Without this, the documented ``cp config.example.yaml
|
|
config.yaml`` first-run flow crashes with an opaque ``Input should be a
|
|
valid list`` / ``valid dictionary`` pydantic error for that section.
|
|
|
|
Dropping the ``None`` lets each field fall back to its default: list
|
|
sections become ``[]`` via ``default_factory=list`` and object sections
|
|
get their default config. This generalizes the earlier list-only
|
|
handling to every section that defines a default. The ``database``
|
|
section is independent and still owned by ``_apply_database_defaults``
|
|
(in ``from_file``), which applies concrete defaults beyond null-coercion.
|
|
Required sections without a default (``sandbox``) intentionally still
|
|
error when null — there is nothing to fall back to.
|
|
"""
|
|
if isinstance(data, dict):
|
|
return {key: value for key, value in data.items() if value is not None}
|
|
return data
|
|
|
|
@classmethod
|
|
def resolve_config_path(cls, config_path: str | None = None) -> Path:
|
|
"""Resolve the config file path.
|
|
|
|
Priority:
|
|
1. If provided `config_path` argument, use it.
|
|
2. If provided `DEER_FLOW_CONFIG_PATH` environment variable, use it.
|
|
3. Otherwise, search the caller project root.
|
|
4. Finally, search legacy backend/repository-root defaults for monorepo compatibility.
|
|
"""
|
|
if config_path:
|
|
path = Path(config_path)
|
|
if not Path.exists(path):
|
|
raise FileNotFoundError(f"Config file specified by param `config_path` not found at {path}")
|
|
return path
|
|
elif os.getenv("DEER_FLOW_CONFIG_PATH"):
|
|
path = Path(os.getenv("DEER_FLOW_CONFIG_PATH"))
|
|
if not Path.exists(path):
|
|
raise FileNotFoundError(f"Config file specified by environment variable `DEER_FLOW_CONFIG_PATH` not found at {path}")
|
|
return path
|
|
else:
|
|
project_config = existing_project_file(("config.yaml",))
|
|
if project_config is not None:
|
|
return project_config
|
|
|
|
for path in _legacy_config_candidates():
|
|
if path.exists():
|
|
return path
|
|
raise FileNotFoundError("`config.yaml` file not found in the project root or legacy backend/repository root locations")
|
|
|
|
@classmethod
|
|
def from_file(cls, config_path: str | None = None) -> Self:
|
|
"""Load config from YAML file.
|
|
|
|
See `resolve_config_path` for more details.
|
|
|
|
Args:
|
|
config_path: Path to the config file.
|
|
|
|
Returns:
|
|
AppConfig: The loaded config.
|
|
"""
|
|
resolved_path = cls.resolve_config_path(config_path)
|
|
with open(resolved_path, encoding="utf-8") as f:
|
|
config_data = yaml.safe_load(f) or {}
|
|
|
|
# Check config version before processing
|
|
cls._check_config_version(config_data, resolved_path)
|
|
|
|
config_data = cls.resolve_env_variables(config_data)
|
|
cls._apply_database_defaults(config_data)
|
|
|
|
# Load circuit_breaker config if present
|
|
if "circuit_breaker" in config_data:
|
|
config_data["circuit_breaker"] = config_data["circuit_breaker"]
|
|
|
|
# Load extensions config separately (it's in a different file)
|
|
extensions_config = ExtensionsConfig.from_file()
|
|
config_data["extensions"] = extensions_config.model_dump()
|
|
|
|
result = cls.model_validate(config_data)
|
|
if not result.models:
|
|
logger.warning(
|
|
"No models are configured in %s. Add at least one entry under `models:` (see the commented examples in config.example.yaml) or run `make setup`.",
|
|
resolved_path,
|
|
)
|
|
acp_agents = cls._validate_acp_agents(config_data.get("acp_agents", {}))
|
|
cls._apply_singleton_configs(result, acp_agents)
|
|
return result
|
|
|
|
@classmethod
|
|
def _validate_acp_agents(
|
|
cls,
|
|
config_data: Mapping[str, Mapping[str, object]] | None,
|
|
) -> dict[str, ACPAgentConfig]:
|
|
if config_data is None:
|
|
config_data = {}
|
|
return {name: ACPAgentConfig(**cfg) for name, cfg in config_data.items()}
|
|
|
|
@classmethod
|
|
def _apply_singleton_configs(cls, config: Self, acp_agents: dict[str, ACPAgentConfig]) -> None:
|
|
from deerflow.config.checkpointer_config import get_checkpointer_config
|
|
|
|
previous_checkpointer_config = get_checkpointer_config()
|
|
|
|
load_title_config_from_dict(config.title.model_dump())
|
|
load_summarization_config_from_dict(config.summarization.model_dump())
|
|
load_memory_config_from_dict(config.memory.model_dump())
|
|
load_agents_api_config_from_dict(config.agents_api.model_dump())
|
|
load_subagents_config_from_dict(config.subagents.model_dump())
|
|
load_tool_search_config_from_dict(config.tool_search.model_dump())
|
|
load_guardrails_config_from_dict(config.guardrails.model_dump())
|
|
load_checkpointer_config_from_dict(config.checkpointer.model_dump() if config.checkpointer is not None else None)
|
|
load_stream_bridge_config_from_dict(config.stream_bridge.model_dump() if config.stream_bridge is not None else None)
|
|
load_acp_config_from_dict({name: agent.model_dump() for name, agent in acp_agents.items()})
|
|
|
|
if previous_checkpointer_config != config.checkpointer:
|
|
# These runtime singletons derive their backend from checkpointer config.
|
|
# Keep imports local to avoid cycles: both providers import get_app_config.
|
|
from deerflow.runtime.checkpointer import reset_checkpointer
|
|
from deerflow.runtime.store import reset_store
|
|
|
|
reset_checkpointer()
|
|
reset_store()
|
|
|
|
@classmethod
|
|
def _apply_database_defaults(cls, config_data: dict[str, Any]) -> None:
|
|
"""Apply config.yaml defaults for persistence when the section is absent."""
|
|
database_config = config_data.get("database")
|
|
if database_config is None:
|
|
database_config = {}
|
|
config_data["database"] = database_config
|
|
if not isinstance(database_config, dict):
|
|
return
|
|
for key, value in CONFIG_FILE_DATABASE_DEFAULTS.items():
|
|
database_config.setdefault(key, value)
|
|
|
|
@classmethod
|
|
def _check_config_version(cls, config_data: dict, config_path: Path) -> None:
|
|
"""Check if the user's config.yaml is outdated compared to config.example.yaml.
|
|
|
|
Emits a warning if the user's config_version is lower than the example's.
|
|
Missing config_version is treated as version 0 (pre-versioning).
|
|
"""
|
|
try:
|
|
user_version = int(config_data.get("config_version", 0))
|
|
except (TypeError, ValueError):
|
|
user_version = 0
|
|
|
|
# Find config.example.yaml by searching config.yaml's directory and its parents
|
|
example_path = None
|
|
search_dir = config_path.parent
|
|
for _ in range(5): # search up to 5 levels
|
|
candidate = search_dir / "config.example.yaml"
|
|
if candidate.exists():
|
|
example_path = candidate
|
|
break
|
|
parent = search_dir.parent
|
|
if parent == search_dir:
|
|
break
|
|
search_dir = parent
|
|
if example_path is None:
|
|
return
|
|
|
|
try:
|
|
with open(example_path, encoding="utf-8") as f:
|
|
example_data = yaml.safe_load(f)
|
|
raw = example_data.get("config_version", 0) if example_data else 0
|
|
try:
|
|
example_version = int(raw)
|
|
except (TypeError, ValueError):
|
|
example_version = 0
|
|
except Exception:
|
|
return
|
|
|
|
if user_version < example_version:
|
|
logger.warning(
|
|
"Your config.yaml (version %d) is outdated — the latest version is %d. Run `make config-upgrade` to merge new fields into your config.",
|
|
user_version,
|
|
example_version,
|
|
)
|
|
|
|
@classmethod
|
|
def resolve_env_variables(cls, config: Any) -> Any:
|
|
"""Recursively resolve environment variables in the config.
|
|
|
|
Environment variables are resolved using the `os.getenv` function. Example: $OPENAI_API_KEY
|
|
|
|
Args:
|
|
config: The config to resolve environment variables in.
|
|
|
|
Returns:
|
|
The config with environment variables resolved.
|
|
"""
|
|
if isinstance(config, str):
|
|
if config.startswith("$"):
|
|
env_value = os.getenv(config[1:])
|
|
if env_value is None:
|
|
raise ValueError(f"Environment variable {config[1:]} not found for config value {config}")
|
|
return env_value
|
|
return config
|
|
elif isinstance(config, dict):
|
|
return {k: cls.resolve_env_variables(v) for k, v in config.items()}
|
|
elif isinstance(config, list):
|
|
return [cls.resolve_env_variables(item) for item in config]
|
|
return config
|
|
|
|
@model_validator(mode="after")
|
|
def _build_name_indexes(self) -> "AppConfig":
|
|
"""Build name -> config lookup tables for O(1) ``get_*_config``.
|
|
|
|
``get_tool_config`` runs 2-3x per community-tool invocation (e.g.
|
|
web_search) and ``get_model_config`` several times per agent build, so
|
|
the previous O(n) ``next(...)`` scans sat on hot paths. Rebuilt here so a
|
|
config reload (which constructs a fresh ``AppConfig``) refreshes them.
|
|
``setdefault`` keeps the first entry on duplicate names, preserving the
|
|
prior ``next(...)`` first-match semantics.
|
|
"""
|
|
models_by_name: dict[str, ModelConfig] = {}
|
|
for model in self.models:
|
|
models_by_name.setdefault(model.name, model)
|
|
tools_by_name: dict[str, ToolConfig] = {}
|
|
for tool in self.tools:
|
|
tools_by_name.setdefault(tool.name, tool)
|
|
tool_groups_by_name: dict[str, ToolGroupConfig] = {}
|
|
for group in self.tool_groups:
|
|
tool_groups_by_name.setdefault(group.name, group)
|
|
self._models_by_name = models_by_name
|
|
self._tools_by_name = tools_by_name
|
|
self._tool_groups_by_name = tool_groups_by_name
|
|
return self
|
|
|
|
def get_model_config(self, name: str) -> ModelConfig | None:
|
|
"""Get the model config by name.
|
|
|
|
Args:
|
|
name: The name of the model to get the config for.
|
|
|
|
Returns:
|
|
The model config if found, otherwise None.
|
|
"""
|
|
return self._models_by_name.get(name)
|
|
|
|
def get_tool_config(self, name: str) -> ToolConfig | None:
|
|
"""Get the tool config by name.
|
|
|
|
Args:
|
|
name: The name of the tool to get the config for.
|
|
|
|
Returns:
|
|
The tool config if found, otherwise None.
|
|
"""
|
|
return self._tools_by_name.get(name)
|
|
|
|
def get_tool_group_config(self, name: str) -> ToolGroupConfig | None:
|
|
"""Get the tool group config by name.
|
|
|
|
Args:
|
|
name: The name of the tool group to get the config for.
|
|
|
|
Returns:
|
|
The tool group config if found, otherwise None.
|
|
"""
|
|
return self._tool_groups_by_name.get(name)
|
|
|
|
|
|
# Compatibility singleton layer for code paths that have not yet been
|
|
# migrated to explicit ``AppConfig`` threading. New composition roots should
|
|
# prefer constructing ``AppConfig`` once and passing it down directly.
|
|
_app_config: AppConfig | None = None
|
|
_app_config_path: Path | None = None
|
|
_app_config_mtime: float | None = None
|
|
_ConfigSignature = tuple[float | None, int | None, str | None]
|
|
_app_config_signature: _ConfigSignature | None = None
|
|
_app_config_is_custom = False
|
|
_current_app_config: ContextVar[AppConfig | None] = ContextVar("deerflow_current_app_config", default=None)
|
|
_current_app_config_stack: ContextVar[tuple[AppConfig | None, ...]] = ContextVar("deerflow_current_app_config_stack", default=())
|
|
|
|
|
|
def _get_config_mtime(config_path: Path) -> float | None:
|
|
"""Get the modification time of a config file if it exists."""
|
|
try:
|
|
return config_path.stat().st_mtime
|
|
except OSError:
|
|
return None
|
|
|
|
|
|
def _get_config_signature(config_path: Path) -> _ConfigSignature | None:
|
|
"""Get cache metadata for a config file, including a content digest."""
|
|
try:
|
|
stat_result = config_path.stat()
|
|
except OSError:
|
|
return None
|
|
|
|
digest = hashlib.sha256()
|
|
try:
|
|
with config_path.open("rb") as f:
|
|
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
|
digest.update(chunk)
|
|
except OSError:
|
|
return (stat_result.st_mtime, stat_result.st_size, None)
|
|
|
|
return (stat_result.st_mtime, stat_result.st_size, digest.hexdigest())
|
|
|
|
|
|
def _load_and_cache_app_config(config_path: str | None = None) -> AppConfig:
|
|
"""Load config from disk and refresh cache metadata."""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
|
|
resolved_path = AppConfig.resolve_config_path(config_path)
|
|
_app_config = AppConfig.from_file(str(resolved_path))
|
|
_app_config_path = resolved_path
|
|
_app_config_mtime = _get_config_mtime(resolved_path)
|
|
_app_config_signature = _get_config_signature(resolved_path)
|
|
_app_config_is_custom = False
|
|
return _app_config
|
|
|
|
|
|
def get_app_config() -> AppConfig:
|
|
"""Get the DeerFlow config instance.
|
|
|
|
Returns a cached singleton instance and automatically reloads it when the
|
|
underlying config file path or content signature changes. Use
|
|
`reload_app_config()` to force a reload, or `reset_app_config()` to clear
|
|
the cache.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature
|
|
|
|
runtime_override = _current_app_config.get()
|
|
if runtime_override is not None:
|
|
return runtime_override
|
|
|
|
if _app_config is not None and _app_config_is_custom:
|
|
return _app_config
|
|
|
|
resolved_path = AppConfig.resolve_config_path()
|
|
current_mtime = _get_config_mtime(resolved_path)
|
|
current_signature = _get_config_signature(resolved_path)
|
|
|
|
should_reload = _app_config is None or _app_config_path != resolved_path or _app_config_signature != current_signature
|
|
if should_reload:
|
|
if _app_config_path == resolved_path and _app_config_mtime is not None and current_mtime is not None and _app_config_mtime != current_mtime:
|
|
logger.info(
|
|
"Config file has been modified (mtime: %s -> %s), reloading AppConfig",
|
|
_app_config_mtime,
|
|
current_mtime,
|
|
)
|
|
elif _app_config_path == resolved_path and _app_config_signature != current_signature:
|
|
logger.info("Config file content signature changed, reloading AppConfig")
|
|
_load_and_cache_app_config(str(resolved_path))
|
|
return _app_config
|
|
|
|
|
|
def reload_app_config(config_path: str | None = None) -> AppConfig:
|
|
"""Reload the config from file and update the cached instance.
|
|
|
|
This is useful when the config file has been modified and you want
|
|
to pick up the changes without restarting the application.
|
|
|
|
Args:
|
|
config_path: Optional path to config file. If not provided,
|
|
uses the default resolution strategy.
|
|
|
|
Returns:
|
|
The newly loaded AppConfig instance.
|
|
"""
|
|
return _load_and_cache_app_config(config_path)
|
|
|
|
|
|
def reset_app_config() -> None:
|
|
"""Reset the cached config instance.
|
|
|
|
This clears the singleton cache, causing the next call to
|
|
`get_app_config()` to reload from file. Useful for testing
|
|
or when switching between different configurations.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
_app_config = None
|
|
_app_config_path = None
|
|
_app_config_mtime = None
|
|
_app_config_signature = None
|
|
_app_config_is_custom = False
|
|
|
|
|
|
def set_app_config(config: AppConfig) -> None:
|
|
"""Set a custom config instance.
|
|
|
|
This allows injecting a custom or mock config for testing purposes.
|
|
|
|
Args:
|
|
config: The AppConfig instance to use.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
_app_config = config
|
|
_app_config_path = None
|
|
_app_config_mtime = None
|
|
_app_config_signature = None
|
|
_app_config_is_custom = True
|
|
|
|
|
|
def peek_current_app_config() -> AppConfig | None:
|
|
"""Return the runtime-scoped AppConfig override, if one is active."""
|
|
return _current_app_config.get()
|
|
|
|
|
|
def push_current_app_config(config: AppConfig) -> None:
|
|
"""Push a runtime-scoped AppConfig override for the current execution context."""
|
|
stack = _current_app_config_stack.get()
|
|
_current_app_config_stack.set(stack + (_current_app_config.get(),))
|
|
_current_app_config.set(config)
|
|
|
|
|
|
def pop_current_app_config() -> None:
|
|
"""Pop the latest runtime-scoped AppConfig override for the current execution context."""
|
|
stack = _current_app_config_stack.get()
|
|
if not stack:
|
|
_current_app_config.set(None)
|
|
return
|
|
previous = stack[-1]
|
|
_current_app_config_stack.set(stack[:-1])
|
|
_current_app_config.set(previous)
|