He Wang fd41fdb065
feat(middleware): add structured tool result meta and tool-progress state machine (#3601)
* feat(middleware): add structured tool result meta and tool-progress state machine

feat:
- Add tool_result_meta.py: ToolResultMeta dataclass (status/error_type/retryable/
  recoverable_by_model/recommended_next_action/source) + normalize_tool_result and
  stamp_exception_meta utilities; classifies every ToolMessage regardless of path
- Add ToolProgressMiddleware: per-(thread_id, tool_name) state machine ACTIVE →
  WARNED (hint injected as HumanMessage) → BLOCKED (call short-circuited); Jaccard
  near-duplicate detection for repeated successful results; auth/config/internal
  errors bypass WARNED and go directly to BLOCKED; LRU-bounded thread state store
- Add ToolProgressConfig: all thresholds configurable (stagnation_threshold,
  warn_escalation_count, jaccard_similarity_threshold, exempt_tools, etc.);
  disabled by default (enabled: false)
- Wire ToolProgressMiddleware as outer wrapper around ToolErrorHandlingMiddleware
  in _build_runtime_middlewares so it receives results already carrying
  deerflow_tool_meta

fix:
- ToolErrorHandlingMiddleware now calls stamp_exception_meta on exception path and
  normalize_tool_result on success path so every ToolMessage carries deerflow_tool_meta

test:
- Add test_tool_result_meta.py: 26 cases covering all classification paths,
  stamp_exception_meta, and normalize_tool_result Command passthrough
- Add test_tool_progress_middleware.py: 27 cases including full async paths,
  Jaccard duplicate detection, LRU eviction, hint injection, and malformed meta
  passthrough
- Extend test_tool_error_handling_middleware.py: middleware ordering invariant and
  meta stamping on exception

docs:
- Add tool_progress section to config.example.yaml with all fields and descriptions
- Update CLAUDE.md middleware chain documentation (entries 8-9)

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(middleware): recoverable errors stay WARNED; fix auth keyword shadowing

fix:
- WARNED is terminal for recoverable_by_model=True errors (no_results, not_found,
  permission); hint re-injected on each problem call instead of escalating to
  BLOCKED, so the model can retry with different parameters (e.g. fresh query,
  new URL) without being hard-blocked by a prior stagnation count.
  Non-recoverable (rate_limited, transient) still escalate WARNED → BLOCKED
  after warn_escalation_count more problems; auth/config/internal remain
  immediately BLOCKED.
- Remove bare "api key" keyword from auth classification rule so "no api key
  configured" correctly classifies as config (not auth), producing the accurate
  block-reason text for the model.

docs:
- CLAUDE.md: document all three ToolProgressMiddleware transition paths
- config.example.yaml: update inline state-machine comment to match new paths

test:
- test_recoverable_errors_stay_warned_indefinitely: WARNED never escalates for
  recoverable errors regardless of how many problem calls accumulate
- test_recoverable_error_re_injects_hint_past_escalation: hints continue past
  the escalation zone for recoverable errors
- test_no_api_key_is_config_not_auth: regression guard for keyword shadowing fix

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(tool_result_meta): add JSON error extraction and fix source classification

fix:
- Fix non-standard error path: source was "exception" but should be "tool_return"
- Add _extract_json_error_text to isolate JSON error fields from noisy JSON bodies
  (e.g. Brave Search {"error": "...", "query": "..."} — query keywords no longer
  pollute error classification)
- Add success-path JSON extraction to catch tools that return HTTP 200 with a JSON
  error body (status="success" but {"error": "API key not configured"})
- Add _SEMANTIC_ZERO_ERROR_STRINGS frozenset to suppress false positives from tools
  that use {"error": "none"} / {"error": "null"} / {"error": "ok"} as success signals
- Document that stamp_exception_meta always overwrites existing TOOL_META_KEY
  (exception-derived classification is authoritative over tool return-time stamps)

test:
- Add parametrized regression tests for all semantic-zero error strings
- Add tests for non-standard error path source field
- Add tests for JSON error extraction (nonstd, success-path, numeric, falsy values)
- Correct test comment for test_no_api_key_is_config_not_auth

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(tool_progress_middleware): fix 6 bugs, add terminal guard and structured logging

fix:
- H1: fix exempt_tools empty-set silently ignored — use `is not None` instead of
  truthiness check so ToolProgressConfig(exempt_tools=set()) correctly disables all
  exemptions
- Fix _get_block_reason creating phantom LRU entries via _get_state (write path);
  now uses dict.get + explicit move_to_end on read path only
- Fix _pending memory leak: LRU eviction of _phase_states now synchronously removes
  all (evicted_thread, *) keys from _pending
- Fix _assess_and_transition missing terminal guard for blocked state — a recoverable
  error result could silently demote blocked → warned in concurrent-race scenarios;
  early return preserves terminal semantics
- Fix recent_word_sets window: stored [-5:] but is_near_duplicate only compared [-3:];
  align to [-3:] and change type list→tuple (prevents accidental in-place mutation
  across dataclasses.replace shallow copies)
- Fix _format_hint missing "success" key and "continue" action: Jaccard near-duplicate
  results produced the generic fallback instead of a specific actionable message

feat:
- Add structured state-transition logging (ACTIVE/WARNED/BLOCKED transitions, blocked
  intercepts, hint injection debug log)

test:
- Add regression tests for all 6 bug fixes (H1, phantom LRU, pending leak, terminal
  guard, window alignment, format_hint near-dup)
- Add Jaccard near-threshold boundary test (7/9 vs 8/9 Jaccard)
- Add production min_words=10 skip test for short content
- Add exempt_tools empty-set and None round-trip tests
- Add _augment_request deduplication test
- Add before_agent current-run preservation test
- Add structured logging tests (WARNED/BLOCKED/ACTIVE/intercepted/debug)

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* chore(config): remove unused backward-compat fields from ToolProgressConfig

Remove max_calls_per_intent and window_size fields that were marked
"Retained for backward compatibility; not used by the current state machine"
when the state machine was introduced.  Pydantic v2 ignores unknown fields
by default, so existing config.yaml files with these keys remain valid.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(tool_progress): address PR review and multi-agent review findings

fix:
- Remove <80-char length gate for partial_success; only _PARTIAL_MARKERS now
- Add word-boundary regex for numeric HTTP codes (401/403/404/500) to avoid
  false positives like "500ms" or "4010 rows" triggering hard-block
- Add "task" to default exempt_tools (delegation primitive, not a search tool)
- Remove move_to_end() from _get_block_reason read path; blocked threads were
  permanently warm in LRU, starving active threads of eviction slots
- Add _reset_blocked_states in before_agent: scope BLOCKED and WARNED states
  to a single run; clear recent_word_sets so stale Jaccard windows don't cause
  false near-duplicate detections in the next run
- Compute word_set() lazily (only for success results); cap content at 8192
  chars to bound memory and CPU cost on large tool results
- Remove unused retryable field from ToolResultMeta (no consumer existed)
- Add isinstance-based ordering guard and warning log for missing meta
- Fix JSON-without-error-key fallback: use _UNKNOWN_ERROR instead of
  classifying incidental field values (e.g. {"user_id": 401} → auth → stop)
- Fix _extract_json_error_text: use json.dumps for dict/list error fields
  instead of str() which produced Python repr matching config rules spuriously
- Add "no results found"/"no content found"/"no images found" to _PARTIAL_MARKERS
  so success responses with empty results trigger stagnation detection
- Fix immediate-block path to increment consecutive_problems (was left at 0)
- Fix _queue_assessment: skip phantom _pending entries for evicted threads
- Bump config_version 13→16 (upstream added 14/15; tool_progress is additive)

test:
- Update test_short_content_is_partial → test_short_terse_success_is_not_partial
- Add parametrized test_numeric_keyword_word_boundary (8 positive + negative cases)
- Add test_before_agent_resets_blocked_states_for_new_run (strengthened assertions)
- Add test_before_agent_resets_warned_states_for_new_run
- Add test_missing_meta_on_non_exempt_tool_emits_warning
- Add test_middleware_ordering_guard_raises_when_progress_is_inner
- Add test_auth_error_immediately_blocked asserts consecutive_problems == 1
- Add tests for JSON-without-error-key, dict error field, no-results partial_success

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(tool_progress): address second PR review — perf, architecture doc, concurrency note

fix:
- Extract content.lower() once before _PARTIAL_MARKERS check in normalize_tool_message;
  previously computed up to 7× per call inside the generator (once per marker)

docs:
- Add division-of-labor paragraph to ToolProgressMiddleware module docstring explaining
  coexistence with LoopDetectionMiddleware: result-quality guard (per-tool BLOCK) vs
  call-pattern guard (whole-turn hard-stop); no shared state, no double-stop risk
- Add threading.Lock comment explaining why asyncio.Lock is not used (short critical
  sections, must also protect sync wrap_tool_call path from subagent executor threads)
- Update backend/CLAUDE.md entry 8 with division-of-labor summary; fix entry 9
  (remove stale retryable field reference, add missing recoverable_by_model/source)

test:
- Add test_tool_progress_and_loop_detection_coexist_without_interfering: drives both
  middlewares to WARNED state simultaneously, verifies independent state, independent
  hint queues, and no cross-contamination; uses snapshot copy for final assertion

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(tool_progress): reset all tool states at run boundary; fix semantic-zero test validity

fix:
- _reset_run_states (formerly _reset_blocked_states) drops the phase filter and
  resets all tracked (thread, tool) pairs unconditionally at before_agent; ACTIVE
  tools with sub-threshold consecutive_problems or cached recent_word_sets no longer
  bleed into the next run, preventing spurious WARNED transitions on clean R2 calls
- test_normalize_json_semantic_zero_error_string_not_treated_as_error: replace
  {error_value!r} f-string (produces invalid JSON with single quotes) with
  json.dumps so _extract_json_error_text actually parses the payload and the
  _SEMANTIC_ZERO_ERROR_STRINGS guard is exercised, not bypassed at json.loads

test:
- add test_before_agent_resets_active_state_consecutive_problems_and_word_sets to
  lock the ACTIVE-phase run-boundary reset: drives tool to active/cp=1/ws≠() in R1,
  asserts both fields are zero/empty after before_agent fires for R2

* docs(tool_progress): document intentional per-run reset vs LoopDetection thread-scoped retention

Addresses reviewer observation in PR #3601 that _reset_run_states diverges
from LoopDetectionMiddleware's cross-run scoping policy without explanation.

Expands the _reset_run_states docstring to record the intentional design
choice: ToolProgressMiddleware resets per-run because result-quality errors
(rate_limited, transient) are time-bound and may resolve between turns —
retaining stale counters would risk false-positive BLOCKED calls.
LoopDetectionMiddleware retains history across runs because call-pattern
loops are time-invariant. The divergence is by design, not oversight.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(middleware): restore ReadBeforeWriteMiddleware as outermost write gate

A merge conflict resolution had accidentally placed ReadBeforeWriteMiddleware
after ToolErrorHandlingMiddleware (inner), reversing the original intent from
b81334cc where it was the outermost write gate before ToolErrorHandling.

fix:
- Restore ReadBeforeWriteMiddleware to outer position: ReadBeforeWrite →
  ToolProgress → ToolErrorHandling. Blocked writes now return immediately
  without consuming a ToolProgress slot.
- Add normalize_tool_result call on blocked ToolMessages so they carry
  deerflow_tool_meta (recoverable_by_model=True) even though they bypass
  ToolErrorHandlingMiddleware.

test:
- Add test_blocked_write_has_deerflow_tool_meta (sync + async) to lock the
  normalize_tool_result behavior on blocked writes.
- Fix chain order assertions in TestChainWiring and
  test_build_lead_runtime_middlewares_chain_order_matches_agents_md.

docs:
- Renumber AGENTS.md items: 10→ReadBeforeWrite, 11→ToolProgress,
  12→ToolErrorHandling; update descriptions to reflect outermost-gate design.
- Fix stale cross-reference: LoopDetectionMiddleware (item 23) → (item 25).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-06 20:57:41 +08:00

653 lines
29 KiB
Python

import hashlib
import logging
import os
from collections.abc import Mapping
from contextvars import ContextVar
from pathlib import Path
from typing import Any, Literal, Self
import yaml
from dotenv import load_dotenv
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, model_validator
from deerflow.config.acp_config import ACPAgentConfig, load_acp_config_from_dict
from deerflow.config.agents_api_config import AgentsApiConfig, load_agents_api_config_from_dict
from deerflow.config.auth_config import AuthAppConfig
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
from deerflow.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
from deerflow.config.database_config import DatabaseConfig
from deerflow.config.extensions_config import ExtensionsConfig
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
from deerflow.config.loop_detection_config import LoopDetectionConfig
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
from deerflow.config.model_config import ModelConfig
from deerflow.config.read_before_write_config import ReadBeforeWriteConfig
from deerflow.config.reload_boundary import format_field_description
from deerflow.config.run_events_config import RunEventsConfig
from deerflow.config.runtime_paths import existing_project_file
from deerflow.config.safety_finish_reason_config import SafetyFinishReasonConfig
from deerflow.config.sandbox_config import SandboxConfig
from deerflow.config.scheduler_config import SchedulerConfig
from deerflow.config.skill_evolution_config import SkillEvolutionConfig
from deerflow.config.skills_config import SkillsConfig
from deerflow.config.stream_bridge_config import StreamBridgeConfig, load_stream_bridge_config_from_dict
from deerflow.config.subagents_config import SubagentsAppConfig, load_subagents_config_from_dict
from deerflow.config.suggestions_config import SuggestionsConfig
from deerflow.config.summarization_config import SummarizationConfig, load_summarization_config_from_dict
from deerflow.config.title_config import TitleConfig, load_title_config_from_dict
from deerflow.config.token_budget_config import TokenBudgetConfig
from deerflow.config.token_usage_config import TokenUsageConfig
from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
from deerflow.config.tool_output_config import ToolOutputConfig
from deerflow.config.tool_progress_config import ToolProgressConfig
from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict
load_dotenv()
logger = logging.getLogger(__name__)
CONFIG_FILE_DATABASE_DEFAULTS = {
"backend": "sqlite",
"sqlite_dir": ".deer-flow/data",
}
class CircuitBreakerConfig(BaseModel):
"""Configuration for the LLM Circuit Breaker."""
failure_threshold: int = Field(default=5, description="Number of consecutive failures before tripping the circuit")
recovery_timeout_sec: int = Field(default=60, description="Time in seconds before attempting to recover the circuit")
class LoggingEnhanceConfig(BaseModel):
"""Request trace logging enhancement settings."""
enabled: bool = Field(default=False, description="Enable request-level trace ids in Gateway response headers and log records.")
format: Literal["text", "json"] = Field(default="text", description="Enhanced log output format.")
class LoggingConfig(BaseModel):
"""Logging configuration."""
enhance: LoggingEnhanceConfig = Field(default_factory=LoggingEnhanceConfig, description="Request trace correlation logging settings.")
def is_trace_correlation_enabled(config: Any) -> bool:
"""Return ``True`` when ``logging.enhance.enabled`` is set on *config*.
Single source of truth for the request-trace-correlation gate, shared by
the Gateway ``TraceMiddleware`` and the embedded ``DeerFlowClient`` so
the two entry points cannot drift on when ``deerflow_trace_id`` is
emitted (Langfuse metadata) and when a request-level trace id is bound
at all. Accepts any object exposing ``logging.enhance.enabled`` via
``getattr`` chains (``AppConfig``, ``SimpleNamespace`` fixtures, etc.);
missing intermediate attributes silently degrade to ``False``.
"""
logging_config = getattr(config, "logging", None)
enhance = getattr(logging_config, "enhance", None)
return bool(getattr(enhance, "enabled", False))
def _legacy_config_candidates() -> tuple[Path, ...]:
"""Return source-tree config.yaml locations for monorepo compatibility."""
backend_dir = Path(__file__).resolve().parents[4]
repo_root = backend_dir.parent
return (backend_dir / "config.yaml", repo_root / "config.yaml")
def logging_level_from_config(name: str | None) -> int:
"""Map ``config.yaml`` ``log_level`` string to a :mod:`logging` level constant."""
mapping = logging.getLevelNamesMapping()
return mapping.get((name or "info").strip().upper(), logging.INFO)
def apply_logging_level(name: str | None) -> None:
"""Resolve *name* to a logging level and apply it to the ``deerflow``/``app`` logger hierarchies.
Only the ``deerflow`` and ``app`` logger levels are changed so that
third-party library verbosity (e.g. uvicorn, sqlalchemy) is not
affected. Root handler levels are lowered (never raised) so that
messages from the configured loggers can propagate through without
being filtered, while preserving handler thresholds that may be
intentionally restrictive for third-party log output.
"""
level = logging_level_from_config(name)
for logger_name in ("deerflow", "app"):
logging.getLogger(logger_name).setLevel(level)
for handler in logging.root.handlers:
if level < handler.level:
handler.setLevel(level)
class AppConfig(BaseModel):
"""Config for the DeerFlow application"""
log_level: str = Field(
default="info",
description=format_field_description(
"log_level",
field_doc="Logging level for deerflow and app modules (debug/info/warning/error); third-party libraries are not affected.",
),
)
logging: LoggingConfig = Field(
default_factory=LoggingConfig,
description=format_field_description(
"logging",
field_doc="Structured logging and request trace correlation settings.",
),
)
token_usage: TokenUsageConfig = Field(default_factory=TokenUsageConfig, description="Token usage tracking configuration")
token_budget: TokenBudgetConfig = Field(default_factory=TokenBudgetConfig, description="Token Budget tracking and limits configuration.")
max_recursion_limit: int = Field(
default=1000,
ge=1,
description="Hard server-side ceiling for a client-supplied run recursion_limit. Client values above this are clamped; prevents runaway LangGraph super-steps (LLM cost / DoS).",
)
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
sandbox: SandboxConfig = Field(
description=format_field_description(
"sandbox",
field_doc="Sandbox provider configuration (local filesystem or Docker-based aio sandbox).",
),
)
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
skills: SkillsConfig = Field(default_factory=SkillsConfig, description="Skills configuration")
skill_evolution: SkillEvolutionConfig = Field(default_factory=SkillEvolutionConfig, description="Agent-managed skill evolution configuration")
extensions: ExtensionsConfig = Field(default_factory=ExtensionsConfig, description="Extensions configuration (MCP servers and skills state)")
tool_output: ToolOutputConfig = Field(default_factory=ToolOutputConfig, description="Tool output budget protection configuration")
tool_search: ToolSearchConfig = Field(default_factory=ToolSearchConfig, description="Tool search / deferred loading configuration")
title: TitleConfig = Field(default_factory=TitleConfig, description="Automatic title generation configuration")
summarization: SummarizationConfig = Field(default_factory=SummarizationConfig, description="Conversation summarization configuration")
memory: MemoryConfig = Field(default_factory=MemoryConfig, description="Memory subsystem configuration")
agents_api: AgentsApiConfig = Field(default_factory=AgentsApiConfig, description="Custom-agent management API configuration")
acp_agents: dict[str, ACPAgentConfig] = Field(default_factory=dict, description="ACP-compatible agent configuration")
subagents: SubagentsAppConfig = Field(default_factory=SubagentsAppConfig, description="Subagent runtime configuration")
guardrails: GuardrailsConfig = Field(default_factory=GuardrailsConfig, description="Guardrail middleware configuration")
suggestions: SuggestionsConfig = Field(default_factory=SuggestionsConfig, description="Follow-up suggestions configuration.")
circuit_breaker: CircuitBreakerConfig = Field(default_factory=CircuitBreakerConfig, description="LLM circuit breaker configuration")
channel_connections: ChannelConnectionsConfig = Field(
default_factory=ChannelConnectionsConfig,
description=format_field_description(
"channel_connections",
field_doc="User-facing IM channel connection configuration.",
),
)
loop_detection: LoopDetectionConfig = Field(default_factory=LoopDetectionConfig, description="Loop detection middleware configuration")
tool_progress: ToolProgressConfig = Field(default_factory=ToolProgressConfig, description="Tool progress state machine middleware configuration")
read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration")
safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration")
auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)")
model_config = ConfigDict(extra="allow")
database: DatabaseConfig = Field(
default_factory=DatabaseConfig,
description=format_field_description(
"database",
field_doc="Unified database backend for run/feedback metadata (memory, sqlite, or postgres).",
),
)
run_events: RunEventsConfig = Field(
default_factory=RunEventsConfig,
description=format_field_description(
"run_events",
field_doc="Run-event store backend (memory for dev, db for production queries, jsonl for lightweight single-node persistence).",
),
)
scheduler: SchedulerConfig = Field(
default_factory=SchedulerConfig,
description=format_field_description(
"scheduler",
field_doc="Scheduled task runtime configuration (background poller for one-time and cron agent runs).",
),
)
checkpointer: CheckpointerConfig | None = Field(
default=None,
description=format_field_description(
"checkpointer",
field_doc="LangGraph state-persistence checkpointer configuration.",
),
)
stream_bridge: StreamBridgeConfig | None = Field(
default=None,
description=format_field_description(
"stream_bridge",
field_doc="Stream bridge connecting agent workers to SSE endpoints.",
),
)
# Name -> config lookup tables, (re)built after validation by
# ``_build_name_indexes``. They make ``get_model_config`` / ``get_tool_config``
# / ``get_tool_group_config`` O(1) instead of an O(n) ``next(...)`` scan per
# call. Private attrs are excluded from serialization.
_models_by_name: dict[str, ModelConfig] = PrivateAttr(default_factory=dict)
_tools_by_name: dict[str, ToolConfig] = PrivateAttr(default_factory=dict)
_tool_groups_by_name: dict[str, ToolGroupConfig] = PrivateAttr(default_factory=dict)
@model_validator(mode="before")
@classmethod
def _drop_null_config_sections(cls, data: Any) -> Any:
"""Treat a present-but-null config section as absent so its default applies.
Commenting out every entry under a top-level YAML key — e.g. ``models:``
(a list) or ``memory:`` (an object), with only comments beneath it as
shipped throughout ``config.example.yaml`` — makes PyYAML parse the value
as ``None``. Without this, the documented ``cp config.example.yaml
config.yaml`` first-run flow crashes with an opaque ``Input should be a
valid list`` / ``valid dictionary`` pydantic error for that section.
Dropping the ``None`` lets each field fall back to its default: list
sections become ``[]`` via ``default_factory=list`` and object sections
get their default config. This generalizes the earlier list-only
handling to every section that defines a default. The ``database``
section is independent and still owned by ``_apply_database_defaults``
(in ``from_file``), which applies concrete defaults beyond null-coercion.
Required sections without a default (``sandbox``) intentionally still
error when null — there is nothing to fall back to.
"""
if isinstance(data, dict):
return {key: value for key, value in data.items() if value is not None}
return data
@classmethod
def resolve_config_path(cls, config_path: str | None = None) -> Path:
"""Resolve the config file path.
Priority:
1. If provided `config_path` argument, use it.
2. If provided `DEER_FLOW_CONFIG_PATH` environment variable, use it.
3. Otherwise, search the caller project root.
4. Finally, search legacy backend/repository-root defaults for monorepo compatibility.
"""
if config_path:
path = Path(config_path)
if not Path.exists(path):
raise FileNotFoundError(f"Config file specified by param `config_path` not found at {path}")
return path
elif os.getenv("DEER_FLOW_CONFIG_PATH"):
path = Path(os.getenv("DEER_FLOW_CONFIG_PATH"))
if not Path.exists(path):
raise FileNotFoundError(f"Config file specified by environment variable `DEER_FLOW_CONFIG_PATH` not found at {path}")
return path
else:
project_config = existing_project_file(("config.yaml",))
if project_config is not None:
return project_config
for path in _legacy_config_candidates():
if path.exists():
return path
raise FileNotFoundError("`config.yaml` file not found in the project root or legacy backend/repository root locations")
@classmethod
def from_file(cls, config_path: str | None = None) -> Self:
"""Load config from YAML file.
See `resolve_config_path` for more details.
Args:
config_path: Path to the config file.
Returns:
AppConfig: The loaded config.
"""
resolved_path = cls.resolve_config_path(config_path)
with open(resolved_path, encoding="utf-8") as f:
config_data = yaml.safe_load(f) or {}
# Check config version before processing
cls._check_config_version(config_data, resolved_path)
config_data = cls.resolve_env_variables(config_data)
cls._apply_database_defaults(config_data)
# Load circuit_breaker config if present
if "circuit_breaker" in config_data:
config_data["circuit_breaker"] = config_data["circuit_breaker"]
# Load extensions config separately (it's in a different file)
extensions_config = ExtensionsConfig.from_file()
config_data["extensions"] = extensions_config.model_dump()
result = cls.model_validate(config_data)
if not result.models:
logger.warning(
"No models are configured in %s. Add at least one entry under `models:` (see the commented examples in config.example.yaml) or run `make setup`.",
resolved_path,
)
acp_agents = cls._validate_acp_agents(config_data.get("acp_agents", {}))
cls._apply_singleton_configs(result, acp_agents)
return result
@classmethod
def _validate_acp_agents(
cls,
config_data: Mapping[str, Mapping[str, object]] | None,
) -> dict[str, ACPAgentConfig]:
if config_data is None:
config_data = {}
return {name: ACPAgentConfig(**cfg) for name, cfg in config_data.items()}
@classmethod
def _apply_singleton_configs(cls, config: Self, acp_agents: dict[str, ACPAgentConfig]) -> None:
from deerflow.config.checkpointer_config import get_checkpointer_config
previous_checkpointer_config = get_checkpointer_config()
load_title_config_from_dict(config.title.model_dump())
load_summarization_config_from_dict(config.summarization.model_dump())
load_memory_config_from_dict(config.memory.model_dump())
load_agents_api_config_from_dict(config.agents_api.model_dump())
load_subagents_config_from_dict(config.subagents.model_dump())
load_tool_search_config_from_dict(config.tool_search.model_dump())
load_guardrails_config_from_dict(config.guardrails.model_dump())
load_checkpointer_config_from_dict(config.checkpointer.model_dump() if config.checkpointer is not None else None)
load_stream_bridge_config_from_dict(config.stream_bridge.model_dump() if config.stream_bridge is not None else None)
load_acp_config_from_dict({name: agent.model_dump() for name, agent in acp_agents.items()})
if previous_checkpointer_config != config.checkpointer:
# These runtime singletons derive their backend from checkpointer config.
# Keep imports local to avoid cycles: both providers import get_app_config.
from deerflow.runtime.checkpointer import reset_checkpointer
from deerflow.runtime.store import reset_store
reset_checkpointer()
reset_store()
@classmethod
def _apply_database_defaults(cls, config_data: dict[str, Any]) -> None:
"""Apply config.yaml defaults for persistence when the section is absent."""
database_config = config_data.get("database")
if database_config is None:
database_config = {}
config_data["database"] = database_config
if not isinstance(database_config, dict):
return
for key, value in CONFIG_FILE_DATABASE_DEFAULTS.items():
database_config.setdefault(key, value)
@classmethod
def _check_config_version(cls, config_data: dict, config_path: Path) -> None:
"""Check if the user's config.yaml is outdated compared to config.example.yaml.
Emits a warning if the user's config_version is lower than the example's.
Missing config_version is treated as version 0 (pre-versioning).
"""
try:
user_version = int(config_data.get("config_version", 0))
except (TypeError, ValueError):
user_version = 0
# Find config.example.yaml by searching config.yaml's directory and its parents
example_path = None
search_dir = config_path.parent
for _ in range(5): # search up to 5 levels
candidate = search_dir / "config.example.yaml"
if candidate.exists():
example_path = candidate
break
parent = search_dir.parent
if parent == search_dir:
break
search_dir = parent
if example_path is None:
return
try:
with open(example_path, encoding="utf-8") as f:
example_data = yaml.safe_load(f)
raw = example_data.get("config_version", 0) if example_data else 0
try:
example_version = int(raw)
except (TypeError, ValueError):
example_version = 0
except Exception:
return
if user_version < example_version:
logger.warning(
"Your config.yaml (version %d) is outdated — the latest version is %d. Run `make config-upgrade` to merge new fields into your config.",
user_version,
example_version,
)
@classmethod
def resolve_env_variables(cls, config: Any) -> Any:
"""Recursively resolve environment variables in the config.
Environment variables are resolved using the `os.getenv` function. Example: $OPENAI_API_KEY
Args:
config: The config to resolve environment variables in.
Returns:
The config with environment variables resolved.
"""
if isinstance(config, str):
if config.startswith("$"):
env_value = os.getenv(config[1:])
if env_value is None:
raise ValueError(f"Environment variable {config[1:]} not found for config value {config}")
return env_value
return config
elif isinstance(config, dict):
return {k: cls.resolve_env_variables(v) for k, v in config.items()}
elif isinstance(config, list):
return [cls.resolve_env_variables(item) for item in config]
return config
@model_validator(mode="after")
def _build_name_indexes(self) -> "AppConfig":
"""Build name -> config lookup tables for O(1) ``get_*_config``.
``get_tool_config`` runs 2-3x per community-tool invocation (e.g.
web_search) and ``get_model_config`` several times per agent build, so
the previous O(n) ``next(...)`` scans sat on hot paths. Rebuilt here so a
config reload (which constructs a fresh ``AppConfig``) refreshes them.
``setdefault`` keeps the first entry on duplicate names, preserving the
prior ``next(...)`` first-match semantics.
"""
models_by_name: dict[str, ModelConfig] = {}
for model in self.models:
models_by_name.setdefault(model.name, model)
tools_by_name: dict[str, ToolConfig] = {}
for tool in self.tools:
tools_by_name.setdefault(tool.name, tool)
tool_groups_by_name: dict[str, ToolGroupConfig] = {}
for group in self.tool_groups:
tool_groups_by_name.setdefault(group.name, group)
self._models_by_name = models_by_name
self._tools_by_name = tools_by_name
self._tool_groups_by_name = tool_groups_by_name
return self
def get_model_config(self, name: str) -> ModelConfig | None:
"""Get the model config by name.
Args:
name: The name of the model to get the config for.
Returns:
The model config if found, otherwise None.
"""
return self._models_by_name.get(name)
def get_tool_config(self, name: str) -> ToolConfig | None:
"""Get the tool config by name.
Args:
name: The name of the tool to get the config for.
Returns:
The tool config if found, otherwise None.
"""
return self._tools_by_name.get(name)
def get_tool_group_config(self, name: str) -> ToolGroupConfig | None:
"""Get the tool group config by name.
Args:
name: The name of the tool group to get the config for.
Returns:
The tool group config if found, otherwise None.
"""
return self._tool_groups_by_name.get(name)
# Compatibility singleton layer for code paths that have not yet been
# migrated to explicit ``AppConfig`` threading. New composition roots should
# prefer constructing ``AppConfig`` once and passing it down directly.
_app_config: AppConfig | None = None
_app_config_path: Path | None = None
_app_config_mtime: float | None = None
_ConfigSignature = tuple[float | None, int | None, str | None]
_app_config_signature: _ConfigSignature | None = None
_app_config_is_custom = False
_current_app_config: ContextVar[AppConfig | None] = ContextVar("deerflow_current_app_config", default=None)
_current_app_config_stack: ContextVar[tuple[AppConfig | None, ...]] = ContextVar("deerflow_current_app_config_stack", default=())
def _get_config_mtime(config_path: Path) -> float | None:
"""Get the modification time of a config file if it exists."""
try:
return config_path.stat().st_mtime
except OSError:
return None
def _get_config_signature(config_path: Path) -> _ConfigSignature | None:
"""Get cache metadata for a config file, including a content digest."""
try:
stat_result = config_path.stat()
except OSError:
return None
digest = hashlib.sha256()
try:
with config_path.open("rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
digest.update(chunk)
except OSError:
return (stat_result.st_mtime, stat_result.st_size, None)
return (stat_result.st_mtime, stat_result.st_size, digest.hexdigest())
def _load_and_cache_app_config(config_path: str | None = None) -> AppConfig:
"""Load config from disk and refresh cache metadata."""
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
resolved_path = AppConfig.resolve_config_path(config_path)
_app_config = AppConfig.from_file(str(resolved_path))
_app_config_path = resolved_path
_app_config_mtime = _get_config_mtime(resolved_path)
_app_config_signature = _get_config_signature(resolved_path)
_app_config_is_custom = False
return _app_config
def get_app_config() -> AppConfig:
"""Get the DeerFlow config instance.
Returns a cached singleton instance and automatically reloads it when the
underlying config file path or content signature changes. Use
`reload_app_config()` to force a reload, or `reset_app_config()` to clear
the cache.
"""
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature
runtime_override = _current_app_config.get()
if runtime_override is not None:
return runtime_override
if _app_config is not None and _app_config_is_custom:
return _app_config
resolved_path = AppConfig.resolve_config_path()
current_mtime = _get_config_mtime(resolved_path)
current_signature = _get_config_signature(resolved_path)
should_reload = _app_config is None or _app_config_path != resolved_path or _app_config_signature != current_signature
if should_reload:
if _app_config_path == resolved_path and _app_config_mtime is not None and current_mtime is not None and _app_config_mtime != current_mtime:
logger.info(
"Config file has been modified (mtime: %s -> %s), reloading AppConfig",
_app_config_mtime,
current_mtime,
)
elif _app_config_path == resolved_path and _app_config_signature != current_signature:
logger.info("Config file content signature changed, reloading AppConfig")
_load_and_cache_app_config(str(resolved_path))
return _app_config
def reload_app_config(config_path: str | None = None) -> AppConfig:
"""Reload the config from file and update the cached instance.
This is useful when the config file has been modified and you want
to pick up the changes without restarting the application.
Args:
config_path: Optional path to config file. If not provided,
uses the default resolution strategy.
Returns:
The newly loaded AppConfig instance.
"""
return _load_and_cache_app_config(config_path)
def reset_app_config() -> None:
"""Reset the cached config instance.
This clears the singleton cache, causing the next call to
`get_app_config()` to reload from file. Useful for testing
or when switching between different configurations.
"""
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
_app_config = None
_app_config_path = None
_app_config_mtime = None
_app_config_signature = None
_app_config_is_custom = False
def set_app_config(config: AppConfig) -> None:
"""Set a custom config instance.
This allows injecting a custom or mock config for testing purposes.
Args:
config: The AppConfig instance to use.
"""
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
_app_config = config
_app_config_path = None
_app_config_mtime = None
_app_config_signature = None
_app_config_is_custom = True
def peek_current_app_config() -> AppConfig | None:
"""Return the runtime-scoped AppConfig override, if one is active."""
return _current_app_config.get()
def push_current_app_config(config: AppConfig) -> None:
"""Push a runtime-scoped AppConfig override for the current execution context."""
stack = _current_app_config_stack.get()
_current_app_config_stack.set(stack + (_current_app_config.get(),))
_current_app_config.set(config)
def pop_current_app_config() -> None:
"""Pop the latest runtime-scoped AppConfig override for the current execution context."""
stack = _current_app_config_stack.get()
if not stack:
_current_app_config.set(None)
return
previous = stack[-1]
_current_app_config_stack.set(stack[:-1])
_current_app_config.set(previous)