mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-08-25 14:18:49 +00:00
* feat(middleware): add structured tool result meta and tool-progress state machine
feat:
- Add tool_result_meta.py: ToolResultMeta dataclass (status/error_type/retryable/
recoverable_by_model/recommended_next_action/source) + normalize_tool_result and
stamp_exception_meta utilities; classifies every ToolMessage regardless of path
- Add ToolProgressMiddleware: per-(thread_id, tool_name) state machine ACTIVE →
WARNED (hint injected as HumanMessage) → BLOCKED (call short-circuited); Jaccard
near-duplicate detection for repeated successful results; auth/config/internal
errors bypass WARNED and go directly to BLOCKED; LRU-bounded thread state store
- Add ToolProgressConfig: all thresholds configurable (stagnation_threshold,
warn_escalation_count, jaccard_similarity_threshold, exempt_tools, etc.);
disabled by default (enabled: false)
- Wire ToolProgressMiddleware as outer wrapper around ToolErrorHandlingMiddleware
in _build_runtime_middlewares so it receives results already carrying
deerflow_tool_meta
fix:
- ToolErrorHandlingMiddleware now calls stamp_exception_meta on exception path and
normalize_tool_result on success path so every ToolMessage carries deerflow_tool_meta
test:
- Add test_tool_result_meta.py: 26 cases covering all classification paths,
stamp_exception_meta, and normalize_tool_result Command passthrough
- Add test_tool_progress_middleware.py: 27 cases including full async paths,
Jaccard duplicate detection, LRU eviction, hint injection, and malformed meta
passthrough
- Extend test_tool_error_handling_middleware.py: middleware ordering invariant and
meta stamping on exception
docs:
- Add tool_progress section to config.example.yaml with all fields and descriptions
- Update CLAUDE.md middleware chain documentation (entries 8-9)
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* fix(middleware): recoverable errors stay WARNED; fix auth keyword shadowing
fix:
- WARNED is terminal for recoverable_by_model=True errors (no_results, not_found,
permission); hint re-injected on each problem call instead of escalating to
BLOCKED, so the model can retry with different parameters (e.g. fresh query,
new URL) without being hard-blocked by a prior stagnation count.
Non-recoverable (rate_limited, transient) still escalate WARNED → BLOCKED
after warn_escalation_count more problems; auth/config/internal remain
immediately BLOCKED.
- Remove bare "api key" keyword from auth classification rule so "no api key
configured" correctly classifies as config (not auth), producing the accurate
block-reason text for the model.
docs:
- CLAUDE.md: document all three ToolProgressMiddleware transition paths
- config.example.yaml: update inline state-machine comment to match new paths
test:
- test_recoverable_errors_stay_warned_indefinitely: WARNED never escalates for
recoverable errors regardless of how many problem calls accumulate
- test_recoverable_error_re_injects_hint_past_escalation: hints continue past
the escalation zone for recoverable errors
- test_no_api_key_is_config_not_auth: regression guard for keyword shadowing fix
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* fix(tool_result_meta): add JSON error extraction and fix source classification
fix:
- Fix non-standard error path: source was "exception" but should be "tool_return"
- Add _extract_json_error_text to isolate JSON error fields from noisy JSON bodies
(e.g. Brave Search {"error": "...", "query": "..."} — query keywords no longer
pollute error classification)
- Add success-path JSON extraction to catch tools that return HTTP 200 with a JSON
error body (status="success" but {"error": "API key not configured"})
- Add _SEMANTIC_ZERO_ERROR_STRINGS frozenset to suppress false positives from tools
that use {"error": "none"} / {"error": "null"} / {"error": "ok"} as success signals
- Document that stamp_exception_meta always overwrites existing TOOL_META_KEY
(exception-derived classification is authoritative over tool return-time stamps)
test:
- Add parametrized regression tests for all semantic-zero error strings
- Add tests for non-standard error path source field
- Add tests for JSON error extraction (nonstd, success-path, numeric, falsy values)
- Correct test comment for test_no_api_key_is_config_not_auth
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* fix(tool_progress_middleware): fix 6 bugs, add terminal guard and structured logging
fix:
- H1: fix exempt_tools empty-set silently ignored — use `is not None` instead of
truthiness check so ToolProgressConfig(exempt_tools=set()) correctly disables all
exemptions
- Fix _get_block_reason creating phantom LRU entries via _get_state (write path);
now uses dict.get + explicit move_to_end on read path only
- Fix _pending memory leak: LRU eviction of _phase_states now synchronously removes
all (evicted_thread, *) keys from _pending
- Fix _assess_and_transition missing terminal guard for blocked state — a recoverable
error result could silently demote blocked → warned in concurrent-race scenarios;
early return preserves terminal semantics
- Fix recent_word_sets window: stored [-5:] but is_near_duplicate only compared [-3:];
align to [-3:] and change type list→tuple (prevents accidental in-place mutation
across dataclasses.replace shallow copies)
- Fix _format_hint missing "success" key and "continue" action: Jaccard near-duplicate
results produced the generic fallback instead of a specific actionable message
feat:
- Add structured state-transition logging (ACTIVE/WARNED/BLOCKED transitions, blocked
intercepts, hint injection debug log)
test:
- Add regression tests for all 6 bug fixes (H1, phantom LRU, pending leak, terminal
guard, window alignment, format_hint near-dup)
- Add Jaccard near-threshold boundary test (7/9 vs 8/9 Jaccard)
- Add production min_words=10 skip test for short content
- Add exempt_tools empty-set and None round-trip tests
- Add _augment_request deduplication test
- Add before_agent current-run preservation test
- Add structured logging tests (WARNED/BLOCKED/ACTIVE/intercepted/debug)
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* chore(config): remove unused backward-compat fields from ToolProgressConfig
Remove max_calls_per_intent and window_size fields that were marked
"Retained for backward compatibility; not used by the current state machine"
when the state machine was introduced. Pydantic v2 ignores unknown fields
by default, so existing config.yaml files with these keys remain valid.
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* fix(tool_progress): address PR review and multi-agent review findings
fix:
- Remove <80-char length gate for partial_success; only _PARTIAL_MARKERS now
- Add word-boundary regex for numeric HTTP codes (401/403/404/500) to avoid
false positives like "500ms" or "4010 rows" triggering hard-block
- Add "task" to default exempt_tools (delegation primitive, not a search tool)
- Remove move_to_end() from _get_block_reason read path; blocked threads were
permanently warm in LRU, starving active threads of eviction slots
- Add _reset_blocked_states in before_agent: scope BLOCKED and WARNED states
to a single run; clear recent_word_sets so stale Jaccard windows don't cause
false near-duplicate detections in the next run
- Compute word_set() lazily (only for success results); cap content at 8192
chars to bound memory and CPU cost on large tool results
- Remove unused retryable field from ToolResultMeta (no consumer existed)
- Add isinstance-based ordering guard and warning log for missing meta
- Fix JSON-without-error-key fallback: use _UNKNOWN_ERROR instead of
classifying incidental field values (e.g. {"user_id": 401} → auth → stop)
- Fix _extract_json_error_text: use json.dumps for dict/list error fields
instead of str() which produced Python repr matching config rules spuriously
- Add "no results found"/"no content found"/"no images found" to _PARTIAL_MARKERS
so success responses with empty results trigger stagnation detection
- Fix immediate-block path to increment consecutive_problems (was left at 0)
- Fix _queue_assessment: skip phantom _pending entries for evicted threads
- Bump config_version 13→16 (upstream added 14/15; tool_progress is additive)
test:
- Update test_short_content_is_partial → test_short_terse_success_is_not_partial
- Add parametrized test_numeric_keyword_word_boundary (8 positive + negative cases)
- Add test_before_agent_resets_blocked_states_for_new_run (strengthened assertions)
- Add test_before_agent_resets_warned_states_for_new_run
- Add test_missing_meta_on_non_exempt_tool_emits_warning
- Add test_middleware_ordering_guard_raises_when_progress_is_inner
- Add test_auth_error_immediately_blocked asserts consecutive_problems == 1
- Add tests for JSON-without-error-key, dict error field, no-results partial_success
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(tool_progress): address second PR review — perf, architecture doc, concurrency note
fix:
- Extract content.lower() once before _PARTIAL_MARKERS check in normalize_tool_message;
previously computed up to 7× per call inside the generator (once per marker)
docs:
- Add division-of-labor paragraph to ToolProgressMiddleware module docstring explaining
coexistence with LoopDetectionMiddleware: result-quality guard (per-tool BLOCK) vs
call-pattern guard (whole-turn hard-stop); no shared state, no double-stop risk
- Add threading.Lock comment explaining why asyncio.Lock is not used (short critical
sections, must also protect sync wrap_tool_call path from subagent executor threads)
- Update backend/CLAUDE.md entry 8 with division-of-labor summary; fix entry 9
(remove stale retryable field reference, add missing recoverable_by_model/source)
test:
- Add test_tool_progress_and_loop_detection_coexist_without_interfering: drives both
middlewares to WARNED state simultaneously, verifies independent state, independent
hint queues, and no cross-contamination; uses snapshot copy for final assertion
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(tool_progress): reset all tool states at run boundary; fix semantic-zero test validity
fix:
- _reset_run_states (formerly _reset_blocked_states) drops the phase filter and
resets all tracked (thread, tool) pairs unconditionally at before_agent; ACTIVE
tools with sub-threshold consecutive_problems or cached recent_word_sets no longer
bleed into the next run, preventing spurious WARNED transitions on clean R2 calls
- test_normalize_json_semantic_zero_error_string_not_treated_as_error: replace
{error_value!r} f-string (produces invalid JSON with single quotes) with
json.dumps so _extract_json_error_text actually parses the payload and the
_SEMANTIC_ZERO_ERROR_STRINGS guard is exercised, not bypassed at json.loads
test:
- add test_before_agent_resets_active_state_consecutive_problems_and_word_sets to
lock the ACTIVE-phase run-boundary reset: drives tool to active/cp=1/ws≠() in R1,
asserts both fields are zero/empty after before_agent fires for R2
* docs(tool_progress): document intentional per-run reset vs LoopDetection thread-scoped retention
Addresses reviewer observation in PR #3601 that _reset_run_states diverges
from LoopDetectionMiddleware's cross-run scoping policy without explanation.
Expands the _reset_run_states docstring to record the intentional design
choice: ToolProgressMiddleware resets per-run because result-quality errors
(rate_limited, transient) are time-bound and may resolve between turns —
retaining stale counters would risk false-positive BLOCKED calls.
LoopDetectionMiddleware retains history across runs because call-pattern
loops are time-invariant. The divergence is by design, not oversight.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(middleware): restore ReadBeforeWriteMiddleware as outermost write gate
A merge conflict resolution had accidentally placed ReadBeforeWriteMiddleware
after ToolErrorHandlingMiddleware (inner), reversing the original intent from
b81334cc where it was the outermost write gate before ToolErrorHandling.
fix:
- Restore ReadBeforeWriteMiddleware to outer position: ReadBeforeWrite →
ToolProgress → ToolErrorHandling. Blocked writes now return immediately
without consuming a ToolProgress slot.
- Add normalize_tool_result call on blocked ToolMessages so they carry
deerflow_tool_meta (recoverable_by_model=True) even though they bypass
ToolErrorHandlingMiddleware.
test:
- Add test_blocked_write_has_deerflow_tool_meta (sync + async) to lock the
normalize_tool_result behavior on blocked writes.
- Fix chain order assertions in TestChainWiring and
test_build_lead_runtime_middlewares_chain_order_matches_agents_md.
docs:
- Renumber AGENTS.md items: 10→ReadBeforeWrite, 11→ToolProgress,
12→ToolErrorHandling; update descriptions to reflect outermost-gate design.
- Fix stale cross-reference: LoopDetectionMiddleware (item 23) → (item 25).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
653 lines
29 KiB
Python
653 lines
29 KiB
Python
import hashlib
|
|
import logging
|
|
import os
|
|
from collections.abc import Mapping
|
|
from contextvars import ContextVar
|
|
from pathlib import Path
|
|
from typing import Any, Literal, Self
|
|
|
|
import yaml
|
|
from dotenv import load_dotenv
|
|
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, model_validator
|
|
|
|
from deerflow.config.acp_config import ACPAgentConfig, load_acp_config_from_dict
|
|
from deerflow.config.agents_api_config import AgentsApiConfig, load_agents_api_config_from_dict
|
|
from deerflow.config.auth_config import AuthAppConfig
|
|
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
|
|
from deerflow.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
|
|
from deerflow.config.database_config import DatabaseConfig
|
|
from deerflow.config.extensions_config import ExtensionsConfig
|
|
from deerflow.config.guardrails_config import GuardrailsConfig, load_guardrails_config_from_dict
|
|
from deerflow.config.loop_detection_config import LoopDetectionConfig
|
|
from deerflow.config.memory_config import MemoryConfig, load_memory_config_from_dict
|
|
from deerflow.config.model_config import ModelConfig
|
|
from deerflow.config.read_before_write_config import ReadBeforeWriteConfig
|
|
from deerflow.config.reload_boundary import format_field_description
|
|
from deerflow.config.run_events_config import RunEventsConfig
|
|
from deerflow.config.runtime_paths import existing_project_file
|
|
from deerflow.config.safety_finish_reason_config import SafetyFinishReasonConfig
|
|
from deerflow.config.sandbox_config import SandboxConfig
|
|
from deerflow.config.scheduler_config import SchedulerConfig
|
|
from deerflow.config.skill_evolution_config import SkillEvolutionConfig
|
|
from deerflow.config.skills_config import SkillsConfig
|
|
from deerflow.config.stream_bridge_config import StreamBridgeConfig, load_stream_bridge_config_from_dict
|
|
from deerflow.config.subagents_config import SubagentsAppConfig, load_subagents_config_from_dict
|
|
from deerflow.config.suggestions_config import SuggestionsConfig
|
|
from deerflow.config.summarization_config import SummarizationConfig, load_summarization_config_from_dict
|
|
from deerflow.config.title_config import TitleConfig, load_title_config_from_dict
|
|
from deerflow.config.token_budget_config import TokenBudgetConfig
|
|
from deerflow.config.token_usage_config import TokenUsageConfig
|
|
from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
|
|
from deerflow.config.tool_output_config import ToolOutputConfig
|
|
from deerflow.config.tool_progress_config import ToolProgressConfig
|
|
from deerflow.config.tool_search_config import ToolSearchConfig, load_tool_search_config_from_dict
|
|
|
|
load_dotenv()
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
CONFIG_FILE_DATABASE_DEFAULTS = {
|
|
"backend": "sqlite",
|
|
"sqlite_dir": ".deer-flow/data",
|
|
}
|
|
|
|
|
|
class CircuitBreakerConfig(BaseModel):
|
|
"""Configuration for the LLM Circuit Breaker."""
|
|
|
|
failure_threshold: int = Field(default=5, description="Number of consecutive failures before tripping the circuit")
|
|
recovery_timeout_sec: int = Field(default=60, description="Time in seconds before attempting to recover the circuit")
|
|
|
|
|
|
class LoggingEnhanceConfig(BaseModel):
|
|
"""Request trace logging enhancement settings."""
|
|
|
|
enabled: bool = Field(default=False, description="Enable request-level trace ids in Gateway response headers and log records.")
|
|
format: Literal["text", "json"] = Field(default="text", description="Enhanced log output format.")
|
|
|
|
|
|
class LoggingConfig(BaseModel):
|
|
"""Logging configuration."""
|
|
|
|
enhance: LoggingEnhanceConfig = Field(default_factory=LoggingEnhanceConfig, description="Request trace correlation logging settings.")
|
|
|
|
|
|
def is_trace_correlation_enabled(config: Any) -> bool:
|
|
"""Return ``True`` when ``logging.enhance.enabled`` is set on *config*.
|
|
|
|
Single source of truth for the request-trace-correlation gate, shared by
|
|
the Gateway ``TraceMiddleware`` and the embedded ``DeerFlowClient`` so
|
|
the two entry points cannot drift on when ``deerflow_trace_id`` is
|
|
emitted (Langfuse metadata) and when a request-level trace id is bound
|
|
at all. Accepts any object exposing ``logging.enhance.enabled`` via
|
|
``getattr`` chains (``AppConfig``, ``SimpleNamespace`` fixtures, etc.);
|
|
missing intermediate attributes silently degrade to ``False``.
|
|
"""
|
|
logging_config = getattr(config, "logging", None)
|
|
enhance = getattr(logging_config, "enhance", None)
|
|
return bool(getattr(enhance, "enabled", False))
|
|
|
|
|
|
def _legacy_config_candidates() -> tuple[Path, ...]:
|
|
"""Return source-tree config.yaml locations for monorepo compatibility."""
|
|
backend_dir = Path(__file__).resolve().parents[4]
|
|
repo_root = backend_dir.parent
|
|
return (backend_dir / "config.yaml", repo_root / "config.yaml")
|
|
|
|
|
|
def logging_level_from_config(name: str | None) -> int:
|
|
"""Map ``config.yaml`` ``log_level`` string to a :mod:`logging` level constant."""
|
|
mapping = logging.getLevelNamesMapping()
|
|
return mapping.get((name or "info").strip().upper(), logging.INFO)
|
|
|
|
|
|
def apply_logging_level(name: str | None) -> None:
|
|
"""Resolve *name* to a logging level and apply it to the ``deerflow``/``app`` logger hierarchies.
|
|
|
|
Only the ``deerflow`` and ``app`` logger levels are changed so that
|
|
third-party library verbosity (e.g. uvicorn, sqlalchemy) is not
|
|
affected. Root handler levels are lowered (never raised) so that
|
|
messages from the configured loggers can propagate through without
|
|
being filtered, while preserving handler thresholds that may be
|
|
intentionally restrictive for third-party log output.
|
|
"""
|
|
level = logging_level_from_config(name)
|
|
for logger_name in ("deerflow", "app"):
|
|
logging.getLogger(logger_name).setLevel(level)
|
|
for handler in logging.root.handlers:
|
|
if level < handler.level:
|
|
handler.setLevel(level)
|
|
|
|
|
|
class AppConfig(BaseModel):
|
|
"""Config for the DeerFlow application"""
|
|
|
|
log_level: str = Field(
|
|
default="info",
|
|
description=format_field_description(
|
|
"log_level",
|
|
field_doc="Logging level for deerflow and app modules (debug/info/warning/error); third-party libraries are not affected.",
|
|
),
|
|
)
|
|
logging: LoggingConfig = Field(
|
|
default_factory=LoggingConfig,
|
|
description=format_field_description(
|
|
"logging",
|
|
field_doc="Structured logging and request trace correlation settings.",
|
|
),
|
|
)
|
|
token_usage: TokenUsageConfig = Field(default_factory=TokenUsageConfig, description="Token usage tracking configuration")
|
|
token_budget: TokenBudgetConfig = Field(default_factory=TokenBudgetConfig, description="Token Budget tracking and limits configuration.")
|
|
max_recursion_limit: int = Field(
|
|
default=1000,
|
|
ge=1,
|
|
description="Hard server-side ceiling for a client-supplied run recursion_limit. Client values above this are clamped; prevents runaway LangGraph super-steps (LLM cost / DoS).",
|
|
)
|
|
models: list[ModelConfig] = Field(default_factory=list, description="Available models")
|
|
sandbox: SandboxConfig = Field(
|
|
description=format_field_description(
|
|
"sandbox",
|
|
field_doc="Sandbox provider configuration (local filesystem or Docker-based aio sandbox).",
|
|
),
|
|
)
|
|
tools: list[ToolConfig] = Field(default_factory=list, description="Available tools")
|
|
tool_groups: list[ToolGroupConfig] = Field(default_factory=list, description="Available tool groups")
|
|
skills: SkillsConfig = Field(default_factory=SkillsConfig, description="Skills configuration")
|
|
skill_evolution: SkillEvolutionConfig = Field(default_factory=SkillEvolutionConfig, description="Agent-managed skill evolution configuration")
|
|
extensions: ExtensionsConfig = Field(default_factory=ExtensionsConfig, description="Extensions configuration (MCP servers and skills state)")
|
|
tool_output: ToolOutputConfig = Field(default_factory=ToolOutputConfig, description="Tool output budget protection configuration")
|
|
tool_search: ToolSearchConfig = Field(default_factory=ToolSearchConfig, description="Tool search / deferred loading configuration")
|
|
title: TitleConfig = Field(default_factory=TitleConfig, description="Automatic title generation configuration")
|
|
summarization: SummarizationConfig = Field(default_factory=SummarizationConfig, description="Conversation summarization configuration")
|
|
memory: MemoryConfig = Field(default_factory=MemoryConfig, description="Memory subsystem configuration")
|
|
agents_api: AgentsApiConfig = Field(default_factory=AgentsApiConfig, description="Custom-agent management API configuration")
|
|
acp_agents: dict[str, ACPAgentConfig] = Field(default_factory=dict, description="ACP-compatible agent configuration")
|
|
subagents: SubagentsAppConfig = Field(default_factory=SubagentsAppConfig, description="Subagent runtime configuration")
|
|
guardrails: GuardrailsConfig = Field(default_factory=GuardrailsConfig, description="Guardrail middleware configuration")
|
|
suggestions: SuggestionsConfig = Field(default_factory=SuggestionsConfig, description="Follow-up suggestions configuration.")
|
|
circuit_breaker: CircuitBreakerConfig = Field(default_factory=CircuitBreakerConfig, description="LLM circuit breaker configuration")
|
|
channel_connections: ChannelConnectionsConfig = Field(
|
|
default_factory=ChannelConnectionsConfig,
|
|
description=format_field_description(
|
|
"channel_connections",
|
|
field_doc="User-facing IM channel connection configuration.",
|
|
),
|
|
)
|
|
loop_detection: LoopDetectionConfig = Field(default_factory=LoopDetectionConfig, description="Loop detection middleware configuration")
|
|
tool_progress: ToolProgressConfig = Field(default_factory=ToolProgressConfig, description="Tool progress state machine middleware configuration")
|
|
read_before_write: ReadBeforeWriteConfig = Field(default_factory=ReadBeforeWriteConfig, description="Read-before-write file gate middleware configuration")
|
|
safety_finish_reason: SafetyFinishReasonConfig = Field(default_factory=SafetyFinishReasonConfig, description="Provider safety-filter finish_reason interception middleware configuration")
|
|
auth: AuthAppConfig = Field(default_factory=AuthAppConfig, description="Authentication configuration (local + OIDC SSO)")
|
|
model_config = ConfigDict(extra="allow")
|
|
database: DatabaseConfig = Field(
|
|
default_factory=DatabaseConfig,
|
|
description=format_field_description(
|
|
"database",
|
|
field_doc="Unified database backend for run/feedback metadata (memory, sqlite, or postgres).",
|
|
),
|
|
)
|
|
run_events: RunEventsConfig = Field(
|
|
default_factory=RunEventsConfig,
|
|
description=format_field_description(
|
|
"run_events",
|
|
field_doc="Run-event store backend (memory for dev, db for production queries, jsonl for lightweight single-node persistence).",
|
|
),
|
|
)
|
|
scheduler: SchedulerConfig = Field(
|
|
default_factory=SchedulerConfig,
|
|
description=format_field_description(
|
|
"scheduler",
|
|
field_doc="Scheduled task runtime configuration (background poller for one-time and cron agent runs).",
|
|
),
|
|
)
|
|
checkpointer: CheckpointerConfig | None = Field(
|
|
default=None,
|
|
description=format_field_description(
|
|
"checkpointer",
|
|
field_doc="LangGraph state-persistence checkpointer configuration.",
|
|
),
|
|
)
|
|
stream_bridge: StreamBridgeConfig | None = Field(
|
|
default=None,
|
|
description=format_field_description(
|
|
"stream_bridge",
|
|
field_doc="Stream bridge connecting agent workers to SSE endpoints.",
|
|
),
|
|
)
|
|
|
|
# Name -> config lookup tables, (re)built after validation by
|
|
# ``_build_name_indexes``. They make ``get_model_config`` / ``get_tool_config``
|
|
# / ``get_tool_group_config`` O(1) instead of an O(n) ``next(...)`` scan per
|
|
# call. Private attrs are excluded from serialization.
|
|
_models_by_name: dict[str, ModelConfig] = PrivateAttr(default_factory=dict)
|
|
_tools_by_name: dict[str, ToolConfig] = PrivateAttr(default_factory=dict)
|
|
_tool_groups_by_name: dict[str, ToolGroupConfig] = PrivateAttr(default_factory=dict)
|
|
|
|
@model_validator(mode="before")
|
|
@classmethod
|
|
def _drop_null_config_sections(cls, data: Any) -> Any:
|
|
"""Treat a present-but-null config section as absent so its default applies.
|
|
|
|
Commenting out every entry under a top-level YAML key — e.g. ``models:``
|
|
(a list) or ``memory:`` (an object), with only comments beneath it as
|
|
shipped throughout ``config.example.yaml`` — makes PyYAML parse the value
|
|
as ``None``. Without this, the documented ``cp config.example.yaml
|
|
config.yaml`` first-run flow crashes with an opaque ``Input should be a
|
|
valid list`` / ``valid dictionary`` pydantic error for that section.
|
|
|
|
Dropping the ``None`` lets each field fall back to its default: list
|
|
sections become ``[]`` via ``default_factory=list`` and object sections
|
|
get their default config. This generalizes the earlier list-only
|
|
handling to every section that defines a default. The ``database``
|
|
section is independent and still owned by ``_apply_database_defaults``
|
|
(in ``from_file``), which applies concrete defaults beyond null-coercion.
|
|
Required sections without a default (``sandbox``) intentionally still
|
|
error when null — there is nothing to fall back to.
|
|
"""
|
|
if isinstance(data, dict):
|
|
return {key: value for key, value in data.items() if value is not None}
|
|
return data
|
|
|
|
@classmethod
|
|
def resolve_config_path(cls, config_path: str | None = None) -> Path:
|
|
"""Resolve the config file path.
|
|
|
|
Priority:
|
|
1. If provided `config_path` argument, use it.
|
|
2. If provided `DEER_FLOW_CONFIG_PATH` environment variable, use it.
|
|
3. Otherwise, search the caller project root.
|
|
4. Finally, search legacy backend/repository-root defaults for monorepo compatibility.
|
|
"""
|
|
if config_path:
|
|
path = Path(config_path)
|
|
if not Path.exists(path):
|
|
raise FileNotFoundError(f"Config file specified by param `config_path` not found at {path}")
|
|
return path
|
|
elif os.getenv("DEER_FLOW_CONFIG_PATH"):
|
|
path = Path(os.getenv("DEER_FLOW_CONFIG_PATH"))
|
|
if not Path.exists(path):
|
|
raise FileNotFoundError(f"Config file specified by environment variable `DEER_FLOW_CONFIG_PATH` not found at {path}")
|
|
return path
|
|
else:
|
|
project_config = existing_project_file(("config.yaml",))
|
|
if project_config is not None:
|
|
return project_config
|
|
|
|
for path in _legacy_config_candidates():
|
|
if path.exists():
|
|
return path
|
|
raise FileNotFoundError("`config.yaml` file not found in the project root or legacy backend/repository root locations")
|
|
|
|
@classmethod
|
|
def from_file(cls, config_path: str | None = None) -> Self:
|
|
"""Load config from YAML file.
|
|
|
|
See `resolve_config_path` for more details.
|
|
|
|
Args:
|
|
config_path: Path to the config file.
|
|
|
|
Returns:
|
|
AppConfig: The loaded config.
|
|
"""
|
|
resolved_path = cls.resolve_config_path(config_path)
|
|
with open(resolved_path, encoding="utf-8") as f:
|
|
config_data = yaml.safe_load(f) or {}
|
|
|
|
# Check config version before processing
|
|
cls._check_config_version(config_data, resolved_path)
|
|
|
|
config_data = cls.resolve_env_variables(config_data)
|
|
cls._apply_database_defaults(config_data)
|
|
|
|
# Load circuit_breaker config if present
|
|
if "circuit_breaker" in config_data:
|
|
config_data["circuit_breaker"] = config_data["circuit_breaker"]
|
|
|
|
# Load extensions config separately (it's in a different file)
|
|
extensions_config = ExtensionsConfig.from_file()
|
|
config_data["extensions"] = extensions_config.model_dump()
|
|
|
|
result = cls.model_validate(config_data)
|
|
if not result.models:
|
|
logger.warning(
|
|
"No models are configured in %s. Add at least one entry under `models:` (see the commented examples in config.example.yaml) or run `make setup`.",
|
|
resolved_path,
|
|
)
|
|
acp_agents = cls._validate_acp_agents(config_data.get("acp_agents", {}))
|
|
cls._apply_singleton_configs(result, acp_agents)
|
|
return result
|
|
|
|
@classmethod
|
|
def _validate_acp_agents(
|
|
cls,
|
|
config_data: Mapping[str, Mapping[str, object]] | None,
|
|
) -> dict[str, ACPAgentConfig]:
|
|
if config_data is None:
|
|
config_data = {}
|
|
return {name: ACPAgentConfig(**cfg) for name, cfg in config_data.items()}
|
|
|
|
@classmethod
|
|
def _apply_singleton_configs(cls, config: Self, acp_agents: dict[str, ACPAgentConfig]) -> None:
|
|
from deerflow.config.checkpointer_config import get_checkpointer_config
|
|
|
|
previous_checkpointer_config = get_checkpointer_config()
|
|
|
|
load_title_config_from_dict(config.title.model_dump())
|
|
load_summarization_config_from_dict(config.summarization.model_dump())
|
|
load_memory_config_from_dict(config.memory.model_dump())
|
|
load_agents_api_config_from_dict(config.agents_api.model_dump())
|
|
load_subagents_config_from_dict(config.subagents.model_dump())
|
|
load_tool_search_config_from_dict(config.tool_search.model_dump())
|
|
load_guardrails_config_from_dict(config.guardrails.model_dump())
|
|
load_checkpointer_config_from_dict(config.checkpointer.model_dump() if config.checkpointer is not None else None)
|
|
load_stream_bridge_config_from_dict(config.stream_bridge.model_dump() if config.stream_bridge is not None else None)
|
|
load_acp_config_from_dict({name: agent.model_dump() for name, agent in acp_agents.items()})
|
|
|
|
if previous_checkpointer_config != config.checkpointer:
|
|
# These runtime singletons derive their backend from checkpointer config.
|
|
# Keep imports local to avoid cycles: both providers import get_app_config.
|
|
from deerflow.runtime.checkpointer import reset_checkpointer
|
|
from deerflow.runtime.store import reset_store
|
|
|
|
reset_checkpointer()
|
|
reset_store()
|
|
|
|
@classmethod
|
|
def _apply_database_defaults(cls, config_data: dict[str, Any]) -> None:
|
|
"""Apply config.yaml defaults for persistence when the section is absent."""
|
|
database_config = config_data.get("database")
|
|
if database_config is None:
|
|
database_config = {}
|
|
config_data["database"] = database_config
|
|
if not isinstance(database_config, dict):
|
|
return
|
|
for key, value in CONFIG_FILE_DATABASE_DEFAULTS.items():
|
|
database_config.setdefault(key, value)
|
|
|
|
@classmethod
|
|
def _check_config_version(cls, config_data: dict, config_path: Path) -> None:
|
|
"""Check if the user's config.yaml is outdated compared to config.example.yaml.
|
|
|
|
Emits a warning if the user's config_version is lower than the example's.
|
|
Missing config_version is treated as version 0 (pre-versioning).
|
|
"""
|
|
try:
|
|
user_version = int(config_data.get("config_version", 0))
|
|
except (TypeError, ValueError):
|
|
user_version = 0
|
|
|
|
# Find config.example.yaml by searching config.yaml's directory and its parents
|
|
example_path = None
|
|
search_dir = config_path.parent
|
|
for _ in range(5): # search up to 5 levels
|
|
candidate = search_dir / "config.example.yaml"
|
|
if candidate.exists():
|
|
example_path = candidate
|
|
break
|
|
parent = search_dir.parent
|
|
if parent == search_dir:
|
|
break
|
|
search_dir = parent
|
|
if example_path is None:
|
|
return
|
|
|
|
try:
|
|
with open(example_path, encoding="utf-8") as f:
|
|
example_data = yaml.safe_load(f)
|
|
raw = example_data.get("config_version", 0) if example_data else 0
|
|
try:
|
|
example_version = int(raw)
|
|
except (TypeError, ValueError):
|
|
example_version = 0
|
|
except Exception:
|
|
return
|
|
|
|
if user_version < example_version:
|
|
logger.warning(
|
|
"Your config.yaml (version %d) is outdated — the latest version is %d. Run `make config-upgrade` to merge new fields into your config.",
|
|
user_version,
|
|
example_version,
|
|
)
|
|
|
|
@classmethod
|
|
def resolve_env_variables(cls, config: Any) -> Any:
|
|
"""Recursively resolve environment variables in the config.
|
|
|
|
Environment variables are resolved using the `os.getenv` function. Example: $OPENAI_API_KEY
|
|
|
|
Args:
|
|
config: The config to resolve environment variables in.
|
|
|
|
Returns:
|
|
The config with environment variables resolved.
|
|
"""
|
|
if isinstance(config, str):
|
|
if config.startswith("$"):
|
|
env_value = os.getenv(config[1:])
|
|
if env_value is None:
|
|
raise ValueError(f"Environment variable {config[1:]} not found for config value {config}")
|
|
return env_value
|
|
return config
|
|
elif isinstance(config, dict):
|
|
return {k: cls.resolve_env_variables(v) for k, v in config.items()}
|
|
elif isinstance(config, list):
|
|
return [cls.resolve_env_variables(item) for item in config]
|
|
return config
|
|
|
|
@model_validator(mode="after")
|
|
def _build_name_indexes(self) -> "AppConfig":
|
|
"""Build name -> config lookup tables for O(1) ``get_*_config``.
|
|
|
|
``get_tool_config`` runs 2-3x per community-tool invocation (e.g.
|
|
web_search) and ``get_model_config`` several times per agent build, so
|
|
the previous O(n) ``next(...)`` scans sat on hot paths. Rebuilt here so a
|
|
config reload (which constructs a fresh ``AppConfig``) refreshes them.
|
|
``setdefault`` keeps the first entry on duplicate names, preserving the
|
|
prior ``next(...)`` first-match semantics.
|
|
"""
|
|
models_by_name: dict[str, ModelConfig] = {}
|
|
for model in self.models:
|
|
models_by_name.setdefault(model.name, model)
|
|
tools_by_name: dict[str, ToolConfig] = {}
|
|
for tool in self.tools:
|
|
tools_by_name.setdefault(tool.name, tool)
|
|
tool_groups_by_name: dict[str, ToolGroupConfig] = {}
|
|
for group in self.tool_groups:
|
|
tool_groups_by_name.setdefault(group.name, group)
|
|
self._models_by_name = models_by_name
|
|
self._tools_by_name = tools_by_name
|
|
self._tool_groups_by_name = tool_groups_by_name
|
|
return self
|
|
|
|
def get_model_config(self, name: str) -> ModelConfig | None:
|
|
"""Get the model config by name.
|
|
|
|
Args:
|
|
name: The name of the model to get the config for.
|
|
|
|
Returns:
|
|
The model config if found, otherwise None.
|
|
"""
|
|
return self._models_by_name.get(name)
|
|
|
|
def get_tool_config(self, name: str) -> ToolConfig | None:
|
|
"""Get the tool config by name.
|
|
|
|
Args:
|
|
name: The name of the tool to get the config for.
|
|
|
|
Returns:
|
|
The tool config if found, otherwise None.
|
|
"""
|
|
return self._tools_by_name.get(name)
|
|
|
|
def get_tool_group_config(self, name: str) -> ToolGroupConfig | None:
|
|
"""Get the tool group config by name.
|
|
|
|
Args:
|
|
name: The name of the tool group to get the config for.
|
|
|
|
Returns:
|
|
The tool group config if found, otherwise None.
|
|
"""
|
|
return self._tool_groups_by_name.get(name)
|
|
|
|
|
|
# Compatibility singleton layer for code paths that have not yet been
|
|
# migrated to explicit ``AppConfig`` threading. New composition roots should
|
|
# prefer constructing ``AppConfig`` once and passing it down directly.
|
|
_app_config: AppConfig | None = None
|
|
_app_config_path: Path | None = None
|
|
_app_config_mtime: float | None = None
|
|
_ConfigSignature = tuple[float | None, int | None, str | None]
|
|
_app_config_signature: _ConfigSignature | None = None
|
|
_app_config_is_custom = False
|
|
_current_app_config: ContextVar[AppConfig | None] = ContextVar("deerflow_current_app_config", default=None)
|
|
_current_app_config_stack: ContextVar[tuple[AppConfig | None, ...]] = ContextVar("deerflow_current_app_config_stack", default=())
|
|
|
|
|
|
def _get_config_mtime(config_path: Path) -> float | None:
|
|
"""Get the modification time of a config file if it exists."""
|
|
try:
|
|
return config_path.stat().st_mtime
|
|
except OSError:
|
|
return None
|
|
|
|
|
|
def _get_config_signature(config_path: Path) -> _ConfigSignature | None:
|
|
"""Get cache metadata for a config file, including a content digest."""
|
|
try:
|
|
stat_result = config_path.stat()
|
|
except OSError:
|
|
return None
|
|
|
|
digest = hashlib.sha256()
|
|
try:
|
|
with config_path.open("rb") as f:
|
|
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
|
digest.update(chunk)
|
|
except OSError:
|
|
return (stat_result.st_mtime, stat_result.st_size, None)
|
|
|
|
return (stat_result.st_mtime, stat_result.st_size, digest.hexdigest())
|
|
|
|
|
|
def _load_and_cache_app_config(config_path: str | None = None) -> AppConfig:
|
|
"""Load config from disk and refresh cache metadata."""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
|
|
resolved_path = AppConfig.resolve_config_path(config_path)
|
|
_app_config = AppConfig.from_file(str(resolved_path))
|
|
_app_config_path = resolved_path
|
|
_app_config_mtime = _get_config_mtime(resolved_path)
|
|
_app_config_signature = _get_config_signature(resolved_path)
|
|
_app_config_is_custom = False
|
|
return _app_config
|
|
|
|
|
|
def get_app_config() -> AppConfig:
|
|
"""Get the DeerFlow config instance.
|
|
|
|
Returns a cached singleton instance and automatically reloads it when the
|
|
underlying config file path or content signature changes. Use
|
|
`reload_app_config()` to force a reload, or `reset_app_config()` to clear
|
|
the cache.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature
|
|
|
|
runtime_override = _current_app_config.get()
|
|
if runtime_override is not None:
|
|
return runtime_override
|
|
|
|
if _app_config is not None and _app_config_is_custom:
|
|
return _app_config
|
|
|
|
resolved_path = AppConfig.resolve_config_path()
|
|
current_mtime = _get_config_mtime(resolved_path)
|
|
current_signature = _get_config_signature(resolved_path)
|
|
|
|
should_reload = _app_config is None or _app_config_path != resolved_path or _app_config_signature != current_signature
|
|
if should_reload:
|
|
if _app_config_path == resolved_path and _app_config_mtime is not None and current_mtime is not None and _app_config_mtime != current_mtime:
|
|
logger.info(
|
|
"Config file has been modified (mtime: %s -> %s), reloading AppConfig",
|
|
_app_config_mtime,
|
|
current_mtime,
|
|
)
|
|
elif _app_config_path == resolved_path and _app_config_signature != current_signature:
|
|
logger.info("Config file content signature changed, reloading AppConfig")
|
|
_load_and_cache_app_config(str(resolved_path))
|
|
return _app_config
|
|
|
|
|
|
def reload_app_config(config_path: str | None = None) -> AppConfig:
|
|
"""Reload the config from file and update the cached instance.
|
|
|
|
This is useful when the config file has been modified and you want
|
|
to pick up the changes without restarting the application.
|
|
|
|
Args:
|
|
config_path: Optional path to config file. If not provided,
|
|
uses the default resolution strategy.
|
|
|
|
Returns:
|
|
The newly loaded AppConfig instance.
|
|
"""
|
|
return _load_and_cache_app_config(config_path)
|
|
|
|
|
|
def reset_app_config() -> None:
|
|
"""Reset the cached config instance.
|
|
|
|
This clears the singleton cache, causing the next call to
|
|
`get_app_config()` to reload from file. Useful for testing
|
|
or when switching between different configurations.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
_app_config = None
|
|
_app_config_path = None
|
|
_app_config_mtime = None
|
|
_app_config_signature = None
|
|
_app_config_is_custom = False
|
|
|
|
|
|
def set_app_config(config: AppConfig) -> None:
|
|
"""Set a custom config instance.
|
|
|
|
This allows injecting a custom or mock config for testing purposes.
|
|
|
|
Args:
|
|
config: The AppConfig instance to use.
|
|
"""
|
|
global _app_config, _app_config_path, _app_config_mtime, _app_config_signature, _app_config_is_custom
|
|
_app_config = config
|
|
_app_config_path = None
|
|
_app_config_mtime = None
|
|
_app_config_signature = None
|
|
_app_config_is_custom = True
|
|
|
|
|
|
def peek_current_app_config() -> AppConfig | None:
|
|
"""Return the runtime-scoped AppConfig override, if one is active."""
|
|
return _current_app_config.get()
|
|
|
|
|
|
def push_current_app_config(config: AppConfig) -> None:
|
|
"""Push a runtime-scoped AppConfig override for the current execution context."""
|
|
stack = _current_app_config_stack.get()
|
|
_current_app_config_stack.set(stack + (_current_app_config.get(),))
|
|
_current_app_config.set(config)
|
|
|
|
|
|
def pop_current_app_config() -> None:
|
|
"""Pop the latest runtime-scoped AppConfig override for the current execution context."""
|
|
stack = _current_app_config_stack.get()
|
|
if not stack:
|
|
_current_app_config.set(None)
|
|
return
|
|
previous = stack[-1]
|
|
_current_app_config_stack.set(stack[:-1])
|
|
_current_app_config.set(previous)
|