diff --git a/README.md b/README.md index 7067aad3c..8df044a74 100644 --- a/README.md +++ b/README.md @@ -291,6 +291,13 @@ single-label cluster hosts, and Docker/Podman internal hostnames do not inherit honor environment proxy settings. Backend processes automatically pick up `config.yaml` changes on the next config access, so model metadata updates do not require a manual restart during development. + +Gateway runs use the top-level `recursion_limit` in `config.yaml` when an API +request does not provide one. The default is `100`; valid per-request values +take precedence, and `max_recursion_limit` (default `1000`) caps both. Changes +apply to the next run without restarting the Gateway. This top-level setting +applies to Gateway API runs; IM channel and embedded `DeerFlowClient` runs +retain their own defaults and per-call override paths. The checkpoint storage settings `database.checkpoint_channel_mode` and `database.checkpoint_delta.snapshot_frequency` (default `10`) are exceptions: both are frozen when the process first builds an agent (including through diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 3b4f29324..4b7de7560 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -2,7 +2,7 @@ ## Project Overview -DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments. +The backend runs a LangGraph-based super agent with sandbox execution, persistent memory, subagent delegation, and extensible tools in isolated per-thread environments. **Architecture**: - **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime diff --git a/backend/app/gateway/AGENTS.md b/backend/app/gateway/AGENTS.md index fe07816ea..0af656549 100644 --- a/backend/app/gateway/AGENTS.md +++ b/backend/app/gateway/AGENTS.md @@ -2,6 +2,11 @@ FastAPI listens on port 8001; health: `GET /health` (liveness) and `GET /health/ready` (readiness; concurrently probes the ORM engine behind `database:` plus the effective LangGraph checkpointer/Store backend - the legacy `checkpointer:` section, otherwise derived from `database:`, resolved from the startup config snapshot recorded on `app.state` - beneath a single bounded deadline, with connection-opening probes serialized behind a strict per-process gate, 503 while either is unreachable or the startup backend cannot be resolved, `not_configured` for process-local backends such as `backend=memory`). Set `GATEWAY_ENABLE_DOCS=false` to disable the default `/docs`, `/redoc`, and `/openapi.json` endpoints. +`build_run_config()` resolves the default LangGraph super-step budget from the +hot-reloaded top-level `recursion_limit` setting. A valid request-level value +takes precedence; invalid values fall back to that default, and +`max_recursion_limit` caps both sources. + Durable MCP notifications use internal Agent runs. Keep their trusted delivery instruction outside the user-input boundary, and frame serialized remote events as untrusted before model invocation. Strict thread existence/ownership admission dead-letters events whose task outlives its deleted chat instead of recreating the thread. CORS is same-origin by default when requests enter through nginx on port 2026. Split-origin or port-forwarded browser clients must opt in with `GATEWAY_CORS_ORIGINS` (exact origins); Gateway `CORSMiddleware` and `CSRFMiddleware` both read that variable so browser CORS and auth-origin checks stay aligned. Those clients also need `CORS_EXPOSED_HEADERS` (`csrf_middleware.py`): run-creating routes return the run's id in `Content-Location`, which is not CORS-safelisted, so JS cannot read it unless it is exposed — and the LangGraph SDK resolves run metadata from that header alone, so withholding it breaks `useStream`'s `onCreated` and thread-gated actions. diff --git a/backend/app/gateway/services.py b/backend/app/gateway/services.py index ef23df0c1..110ba9184 100644 --- a/backend/app/gateway/services.py +++ b/backend/app/gateway/services.py @@ -661,12 +661,36 @@ def resolve_agent_factory(assistant_id: str | None): # client-supplied ``recursion_limit`` verbatim: an arbitrarily large value lets # a single run execute unbounded LangGraph super-steps (each at least one LLM # call), enabling runaway API cost / DoS. ``_DEFAULT_RECURSION_LIMIT`` is the -# server default when the client sends nothing; the hard ceiling any client -# value is clamped to is configurable via ``AppConfig.max_recursion_limit``. +# fallback when app config cannot be loaded; the normal server default and hard +# ceiling are configurable via ``AppConfig.recursion_limit`` and +# ``AppConfig.max_recursion_limit``. _DEFAULT_RECURSION_LIMIT = 100 _DEFAULT_MAX_RECURSION_LIMIT = 1000 +def _resolve_gateway_recursion_limits() -> tuple[int, int]: + """Resolve the run default and ceiling from one hot-reloaded snapshot.""" + try: + app_config = get_app_config() + raw = app_config.recursion_limit + max_limit = app_config.max_recursion_limit + if raw > max_limit: + logger.warning( + "recursion_limit %d exceeds max_recursion_limit %d; clamped to %d for Gateway runs", + raw, + max_limit, + max_limit, + ) + return min(raw, max_limit), max_limit + except Exception: + logger.warning( + "failed to load app config; falling back to recursion_limit=%d and max_recursion_limit=%d for Gateway runs", + _DEFAULT_RECURSION_LIMIT, + _DEFAULT_MAX_RECURSION_LIMIT, + ) + return _DEFAULT_RECURSION_LIMIT, _DEFAULT_MAX_RECURSION_LIMIT + + def _resolve_max_recursion_limit() -> int: """Resolve the clamp ceiling from ``AppConfig.max_recursion_limit``. @@ -717,15 +741,15 @@ def _resolve_scheduler_recursion_limit() -> int: return _DEFAULT_RECURSION_LIMIT -def _clamp_recursion_limit(value: Any, max_limit: int) -> int: +def _clamp_recursion_limit(value: Any, max_limit: int, default_limit: int) -> int: """Clamp a client-supplied ``recursion_limit`` into a safe server range. Non-integer values (including ``bool``, an ``int`` subclass) and non-positive - values fall back to ``_DEFAULT_RECURSION_LIMIT``; valid positive integers are + values fall back to the configured default; valid positive integers are capped at ``max_limit`` (from ``AppConfig.max_recursion_limit``). """ if isinstance(value, bool) or not isinstance(value, int) or value <= 0: - return _DEFAULT_RECURSION_LIMIT + return default_limit return min(value, max_limit) @@ -749,16 +773,17 @@ def build_run_config( load the matching ``agents//SOUL.md`` and per-agent config — without it the agent silently runs as the default lead agent. - This mirrors the channel manager's ``_resolve_run_params`` logic so that - the LangGraph Platform-compatible HTTP API and the IM channel path behave - identically. + This mirrors the channel manager's ``_resolve_run_params`` logic except for + the recursion default: Gateway API runs use the configured top-level + ``recursion_limit``, while IM channel runs retain their own default. """ # Lead-agent recursion budget (LangGraph super-steps for the lead graph # only). Independent of subagent depth: a `task()` dispatch runs the whole # subagent inside ONE lead tools-node step, and subagents enforce their own - # limit via `subagents.max_turns`. Do not conflate this 100 with the + # limit via `subagents.max_turns`. Do not conflate this budget with the # general-purpose subagent's max_turns. - config: dict[str, Any] = {"recursion_limit": _DEFAULT_RECURSION_LIMIT} + default_recursion_limit, max_recursion_limit = _resolve_gateway_recursion_limits() + config: dict[str, Any] = {"recursion_limit": default_recursion_limit} if request_config: # LangGraph >= 0.6.0 introduced ``context`` as the preferred way to # pass thread-level data and rejects requests that include both @@ -807,14 +832,13 @@ def build_run_config( # super-steps (runaway LLM cost / DoS). Applied after the passthrough so # it overrides whatever the client sent. if "recursion_limit" in request_config: - max_limit = _resolve_max_recursion_limit() - clamped = _clamp_recursion_limit(request_config["recursion_limit"], max_limit) + clamped = _clamp_recursion_limit(request_config["recursion_limit"], max_recursion_limit, default_recursion_limit) if clamped != request_config["recursion_limit"]: logger.warning( "build_run_config: clamped client recursion_limit %r -> %d (max %d). thread_id=%s", request_config["recursion_limit"], clamped, - max_limit, + max_recursion_limit, thread_id, ) config["recursion_limit"] = clamped diff --git a/backend/docs/API.md b/backend/docs/API.md index 9f1737879..e8ad7e461 100644 --- a/backend/docs/API.md +++ b/backend/docs/API.md @@ -265,16 +265,19 @@ for runs without changed outputs keep their existing shape. **Recursion Limit:** `config.recursion_limit` caps the number of graph steps LangGraph will execute -in a single run. The unified Gateway path defaults to `100` in -`build_run_config` (see `backend/app/gateway/services.py`), which is a safer -starting point for plan-mode or subagent-heavy runs. Clients can still set -`recursion_limit` explicitly in the request body; increase it if you run deeply -nested subagent graphs. Scheduled-task launches do not take a client body: they +in a single run. The unified Gateway path uses the top-level `recursion_limit` +from `config.yaml` (default `100`) when a request does not provide one. Clients +can still set `recursion_limit` explicitly in the request body, and a valid +request value takes precedence. Scheduled-task launches do not take a client body: they use `scheduler.recursion_limit` from `config.yaml` (default `1000`, matching the web UI). For safety, the Gateway clamps any supplied -value to a configurable server ceiling (`max_recursion_limit` in `config.yaml`, +or configured value to a server ceiling (`max_recursion_limit` in `config.yaml`, default `1000`) so a single run cannot execute unbounded graph steps (runaway -LLM cost / DoS); invalid or non-positive values fall back to the `100` default. +LLM cost / DoS); invalid or non-positive request values fall back to the +configured default. Both top-level fields are read per run, so edits apply to +the next request without restarting the Gateway. This top-level setting applies +to Gateway API runs only; IM channel and embedded `DeerFlowClient` runs retain +their own defaults and override paths. **Configurable Options:** - `model_name` (string): Override the default model diff --git a/backend/docs/CONFIGURATION.md b/backend/docs/CONFIGURATION.md index 895f8389f..b14c32c2f 100644 --- a/backend/docs/CONFIGURATION.md +++ b/backend/docs/CONFIGURATION.md @@ -25,6 +25,20 @@ preference hints for requests that should prefer a specific MCP server or tool. See [MCP Server Configuration](MCP_SERVER.md#routing-hints) for the schema, example, and soft-vs-hard routing boundary. +### Recursion Limits + +Gateway runs use the top-level `recursion_limit` as their LangGraph super-step +budget when the request does not include an explicit value. It defaults to +`100`; raise it for deployments whose normal tasks need longer agent loops. +Valid request values take precedence, while invalid values fall back to the +configured default. `max_recursion_limit` (default `1000`) caps both sources to +limit runaway LLM cost. Both settings are read per run, so changes apply to the +next request without a Gateway restart. + +These settings apply to Gateway API runs. IM channel runs and embedded +`DeerFlowClient` runs retain their own defaults and can be overridden through +their channel/client-specific configuration or per-call options. + ### Models Configure the LLM models available to the agent: diff --git a/backend/packages/harness/deerflow/config/AGENTS.md b/backend/packages/harness/deerflow/config/AGENTS.md index ddb225d8b..a72bc07fe 100644 --- a/backend/packages/harness/deerflow/config/AGENTS.md +++ b/backend/packages/harness/deerflow/config/AGENTS.md @@ -24,6 +24,8 @@ Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** direc **Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`. +Top-level `recursion_limit` and `max_recursion_limit` are hot-reloaded per Gateway run. The former supplies the default when a request omits or provides an invalid value; the latter caps both configured and client-provided budgets. + **Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path or file content signature changes. The signature includes file metadata and a content digest, so Gateway and LangGraph reads stay aligned with `config.yaml` edits even on object-store or network mounts where mtime can remain stale. **Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `verification.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`. diff --git a/backend/packages/harness/deerflow/config/app_config.py b/backend/packages/harness/deerflow/config/app_config.py index 10bab8c09..93534e1a9 100644 --- a/backend/packages/harness/deerflow/config/app_config.py +++ b/backend/packages/harness/deerflow/config/app_config.py @@ -215,10 +215,15 @@ class AppConfig(BaseModel): ), ), ) + recursion_limit: int = Field( + default=100, + ge=1, + description="Default LangGraph recursion_limit for Gateway runs when the client does not provide one. Applied per run and capped by max_recursion_limit.", + ) max_recursion_limit: int = Field( default=1000, ge=1, - description="Hard server-side ceiling for a client-supplied run recursion_limit. Client values above this are clamped; prevents runaway LangGraph super-steps (LLM cost / DoS).", + description="Hard server-side ceiling for configured defaults and client-supplied run recursion_limit values. Values above this are clamped; prevents runaway LangGraph super-steps (LLM cost / DoS).", ) models: list[ModelConfig] = Field(default_factory=list, description="Available models") sandbox: SandboxConfig = Field( diff --git a/backend/tests/test_gateway_services.py b/backend/tests/test_gateway_services.py index fe988a5d8..e587c4d51 100644 --- a/backend/tests/test_gateway_services.py +++ b/backend/tests/test_gateway_services.py @@ -623,6 +623,26 @@ def test_build_run_config_basic(): assert config["recursion_limit"] == 100 +def test_build_run_config_uses_configured_default_recursion_limit(_stub_app_config): + """Runs without a request override use the operator-configured default.""" + from app.gateway.services import build_run_config + from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config + + set_app_config( + AppConfig.model_validate( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "recursion_limit": 700, + } + ) + ) + try: + config = build_run_config("thread-1", None, None) + assert config["recursion_limit"] == 700 + finally: + reset_app_config() + + def test_build_run_config_with_overrides(): from app.gateway.services import build_run_config @@ -721,13 +741,102 @@ def test_build_run_config_preserves_reasonable_recursion_limit(_stub_app_config) assert config["recursion_limit"] == 250 -def test_build_run_config_rejects_invalid_recursion_limit(_stub_app_config): - """Non-positive / non-int / bool values fall back to the server default.""" - from app.gateway.services import _DEFAULT_RECURSION_LIMIT, build_run_config +def test_build_run_config_client_recursion_limit_overrides_configured_default(_stub_app_config): + """An explicit valid client value takes precedence over the server default.""" + from app.gateway.services import build_run_config + from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config - for bad in (0, -5, "1000", 3.5, True, None): - config = build_run_config("thread-1", {"recursion_limit": bad}, None) - assert config["recursion_limit"] == _DEFAULT_RECURSION_LIMIT, bad + set_app_config( + AppConfig.model_validate( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "recursion_limit": 700, + } + ) + ) + try: + config = build_run_config("thread-1", {"recursion_limit": 250}, None) + assert config["recursion_limit"] == 250 + finally: + reset_app_config() + + +def test_build_run_config_rejects_invalid_recursion_limit(_stub_app_config): + """Non-positive / non-int / bool values fall back to the configured default.""" + from app.gateway.services import build_run_config + from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config + + set_app_config( + AppConfig.model_validate( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "recursion_limit": 700, + } + ) + ) + + try: + for bad in (0, -5, "1000", 3.5, True, None): + config = build_run_config("thread-1", {"recursion_limit": bad}, None) + assert config["recursion_limit"] == 700, bad + finally: + reset_app_config() + + +def test_build_run_config_logs_and_uses_fallback_when_app_config_unavailable(monkeypatch, caplog): + """A config-load failure falls back visibly instead of silently.""" + from app.gateway import services + + monkeypatch.setattr(services, "get_app_config", lambda: (_ for _ in ()).throw(RuntimeError("broken config"))) + caplog.set_level(logging.WARNING, logger="app.gateway.services") + + config = services.build_run_config("thread-1", {"recursion_limit": 0}, None) + + assert config["recursion_limit"] == services._DEFAULT_RECURSION_LIMIT + assert any("failed to load app config; falling back to recursion_limit=100" in record.message for record in caplog.records) + + +def test_build_run_config_invalid_client_recursion_limit_uses_configured_default(_stub_app_config): + """An invalid client value cannot erase the operator-configured default.""" + from app.gateway.services import build_run_config + from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config + + set_app_config( + AppConfig.model_validate( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "recursion_limit": 700, + } + ) + ) + try: + config = build_run_config("thread-1", {"recursion_limit": 0}, None) + assert config["recursion_limit"] == 700 + finally: + reset_app_config() + + +def test_build_run_config_clamps_configured_default_to_ceiling(_stub_app_config, caplog): + """The operator default remains bounded by max_recursion_limit.""" + from app.gateway.services import build_run_config + from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config + + set_app_config( + AppConfig.model_validate( + { + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "recursion_limit": 700, + "max_recursion_limit": 500, + } + ) + ) + try: + caplog.set_level(logging.WARNING, logger="app.gateway.services") + config = build_run_config("thread-1", None, None) + assert config["recursion_limit"] == 500 + assert any("recursion_limit 700 exceeds max_recursion_limit 500" in record.message for record in caplog.records) + finally: + reset_app_config() def test_build_run_config_clamps_recursion_limit_with_context(_stub_app_config): diff --git a/config.example.yaml b/config.example.yaml index 49167c05d..043eb349b 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -20,7 +20,7 @@ # ============================================================================ # Bump this number when the config schema changes. # Run `make config-upgrade` to merge new fields into your local config.yaml. -config_version: 42 +config_version: 43 # ============================================================================ # Logging @@ -94,14 +94,16 @@ token_budget: hard_stop_threshold: 1.0 # Force stop at 100% of the budget # ============================================================================ -# Recursion Limit — Hard ceiling for a client-supplied run recursion_limit +# Recursion Limits # ============================================================================ # A run's recursion_limit caps the number of LangGraph super-steps (each is at -# least one LLM call). The Gateway never trusts a client-supplied value -# verbatim: any value above this ceiling is clamped down to it, preventing -# runaway API cost / DoS. Invalid or non-positive client values fall back to -# the server default of 100. Raise this only if you legitimately run very -# deeply nested subagent graphs. +# least one LLM call). Clients may override the default per request; invalid or +# non-positive client values fall back to this configured default. +recursion_limit: 100 + +# The Gateway never trusts a configured or client-supplied value above this +# ceiling, preventing runaway API cost / DoS. Raise it only if you legitimately +# run very deeply nested subagent graphs. max_recursion_limit: 1000 diff --git a/deploy/helm/deer-flow/README.md b/deploy/helm/deer-flow/README.md index 867ea016e..a63ecf608 100644 --- a/deploy/helm/deer-flow/README.md +++ b/deploy/helm/deer-flow/README.md @@ -131,7 +131,7 @@ they resolve from the `secrets` map): ```yaml config: | - config_version: 42 + config_version: 43 models: - name: gpt-4 use: langchain_openai:ChatOpenAI diff --git a/deploy/helm/deer-flow/values.yaml b/deploy/helm/deer-flow/values.yaml index bd580ce47..044f9206b 100644 --- a/deploy/helm/deer-flow/values.yaml +++ b/deploy/helm/deer-flow/values.yaml @@ -249,8 +249,10 @@ ingress: # -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never # inline literal secret values here. The default enables provisioner sandbox. config: | - config_version: 42 + config_version: 43 log_level: info + recursion_limit: 100 + max_recursion_limit: 1000 models: [] # Example (uncomment & set the matching secret in `secrets`):