青榆牧 a94b2d8897
feat(mcp): map request-scoped secrets to MCP HTTP/SSE headers (#5010)
* feat(mcp): map request-scoped secrets to HTTP/SSE headers

`user_auth` binds a credential to a configured DeerFlow user, so a caller
that picks the credential per request — a multi-tenant gateway, a per-run
API key, one shared MCP server fronting several environments — had to
register one MCP server entry per credential.

Add a declarative `mcpServers.<server>.headers_from_context` block mapping
HTTP header names to keys of the run request's `config.context.secrets`
carrier. A new built-in interceptor resolves the mapping on every tool call
and rewrites those headers, mirroring `user_scoped_auth`. The config file
stores names only, never a credential, so the Gateway returns the block
unmasked.

Registered after OAuth and `user_auth` in the interceptor chain: the later
interceptor runs closer to the transport, and the value chosen for this one
request is the most specific, so it wins. Fail-closed by default — a mapped
key missing from the request raises a `ToolException` naming only that key,
because falling back to the server's discovery credential would send one
tenant's call under another tenant's authority. `on_missing: "passthrough"`
opts out.

Durable background tasks are excluded: `McpTaskToolCaller` drives status and
cancel polls after the Agent run ends, where no run context exists, so the
fail-closed interceptor would deny every poll. Those calls keep using
server-level credentials, and a server declaring both `headers_from_context`
and `task_toolsets` now logs a warning.

Also corrects the custom-interceptor example in docs/MCP_SERVER.md (and the
matching claim in skills/AGENTS.md), which read request secrets from
`langgraph.config.get_config()["context"]`. That key is `None` inside a tool
call — the run context rides the LangGraph runtime, not the RunnableConfig
propagated to child runnables — so interceptors written from that example
never saw a value. The example now reads `request.runtime`, and
tests/test_mcp_context_headers.py pins LangGraph's runtime-injection rule by
driving a real langchain-mcp-adapters tool through a real graph with the
ambient-runtime fallback disabled.

Closes #5005

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(mcp): resolve credential headers case-insensitively, carry them on durable submit

Review follow-ups on `headers_from_context`.

HTTP field names are case-insensitive, but every dict on the path to the wire
is not: `build_server_params` copies the operator's static `headers` spelling
verbatim, and langchain-mcp-adapters merges interceptor overrides into the
connection with a plain `{**connection_headers, **override_headers}` splat. A
static `authorization` and an injected `Authorization` therefore both reached
httpx as separate field lines, and a server reading the field with a
single-value accessor got the static discovery credential — inverting the
documented `headers` < `oauth` < `user_auth` < `headers_from_context`
precedence and running a per-request call under the shared credential.

Normalizing inside the interceptor cannot fix that on its own: the adapter
builds the request with `headers=None`, so an interceptor never sees the
connection's static headers and cannot displace them however it spells its own
key. A new `mcp/headers.py::apply_header_overrides` therefore drops any key
differing only in case and emits the spelling the connection already uses.
Applied to `headers_from_context`, `user_auth`, the OAuth interceptor, the
OAuth discovery-header write, and the durable-task connection merge, which all
carried the same collision. `headers_from_context.headers` now also rejects one
header mapped under two spellings at config load, in both the harness model and
the Gateway mirror.

Durable submit now carries the mapped headers, as docs/MCP_SERVER.md already
promised. `McpTaskToolCaller` disabled the interceptor for the whole caller, but
that caller serves submit as well as the polls, and submit is awaited inline
inside the Agent's tool call — where the run's LangGraph runtime is still the
ambient contextvar, so no secret has to be threaded through `TaskSubmitRequest`
or reach durable storage. The caller builds one chain and keeps a second view of
it without the context-headers interceptor; `call_tool` takes
`request_scoped_headers`, set only by `OrdinaryMcpTaskDriver.submit`. Status and
cancel keep server-level credentials, so background polls still cannot fail
closed, and the startup warning now describes the half it actually covers.

`_merge_preserving_secrets` restores masked extras inside `headers_from_context`
instead of writing the `***` sentinel back over the stored value, matching the
treatment `user_auth` extras and server-level extras already get; extras a PUT
omits carry over as well, while the declared mapping still replaces verbatim so
a round trip can remove an entry. `extra="allow"` plus name-based sensitivity
detection means the usual casualty is a name-valued key such as `tokenHeader`,
not only a credential.

The existing override test seeded the static header onto `request.headers`,
which production never does, so it modelled a merge that really happens one
layer down; the new tests drive a real adapter tool through a real connection
and assert on the headers the session is opened with, and the durable-submit
test runs through a real tool node with no runtime patching.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(mcp): reject case-insensitive duplicate static header names

* fix(mcp): preserve omitted headers_from_context fields on partial updates

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 10:42:42 +08:00

735 lines
32 KiB
Python

"""Unified extensions configuration for MCP servers and skills."""
import errno
import json
import logging
import os
import stat
import tempfile
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from deerflow.config.runtime_paths import existing_project_file
from deerflow.constants import (
DEFAULT_MCP_SESSION_INIT_TIMEOUT,
MCP_TASK_NAME_MAX_LENGTH,
MCP_TASK_SERVER_NAME_MAX_LENGTH,
)
logger = logging.getLogger(__name__)
_non_atomic_fallback_targets: set[Path] = set()
_non_atomic_fallback_targets_lock = threading.Lock()
def normalize_mcp_transport_alias(data: Any) -> Any:
"""Promote MCP-spec ``transport`` to ``type`` when ``type`` is absent."""
if isinstance(data, dict):
transport = data.get("transport")
if transport and not data.get("type"):
return {**data, "type": transport}
return data
class McpRoutingConfig(BaseModel):
"""Soft routing hints for MCP tool preference."""
mode: Literal["off", "prefer"] = Field(
default="off",
description="Whether to emit prompt hints preferring this MCP tool for matching requests.",
)
priority: int = Field(
default=0,
description="Ordering key for routing hints. Higher values are rendered first.",
)
keywords: list[str] = Field(
default_factory=list,
description="Operator-authored keywords that describe when this MCP tool should be preferred.",
)
model_config = ConfigDict(extra="forbid")
@field_validator("priority")
@classmethod
def _clamp_priority(cls, value: int) -> int:
if value < 0:
logger.warning("MCP routing priority %s is below 0; clamping to 0.", value)
return 0
if value > 100:
logger.warning("MCP routing priority %s is above 100; clamping to 100.", value)
return 100
return value
class McpToolOverride(BaseModel):
"""Per-tool MCP configuration overrides."""
routing: McpRoutingConfig = Field(default_factory=McpRoutingConfig)
model_config = ConfigDict(extra="allow")
class McpTaskToolsetConfig(BaseModel):
"""One ordinary submit/status/cancel contract exposed by an MCP server.
Tool names are the exact raw names advertised by that server. The
presentation prefix added by ``langchain-mcp-adapters`` is deliberately not
part of this durable binding.
"""
name: str = Field(
min_length=1,
max_length=MCP_TASK_NAME_MAX_LENGTH,
description="Stable local name shown for tasks from this toolset",
)
submit_tool: str = Field(min_length=1, description="Raw MCP tool name used to submit work")
status_tool: str = Field(min_length=1, description="Raw MCP tool name used to poll work")
cancel_tool: str = Field(min_length=1, description="Raw MCP tool name used to cancel work")
model_config = ConfigDict(extra="forbid")
@field_validator("name")
@classmethod
def _validate_name_is_not_blank(cls, value: str) -> str:
if not value.strip():
raise ValueError("MCP task toolset name must not be empty")
return value
class McpUserScopedAuthConfig(BaseModel):
"""Per-user credential injection for a shared MCP server (HTTP/SSE transports).
Maps DeerFlow user ids to credential header values so that one configured
MCP server can serve several users, each authenticated to the remote
service with their own credential. The credential for the authenticated
user is injected into every tool call by the built-in user-scoped auth
interceptor; the server entry's static ``headers`` are only used for
startup tool discovery.
Values support the same ``$ENV_VAR`` resolution as the rest of this file,
so raw secrets can stay in the process environment.
"""
enabled: bool = Field(default=True, description="Whether user-scoped credential injection is enabled")
header: str = Field(default="Authorization", description="HTTP header to set with the resolved user credential")
users: dict[str, str] = Field(
default_factory=dict,
description="Map of DeerFlow user id to full credential header value (e.g. 'Bearer <token>'); values support $ENV_VAR references",
)
on_missing: Literal["deny", "passthrough"] = Field(
default="deny",
description=("Behavior when the calling user has no mapped credential (or the mapped value resolved empty): 'deny' fails the tool call with an actionable error; 'passthrough' forwards the request with the server's static headers"),
)
model_config = ConfigDict(extra="allow")
@field_validator("header")
@classmethod
def _validate_header_not_blank(cls, value: str) -> str:
if not value.strip():
raise ValueError("user_auth.header must not be empty")
return value
class McpContextHeadersConfig(BaseModel):
"""Per-request credential injection for an MCP server (HTTP/SSE transports).
Maps HTTP header names to keys of the run request's ``config.context.secrets``
carrier, so one configured MCP server can serve callers that each supply their
own credential *per request* rather than per configured user. The built-in
context-headers interceptor resolves the mapping on every tool call; the
server entry's static ``headers`` are only used for startup tool discovery.
Unlike ``user_auth``, this block stores **no credential** — only header names
and run-context key names — so it is safe to return unmasked from the config
API. The values arrive out-of-band with each run and never enter the prompt,
tool arguments, or trace payloads (see ``runtime/secret_context.py``).
"""
enabled: bool = Field(default=True, description="Whether request-scoped header injection is enabled")
headers: dict[str, str] = Field(
default_factory=dict,
description="Map of HTTP header name to the key to read from the run request's config.context.secrets (e.g. {'X-Tenant-Id': 'tenant_id'})",
)
on_missing: Literal["deny", "passthrough"] = Field(
default="deny",
description=("Behavior when a mapped key is absent from the request secrets (or resolved empty): 'deny' fails the tool call with an actionable error; 'passthrough' forwards the request with the server's static headers"),
)
model_config = ConfigDict(extra="allow")
@field_validator("headers")
@classmethod
def _validate_mapping_entries(cls, value: dict[str, str]) -> dict[str, str]:
seen: dict[str, str] = {}
for header_name, secret_key in value.items():
if not header_name.strip():
raise ValueError("headers_from_context.headers must not contain a blank header name")
if not isinstance(secret_key, str) or not secret_key.strip():
raise ValueError(f"headers_from_context.headers[{header_name!r}] must name a non-blank secret key from config.context.secrets")
# HTTP field names are case-insensitive, so two spellings of one
# header are one header with two candidate values, and which one
# reaches the remote would depend on dict ordering.
lowered = header_name.lower()
if lowered in seen:
raise ValueError(f"headers_from_context.headers maps the same HTTP header under two spellings ({seen[lowered]!r} and {header_name!r}); header names are case-insensitive, so keep only one")
seen[lowered] = header_name
return value
class McpOAuthConfig(BaseModel):
"""OAuth configuration for an MCP server (HTTP/SSE transports)."""
enabled: bool = Field(default=True, description="Whether OAuth token injection is enabled")
token_url: str = Field(description="OAuth token endpoint URL")
grant_type: Literal["client_credentials", "refresh_token"] = Field(
default="client_credentials",
description="OAuth grant type",
)
client_id: str | None = Field(default=None, description="OAuth client ID")
client_secret: str | None = Field(default=None, description="OAuth client secret")
refresh_token: str | None = Field(default=None, description="OAuth refresh token (for refresh_token grant)")
scope: str | None = Field(default=None, description="OAuth scope")
audience: str | None = Field(default=None, description="OAuth audience (provider-specific)")
token_field: str = Field(default="access_token", description="Field name containing access token in token response")
token_type_field: str = Field(default="token_type", description="Field name containing token type in token response")
expires_in_field: str = Field(default="expires_in", description="Field name containing expiry (seconds) in token response")
default_token_type: str = Field(default="Bearer", description="Default token type when missing in token response")
refresh_skew_seconds: int = Field(default=60, description="Refresh token this many seconds before expiry")
extra_token_params: dict[str, str] = Field(default_factory=dict, description="Additional form params sent to token endpoint")
model_config = ConfigDict(extra="allow")
class McpServerConfig(BaseModel):
"""Configuration for a single MCP server."""
enabled: bool = Field(default=True, description="Whether this MCP server is enabled")
type: str = Field(default="stdio", description="Transport type: 'stdio', 'sse', or 'http'")
command: str | None = Field(default=None, description="Command to execute to start the MCP server (for stdio type)")
args: list[str] = Field(default_factory=list, description="Arguments to pass to the command (for stdio type)")
env: dict[str, str] = Field(default_factory=dict, description="Environment variables for the MCP server")
url: str | None = Field(default=None, description="URL of the MCP server (for sse or http type)")
headers: dict[str, str] = Field(default_factory=dict, description="HTTP headers to send (for sse or http type)")
oauth: McpOAuthConfig | None = Field(default=None, description="OAuth configuration (for sse or http type)")
user_auth: McpUserScopedAuthConfig | None = Field(
default=None,
description="Per-user credential injection (for sse or http type): map DeerFlow user ids to per-user credential header values",
)
headers_from_context: McpContextHeadersConfig | None = Field(
default=None,
description="Per-request credential injection (for sse or http type): map HTTP header names to keys of the run request's config.context.secrets",
)
description: str = Field(default="", description="Human-readable description of what this MCP server provides")
routing: McpRoutingConfig = Field(default_factory=McpRoutingConfig, description="Soft routing hints for tools from this MCP server")
tools: dict[str, McpToolOverride] = Field(default_factory=dict, description="Per-original-tool MCP configuration overrides")
tool_name_prefix: bool = Field(
default=True,
description="Whether to prefix discovered tool names with the MCP server name to avoid cross-server collisions",
)
tool_call_timeout: float | None = Field(
default=None,
description=("Timeout in seconds for individual stdio MCP tool calls and durable-task calls on every transport. Other HTTP/SSE tools use transport-level timeouts. None means no call-level timeout."),
)
session_init_timeout: float | None = Field(
default=DEFAULT_MCP_SESSION_INIT_TIMEOUT,
description=(
"Timeout in seconds for MCP server bring-up: tool discovery (subprocess spawn + initialize + tools/list) "
"and persistent stdio session initialization, plus ephemeral HTTP/SSE durable-task session "
"initialization. Defaults to DEFAULT_MCP_SESSION_INIT_TIMEOUT so a hung server cannot block agent "
"construction or the task poller indefinitely. None means no timeout."
),
)
task_toolsets: list[McpTaskToolsetConfig] = Field(
default_factory=list,
description="Ordinary submit/status/cancel tool groups managed by the durable MCP task runtime",
)
model_config = ConfigDict(extra="allow")
@field_validator("headers")
@classmethod
def _validate_header_names(cls, value: dict[str, str]) -> dict[str, str]:
# HTTP field names are case-insensitive, so two spellings of one header
# are one field with two candidate values. The adapter copies the static
# mapping verbatim, so both would reach the wire; a later per-request or
# OAuth override only replaces one spelling, leaving the other to leak a
# shared credential across tenant authority. Reject at config time so a
# bad mapping cannot reach the connection.
seen: dict[str, str] = {}
for header_name in value:
lowered = header_name.lower()
if lowered in seen:
raise ValueError(f"headers maps the same HTTP header under two spellings ({seen[lowered]!r} and {header_name!r}); header names are case-insensitive, so keep only one")
seen[lowered] = header_name
return value
@model_validator(mode="before")
@classmethod
def _accept_transport_alias(cls, data: Any) -> Any:
"""Accept the MCP-spec ``transport`` field as an alias for ``type``.
The official MCP configuration schema uses ``transport`` to indicate
the transport mechanism (``stdio``/``sse``/``http``). Earlier versions
of this project only honored ``type``, which caused remote SSE/HTTP
servers configured with just ``transport`` to be incorrectly treated as
``stdio`` (the default). This validator normalizes the two so either
spelling works, with ``type`` taking precedence when both are provided.
"""
return normalize_mcp_transport_alias(data)
@model_validator(mode="after")
def _validate_task_tool_bindings(self) -> "McpServerConfig":
claimed: dict[str, str] = {}
for toolset in self.task_toolsets:
for role in ("submit_tool", "status_tool", "cancel_tool"):
raw_name = getattr(toolset, role)
previous = claimed.get(raw_name)
if previous is not None:
raise ValueError(f"MCP task tool {raw_name!r} must be unique across task_toolsets and roles; it is configured as both {previous} and {toolset.name}.{role}")
claimed[raw_name] = f"{toolset.name}.{role}"
return self
def resolve_effective_mcp_routing(server_config: McpServerConfig | None, original_tool_name: str) -> dict[str, Any]:
"""Merge server-level routing with per-tool overrides for one MCP tool."""
if server_config is None:
return McpRoutingConfig().model_dump(mode="json")
effective = server_config.routing.model_dump(mode="json")
override = server_config.tools.get(original_tool_name)
if override is not None and "routing" in override.model_fields_set:
effective.update(override.routing.model_dump(mode="json", exclude_unset=True))
return effective
class SkillStateConfig(BaseModel):
"""Configuration for a single skill's state."""
enabled: bool = Field(default=True, description="Whether this skill is enabled")
class ExtensionsConfig(BaseModel):
"""Unified configuration for MCP servers and skills."""
middlewares: list[str] = Field(
default_factory=list,
description="AgentMiddleware class paths loaded into the lead-agent middleware chain. Each entry uses 'module.path:ClassName'.",
)
mcp_servers: dict[str, McpServerConfig] = Field(
default_factory=dict,
description="Map of MCP server name to configuration",
alias="mcpServers",
)
skills: dict[str, SkillStateConfig] = Field(
default_factory=dict,
description="Map of skill name to state configuration",
)
model_config = ConfigDict(extra="allow", populate_by_name=True)
@model_validator(mode="after")
def _validate_task_server_names_fit_storage(self) -> "ExtensionsConfig":
for server_name, server in self.mcp_servers.items():
if not server.task_toolsets:
continue
if not server_name.strip() or len(server_name) > MCP_TASK_SERVER_NAME_MAX_LENGTH:
raise ValueError(f"MCP task server name must contain 1 to {MCP_TASK_SERVER_NAME_MAX_LENGTH} characters")
return self
def to_file_dict(self) -> dict[str, Any]:
"""Serialize in the public extensions_config.json shape."""
return self.model_dump(by_alias=True)
@classmethod
def resolve_config_path(cls, config_path: str | None = None) -> Path | None:
"""Resolve the extensions config file path.
Priority:
1. If provided `config_path` argument, use it.
2. If provided `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable, use it.
3. Otherwise, search the caller project root for `extensions_config.json`, then `mcp_config.json`.
4. For backward compatibility, also search legacy backend/repository-root defaults.
5. If not found via search, return None (extensions are optional).
Args:
config_path: Optional path to extensions config file.
Resolution order:
1. If provided `config_path` argument, use it.
2. If provided `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable, use it.
3. Otherwise, search the caller project root for
`extensions_config.json`, then legacy `mcp_config.json`.
4. Finally, search backend/repository-root defaults for monorepo compatibility.
Returns:
Path to the extensions config file if found via the resolution
order above.
An explicit `config_path` argument or a set
`DEER_FLOW_EXTENSIONS_CONFIG_PATH` is an operator assertion that
one particular file must be used, so a missing file in either of
those two modes raises ``FileNotFoundError`` (see Raises below)
instead of degrading to "no config" — a bad Docker mount, typo,
or deleted production config should surface as a loud, actionable
error rather than silently starting with every MCP server and
skill absent.
Only the fallback *search* mode (no explicit argument and no env
var set) returns ``None`` when nothing is found: that case means
extensions were never configured in the first place, which is the
legitimate "extensions are optional" case some callers (e.g. the
MCP tools-cache staleness check in `deerflow.mcp.cache`) rely on
as a clean, expected signal.
Raises:
FileNotFoundError: If `config_path` is given, or
`DEER_FLOW_EXTENSIONS_CONFIG_PATH` is set, and the resolved
path does not exist.
"""
if config_path:
path = Path(config_path)
if not path.exists():
raise FileNotFoundError(f"Extensions config file specified by param `config_path` not found at {path}")
return path
elif env_path := os.getenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH"):
path = Path(env_path)
if not path.exists():
raise FileNotFoundError(f"Extensions config file specified by environment variable `DEER_FLOW_EXTENSIONS_CONFIG_PATH` not found at {path}")
return path
else:
project_config = existing_project_file(("extensions_config.json", "mcp_config.json"))
if project_config is not None:
return project_config
backend_dir = Path(__file__).resolve().parents[4]
repo_root = backend_dir.parent
for path in (
backend_dir / "extensions_config.json",
repo_root / "extensions_config.json",
backend_dir / "mcp_config.json",
repo_root / "mcp_config.json",
):
if path.exists():
return path
# Extensions are optional: unlike the explicit config_path/env-var
# branches above, finding nothing here is the expected case, so
# return None rather than raising.
return None
@classmethod
def from_file(cls, config_path: str | None = None) -> "ExtensionsConfig":
"""Load extensions config from JSON file.
See `resolve_config_path` for more details.
Args:
config_path: Path to the extensions config file.
Returns:
ExtensionsConfig: The loaded config, or empty config if file not found.
"""
resolved_path = cls.resolve_config_path(config_path)
if resolved_path is None:
# Return empty config if extensions config file is not found
return cls(mcp_servers={}, skills={})
try:
with open(resolved_path, encoding="utf-8") as f:
config_data = json.load(f)
config_data = cls.resolve_env_variables(config_data)
return cls.model_validate(config_data)
except json.JSONDecodeError as e:
raise ValueError(f"Extensions config file at {resolved_path} is not valid JSON: {e}") from e
except Exception as e:
raise RuntimeError(f"Failed to load extensions config from {resolved_path}: {e}") from e
@classmethod
def resolve_env_variables(cls, config: Any) -> Any:
"""Recursively resolve environment variables in the config.
Environment variables are resolved using the `os.getenv` function. Example: $OPENAI_API_KEY
Args:
config: The config to resolve environment variables in.
Returns:
The config with environment variables resolved.
"""
if isinstance(config, str):
if not config.startswith("$"):
return config
env_value = os.getenv(config[1:])
if env_value is None:
# Unresolved placeholder — store empty string so downstream
# consumers (e.g. MCP servers) don't receive the literal "$VAR"
# token as an actual environment value.
return ""
return env_value
if isinstance(config, dict):
return {key: cls.resolve_env_variables(value) for key, value in config.items()}
if isinstance(config, list):
return [cls.resolve_env_variables(item) for item in config]
if isinstance(config, tuple):
return tuple(cls.resolve_env_variables(item) for item in config)
return config
def get_enabled_mcp_servers(self) -> dict[str, McpServerConfig]:
"""Get only the enabled MCP servers.
Returns:
Dictionary of enabled MCP servers.
"""
return {name: config for name, config in self.mcp_servers.items() if config.enabled}
def is_skill_enabled(self, skill_name: str, skill_category: str) -> bool:
"""Check if a skill is enabled.
Args:
skill_name: Name of the skill
skill_category: Category of the skill (public, custom, or legacy)
Returns:
True if enabled, False otherwise.
Note:
All skill categories (public, custom, legacy) respect the
extensions_config enabled/disabled state. When no explicit
entry exists, skills default to enabled.
"""
skill_config = self.skills.get(skill_name)
if skill_config is None:
# Default to enabled for all skill categories
return skill_category in ("public", "custom", "legacy", "integrations")
return skill_config.enabled
_extensions_config: ExtensionsConfig | None = None
def _fsync_directory_best_effort(directory: Path) -> None:
"""Persist a directory entry update where the platform supports it."""
if os.name == "nt":
return
try:
directory_fd = os.open(directory, os.O_RDONLY)
except OSError:
return
try:
os.fsync(directory_fd)
except OSError:
logger.debug("Could not fsync extensions config directory: %s", directory, exc_info=True)
finally:
try:
os.close(directory_fd)
except OSError:
logger.debug("Could not close extensions config directory: %s", directory, exc_info=True)
def _overwrite_in_place(target_path: Path, source_path: Path) -> None:
"""Copy *source_path* onto *target_path* without unlinking the destination inode.
Fallback for destinations that cannot be replaced by rename — see
:func:`atomic_write_extensions_config`. This deliberately truncates the
live file, so a crash mid-write leaves it short; the caller only reaches
this path when the atomic route is impossible.
"""
payload = source_path.read_bytes()
with open(target_path, "wb") as target_file:
target_file.write(payload)
target_file.flush()
os.fsync(target_file.fileno())
def _log_non_atomic_fallback(target_path: Path) -> None:
"""Warn once per target when a bind mount forces the unsafe write path."""
warning_key = target_path.resolve(strict=False)
with _non_atomic_fallback_targets_lock:
first_fallback = warning_key not in _non_atomic_fallback_targets
_non_atomic_fallback_targets.add(warning_key)
logger.log(
logging.WARNING if first_fallback else logging.DEBUG,
"Cannot atomically replace %s (it is a bind-mount point); overwriting in place. A crash during this write can leave the file truncated.",
target_path,
)
def atomic_write_extensions_config(path: Path, data: dict[str, Any]) -> None:
"""Write extensions config without exposing a truncated or partial file.
Falls back to a non-atomic in-place overwrite when the destination is a
bind-mounted file: Docker mounts ``extensions_config.json`` as its own
mount point, and the kernel refuses to rename over a mount point with
``EBUSY`` regardless of whether the mount is read-only. Without the
fallback every Gateway write to this file fails in the production
compose stack (MCP enable/disable, ``PUT``/``PATCH /api/mcp/config``,
skill updates), contradicting the documented promise that the file is
editable at runtime through the API.
"""
path = Path(path)
target_path = path.resolve(strict=False) if path.is_symlink() else path
target_path.parent.mkdir(parents=True, exist_ok=True)
existing_mode: int | None = None
try:
existing_mode = stat.S_IMODE(target_path.stat().st_mode)
except FileNotFoundError:
pass
temporary_path: Path | None = None
try:
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
dir=target_path.parent,
prefix=f".{target_path.name}.",
suffix=".tmp",
delete=False,
) as temporary_file:
temporary_path = Path(temporary_file.name)
json.dump(data, temporary_file, indent=2)
if existing_mode is not None:
temporary_path.chmod(existing_mode)
temporary_file.flush()
os.fsync(temporary_file.fileno())
try:
os.replace(temporary_path, target_path)
except OSError as exc:
if exc.errno != errno.EBUSY:
raise
_log_non_atomic_fallback(target_path)
_overwrite_in_place(target_path, temporary_path)
_fsync_directory_best_effort(target_path.parent)
finally:
if temporary_path is not None:
try:
temporary_path.unlink(missing_ok=True)
except OSError:
logger.warning(
"Could not remove temporary extensions config file: %s",
temporary_path,
exc_info=True,
)
def get_extensions_config() -> ExtensionsConfig:
"""Get the extensions config instance.
Returns a cached singleton instance. Use `reload_extensions_config()` to reload
from file, or `reset_extensions_config()` to clear the cache.
Returns:
The cached ExtensionsConfig instance.
"""
global _extensions_config
if _extensions_config is None:
_extensions_config = ExtensionsConfig.from_file()
return _extensions_config
#: Serializes read-modify-write cycles on ``extensions_config.json`` across every
#: writer. Both the skills router (skill enable/disable) and the MCP router
#: (server config updates) read this file, merge a change and write it back.
#: While each RMW ran inline on the event loop they were implicitly serialized;
#: once a writer offloads its RMW to a worker thread the loop is free to
#: interleave the other writer inside the read->write window, and the second
#: write silently drops the first one's change.
#:
#: This is a ``threading.Lock`` rather than an ``asyncio.Lock``, and it must be
#: acquired *inside* the worker that performs the RMW. An asyncio lock held
#: around ``await asyncio.to_thread(...)`` protects only the awaiting task: if
#: that task is cancelled the context manager releases immediately while the
#: worker thread keeps writing, letting a second writer in. Owning the lock from
#: the worker keeps it held until the write and reload actually finish. It also
#: has no event-loop affinity, so writers running on different loops still
#: exclude each other.
extensions_config_write_lock = threading.Lock()
@contextmanager
def extensions_config_file_lock(path: Path) -> Iterator[None]:
"""Exclude read-modify-write cycles in other Gateway processes.
``extensions_config_write_lock`` serializes threads in this process. This
sidecar advisory lock extends the same critical section across worker
processes and separate embedded clients that share the config directory.
Callers must hold both locks around the complete read, merge, write, and
reload cycle; locking only the final atomic replace still permits lost
updates.
"""
target_path = Path(path)
target_path = target_path.resolve(strict=False) if target_path.is_symlink() else target_path.absolute()
target_path.parent.mkdir(parents=True, exist_ok=True)
lock_path = target_path.parent / f".{target_path.name}.lock"
with open(lock_path, "a+b") as lock_file:
if os.name == "nt":
import msvcrt
lock_file.seek(0, os.SEEK_END)
if lock_file.tell() == 0:
lock_file.write(b"\0")
lock_file.flush()
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_LOCK, 1)
else:
import fcntl
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX)
try:
yield
finally:
if os.name == "nt":
lock_file.seek(0)
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
else:
fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
def reload_extensions_config(config_path: str | None = None) -> ExtensionsConfig:
"""Reload the extensions config from file and update the cached instance.
This is useful when the config file has been modified and you want
to pick up the changes without restarting the application.
Args:
config_path: Optional path to extensions config file. If not provided,
uses the default resolution strategy.
Returns:
The newly loaded ExtensionsConfig instance.
"""
global _extensions_config
_extensions_config = ExtensionsConfig.from_file(config_path)
return _extensions_config
def reset_extensions_config() -> None:
"""Reset the cached extensions config instance.
This clears the singleton cache, causing the next call to
`get_extensions_config()` to reload from file. Useful for testing
or when switching between different configurations.
"""
global _extensions_config
_extensions_config = None
def set_extensions_config(config: ExtensionsConfig) -> None:
"""Set a custom extensions config instance.
This allows injecting a custom or mock config for testing purposes.
Args:
config: The ExtensionsConfig instance to use.
"""
global _extensions_config
_extensions_config = config