diff --git a/README.md b/README.md index 2ed78d765e..0fb635fea8 100644 --- a/README.md +++ b/README.md @@ -304,7 +304,7 @@ streams simulate chunks from a non-streaming response, while no-tool streams sta Models whose provider contract differs from DeerFlow's generic thinking/effort assumptions can declare a per-model mapping-valued `reasoning:` block (thinking `unsupported`/`optional`/`required`, the accepted effort values with aliases and a default, the payload dialect, and the reasoning-history requirement). The setup wizard's Z.AI GLM-5.3-Flash profile uses it: thinking stays on for every foreground and background call, and the effort selector offers the model's own `low`/`high`/`max` levels. Ollama's existing boolean `reasoning: true` remains a native provider setting and is forwarded to ChatOllama. When migrating a profile to a custom effort `path`, remove any old `reasoning_effort` setting from the profile and thinking templates; configuration validation rejects the leftover key. The chat UI drops a remembered provider-specific effort when switching to a legacy model that does not advertise it. Profiles without the block keep their existing provider behavior. See `config.example.yaml` for the shape and the equivalent manual configuration. - For vLLM 0.19.0, use `deerflow.models.vllm_provider:VllmChatModel`. For Qwen-style reasoning models, DeerFlow toggles reasoning with `extra_body.chat_template_kwargs.enable_thinking` and preserves vLLM's non-standard `reasoning` field across multi-turn tool-call conversations. Legacy `thinking` configs are normalized automatically for backward compatibility. If the endpoint reports a cumulative usage snapshot on every streaming chunk, set `cumulative_stream_usage: true` so DeerFlow converts those snapshots into per-chunk deltas; the option is disabled by default and leaves usage unchanged when a stable completion id is unavailable. Reasoning models may also require the server to be started with `--reasoning-parser ...`. If your local vLLM deployment accepts any non-empty API key, you can still set `VLLM_API_KEY` to a placeholder value. + For vLLM 0.19.0, use `deerflow.models.vllm_provider:VllmChatModel`. For Qwen-style reasoning models, DeerFlow toggles reasoning with `extra_body.chat_template_kwargs.enable_thinking` and preserves vLLM's non-standard `reasoning` field across multi-turn tool-call conversations. Legacy `thinking` configs are normalized automatically for backward compatibility, without modifying model defaults or caller-owned `extra_body` dictionaries; a reused dictionary can change the legacy switch between requests. If the endpoint reports a cumulative usage snapshot on every streaming chunk, set `cumulative_stream_usage: true` so DeerFlow converts those snapshots into per-chunk deltas; the option is disabled by default and leaves usage unchanged when a stable completion id is unavailable. Reasoning models may also require the server to be started with `--reasoning-parser ...`. If your local vLLM deployment accepts any non-empty API key, you can still set `VLLM_API_KEY` to a placeholder value. When prompt caching is enabled for a model configured with `deerflow.models.claude_provider:ClaudeChatModel`, DeerFlow preserves thinking diff --git a/backend/packages/harness/deerflow/models/AGENTS.md b/backend/packages/harness/deerflow/models/AGENTS.md index c8828e290b..3fffbd8b75 100644 --- a/backend/packages/harness/deerflow/models/AGENTS.md +++ b/backend/packages/harness/deerflow/models/AGENTS.md @@ -155,6 +155,7 @@ finite negative values clamp to zero. Sync/async 429/500 regressions live in - `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints - Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns, falling back to the legacy `reasoning_content` wire field when `reasoning` is absent or null (a payload carrying both keeps `reasoning`); `_pick_reasoning` owns this precedence across all three paths and preserves empty-string `reasoning` rather than falling back - Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias +- Normalize the legacy thinking alias on a request-owned `extra_body` copy. Never modify model defaults or caller-owned mappings: a reused request must be able to switch `thinking` off without inheriting a synthesized `enable_thinking=True`. Preserve explicit canonical-key precedence and unrelated fields. Coverage: `tests/test_vllm_provider.py`. - `cumulative_stream_usage` is an opt-in model setting (default `false`) for endpoints that repeat cumulative token totals on each streaming chunk. The provider converts snapshots to deltas only when a stable completion id is present, isolates interleaved streams by id, and leaves the original usage untouched otherwise. Per-model tracking is lock-protected and cleared on the trailing empty-`choices` frame whether or not that frame carries usage. A soft cap of 1024 ids evicts only entries idle for at least one hour; active streams may temporarily exceed the cap so eviction cannot corrupt their deltas. Regression coverage lives in `tests/test_vllm_provider.py`. ### MindIE Provider (`packages/harness/deerflow/models/mindie_provider.py`) diff --git a/backend/packages/harness/deerflow/models/vllm_provider.py b/backend/packages/harness/deerflow/models/vllm_provider.py index 7eca9c012d..1d66a453cf 100644 --- a/backend/packages/harness/deerflow/models/vllm_provider.py +++ b/backend/packages/harness/deerflow/models/vllm_provider.py @@ -72,7 +72,8 @@ def _normalize_vllm_chat_template_kwargs(payload: dict[str, Any]) -> None: normalized_chat_template_kwargs = dict(chat_template_kwargs) normalized_chat_template_kwargs.setdefault("enable_thinking", normalized_chat_template_kwargs["thinking"]) normalized_chat_template_kwargs.pop("thinking", None) - extra_body["chat_template_kwargs"] = normalized_chat_template_kwargs + # LangChain may reuse the caller's or model's extra_body mapping across requests. + payload["extra_body"] = {**extra_body, "chat_template_kwargs": normalized_chat_template_kwargs} def _reasoning_to_text(reasoning: Any) -> str: diff --git a/backend/tests/test_vllm_provider.py b/backend/tests/test_vllm_provider.py index 3f98b6ca85..b1002c0beb 100644 --- a/backend/tests/test_vllm_provider.py +++ b/backend/tests/test_vllm_provider.py @@ -1,5 +1,9 @@ from __future__ import annotations +import copy +import json + +import httpx import pytest from langchain_core.messages import AIMessage, AIMessageChunk, HumanMessage, ToolMessage @@ -140,6 +144,77 @@ def test_vllm_provider_preserves_explicit_enable_thinking_kwarg(): } +def test_vllm_provider_keeps_legacy_model_defaults_unmodified(): + extra_body = {"chat_template_kwargs": {"thinking": True}, "tool_stream": True} + original = copy.deepcopy(extra_body) + model = VllmChatModel(model="qwen3", api_key="dummy", extra_body=extra_body) + + payload = model._get_request_payload([HumanMessage(content="Hello")]) + + assert payload["extra_body"] == {"chat_template_kwargs": {"enable_thinking": True}, "tool_stream": True} + assert model.extra_body == original + assert extra_body == original + + +@pytest.mark.asyncio +@pytest.mark.parametrize("async_mode", [False, True], ids=["sync", "async"]) +@pytest.mark.parametrize("streaming", [False, True], ids=["invoke", "stream"]) +async def test_vllm_provider_reused_request_body_can_disable_thinking(async_mode, streaming): + requests = [] + + def handle(request): + body = json.loads(request.content) + requests.append(body) + if body.get("stream"): + chunks = [ + {"id": "completion", "object": "chat.completion.chunk", "created": 0, "model": "qwen3", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "ok"}, "finish_reason": None}]}, + {"id": "completion", "object": "chat.completion.chunk", "created": 0, "model": "qwen3", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}, + ] + content = "".join(f"data: {json.dumps(chunk)}\n\n" for chunk in chunks) + "data: [DONE]\n\n" + return httpx.Response(200, headers={"content-type": "text/event-stream"}, content=content) + return httpx.Response(200, json={"id": "completion", "model": "qwen3", "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}]}) + + async def call(model, prompt, extra_body): + if streaming: + if async_mode: + chunks = [chunk async for chunk in model.astream(prompt, extra_body=extra_body)] + else: + chunks = list(model.stream(prompt, extra_body=extra_body)) + assert "".join(chunk.content for chunk in chunks) == "ok" + assert any(chunk.response_metadata.get("finish_reason") == "stop" for chunk in chunks) + elif async_mode: + assert (await model.ainvoke(prompt, extra_body=extra_body)).content == "ok" + else: + assert model.invoke(prompt, extra_body=extra_body).content == "ok" + + extra_body = {"chat_template_kwargs": {"thinking": True, "foo": "bar"}, "tool_stream": True} + original = copy.deepcopy(extra_body) + with httpx.Client(transport=httpx.MockTransport(handle)) as sync_client: + async with httpx.AsyncClient(transport=httpx.MockTransport(handle)) as async_client: + model = VllmChatModel(model="qwen3", api_key="dummy", base_url="https://offline.invalid/v1", http_client=sync_client, http_async_client=async_client, max_retries=0) + await call(model, "first", extra_body) + after_first = copy.deepcopy(extra_body) + extra_body["chat_template_kwargs"]["thinking"] = False + await call(model, "second", extra_body) + + assert [request["stream"] for request in requests] == [streaming, streaming] + assert [request["chat_template_kwargs"]["enable_thinking"] for request in requests] == [True, False] + assert all(request["chat_template_kwargs"]["foo"] == "bar" and request["tool_stream"] is True for request in requests) + assert after_first == original + assert extra_body == {"chat_template_kwargs": {"thinking": False, "foo": "bar"}, "tool_stream": True} + + +@pytest.mark.parametrize("enabled", [False, True]) +def test_vllm_provider_explicit_switch_wins_without_mutating_request(enabled): + extra_body = {"chat_template_kwargs": {"thinking": not enabled, "enable_thinking": enabled}, "tool_stream": True} + original = copy.deepcopy(extra_body) + + payload = _make_model()._get_request_payload([HumanMessage(content="Hello")], extra_body=extra_body) + + assert payload["extra_body"] == {"chat_template_kwargs": {"enable_thinking": enabled}, "tool_stream": True} + assert extra_body == original + + def test_vllm_provider_preserves_reasoning_in_chat_result(): model = _make_model() result = model._create_chat_result(