mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-09 21:49:37 +00:00
fix: preserve assistant/tool history in compaction summaries (#5248)
* fix: preserve bounded assistant and tool input during compaction * fix: retain recent fallback summary input and clarify budget * fix: preserve recent content in mixed-history summary fallback * docs: trim middleware guidance to pass size check --------- Co-authored-by: Sami Belhareth <6599699+belharethsami@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Sami Belhareth
GitHub
parent
48bbea6df3
commit
f8f6cde23f
@ -1211,6 +1211,8 @@ The Web UI shows the active goal above the composer. The same command is availab
|
||||
|
||||
### Manual Context Compaction
|
||||
|
||||
Compaction keeps the current user request and summarizes older assistant/tool activity. When rescuing that request leaves an assistant/tool-only summary window, input trimming favors its most recent content. For mixed histories whose user-message anchor falls outside the trimming budget, compaction retains the existing final-message fallback. `summarization.trim_tokens_to_summarize` (4000 by default) controls trimming of the raw summary input; escaping and prompt formatting add overhead beyond that budget. Setting this option to `null` disables input trimming for the summary model; choose that only when the model can accept the full history being compacted.
|
||||
|
||||
Use `/compact` in the Web UI composer to summarize older context for the current thread. DeerFlow keeps the full chat visible, but future model calls use the compacted summary plus recent messages. The command is ignored when there is not enough history to compact, and it is blocked while the thread has a run in flight, including when that run is owned by another Gateway worker. If a multi-worker reservation loses its lease, DeerFlow cancels the checkpoint writer before the replacing run proceeds and returns a retryable conflict after cleanup. Thread-title edits are serialized through the same state-write boundary and show a conflict without closing the rename dialog when a run is active.
|
||||
|
||||
The chat header also shows a context-window gauge when the selected model has a positive `context_window` configured. It estimates the latest materialized checkpoint's message tokens and keeps the previous same-thread percentage visible while data refetches, independently of the cumulative token-usage setting.
|
||||
|
||||
@ -142,7 +142,7 @@ keep:
|
||||
#### `trim_tokens_to_summarize`
|
||||
- **Type**: Integer or null
|
||||
- **Default**: `4000`
|
||||
- **Description**: Maximum tokens to include when preparing messages for the summarization call itself. Set to `null` to skip trimming (not recommended for very long conversations).
|
||||
- **Description**: Token budget used to trim the raw input sections for the summarization call. Escaping, wrapper tags, and the summary prompt add overhead beyond this budget; it is not a hard limit on the final model request. When preserving the current user request leaves an assistant/tool-only summary window, trimming favors the most recent content in that window. If a mixed window still contains a human message but the human-anchored trim is empty, the existing final-message fallback is preserved. Set to `null` to skip trimming (not recommended for very long conversations).
|
||||
|
||||
#### `summary_prompt`
|
||||
- **Type**: String or null
|
||||
|
||||
@ -1,36 +1,37 @@
|
||||
### Middleware Chain
|
||||
|
||||
After latest-user rescue, if the inherited trimmer empties an AI/Tool-only
|
||||
window, format it and use `_build_summary_input_text(strategy="last")`.
|
||||
Keep normal human-anchored trimming and the final-message fallback for mixed
|
||||
windows whose human anchor falls outside the token-limited tail; head-first
|
||||
restoration can lose recent tool results. Tail truncation prefixes `\n...\n`
|
||||
only when marker and content fit. Budget raw sections before HTML escaping,
|
||||
wrappers, and prompt (not the final request); escape after trimming to preserve
|
||||
entities. Pass `trim_tokens_to_summarize=None` explicitly through the factory;
|
||||
omission restores LangChain's 4000-token default.
|
||||
|
||||
Persisted delegation verdicts are untrusted durable context; ledger rendering revalidates them and ignores malformed values.
|
||||
|
||||
Assembly order: `tool_error_handling_middleware.py::_build_runtime_middlewares` (exposed as `build_lead_runtime_middlewares`), then `../lead_agent/agent.py::build_middlewares` appends lead-only entries. Optional entries require their config/runtime condition.
|
||||
|
||||
**Message provenance.** A middleware that injects or rewrites a message stamps
|
||||
`additional_kwargs` with the neutral provenance keys from
|
||||
`deerflow_extension_api.provenance` (`deerflow_content_kind`,
|
||||
`deerflow_producer_kind`, and optionally `deerflow_producer_entity_id`) via
|
||||
`provenance_kwargs()`. Stamp at injection/rewrite regardless of installed
|
||||
observers; downstream cannot recover the producer. All three
|
||||
keys are in `_SERVER_OWNED_MESSAGE_METADATA_KEYS`, so a caller cannot forge
|
||||
provenance on inbound messages. Currently stamped by: DynamicContext (reminder + memory),
|
||||
DurableContext (contract + data), SystemMessageCoalescing, ViewImage,
|
||||
SkillActivation. Summarization, Title, and Memory are deliberately absent:
|
||||
Summarization's and Title's own model calls are already attributed through
|
||||
system-model-call observation (`SystemOperationKind.SUMMARIZATION` /
|
||||
`.TITLE`), and the summary text they produce only ever enters a request via
|
||||
`DurableContextMiddleware`'s already-stamped `durable_context_data` block —
|
||||
there is no separate message of theirs to stamp. Memory only *reads*
|
||||
messages to queue them for extraction; the recalled-memory content that
|
||||
actually re-enters context is DynamicContext's `dynamic_context_memory`
|
||||
stamp, not anything Memory itself produces.
|
||||
**Message provenance.** At injection/rewrite, always stamp `additional_kwargs`
|
||||
via `deerflow_extension_api.provenance.provenance_kwargs()`:
|
||||
`deerflow_content_kind`, `deerflow_producer_kind`, optional
|
||||
`deerflow_producer_entity_id`. All are server-owned inbound metadata; stamp even
|
||||
without observers, since downstream cannot recover producers. Producers:
|
||||
DynamicContext (reminder/memory), DurableContext (contract/data),
|
||||
SystemMessageCoalescing, ViewImage, SkillActivation. Summarization/Title use
|
||||
`SystemOperationKind.SUMMARIZATION`/`.TITLE` model-call attribution; summaries
|
||||
enter via DurableContext's stamped `durable_context_data`, not separate
|
||||
messages. Memory only queues extraction; recall uses DynamicContext's
|
||||
`dynamic_context_memory` stamp.
|
||||
|
||||
**Middleware self-description.** A middleware whose configuration changes agent
|
||||
behaviour implements `release_policy_parameters() -> dict[str, object]`
|
||||
(`deerflow_extension_api.release.ReleasePolicyProvider`, duck-typed — no base
|
||||
class). Values must be JSON-serialisable; long text is hashed with
|
||||
`canonical_hash` rather than embedded, because a declaration is an identity and
|
||||
not a copy of the prompt. `collect_release_policies()` gathers them from an
|
||||
assembled stack. Adding a behaviour-affecting field to a middleware means adding
|
||||
it to that middleware's declaration in the same change.
|
||||
**Middleware self-description.** Behaviour-configurable middleware implements
|
||||
`release_policy_parameters() -> dict[str, object]` (duck-typed
|
||||
`deerflow_extension_api.release.ReleasePolicyProvider`, no base class).
|
||||
Use JSON-serialisable values and `canonical_hash` for long text, not prompt
|
||||
copies. `collect_release_policies()` gathers stack declarations; update them
|
||||
alongside every behaviour-affecting field.
|
||||
|
||||
**Shared runtime base** (`build_lead_runtime_middlewares`; subagents reuse most of this via `build_subagent_runtime_middlewares`):
|
||||
|
||||
|
||||
@ -6,7 +6,7 @@ import html
|
||||
import logging
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Protocol, override, runtime_checkable
|
||||
from typing import Any, Literal, Protocol, override, runtime_checkable
|
||||
|
||||
from deerflow_extension_api import CompactionEvent, canonical_hash
|
||||
from langchain.agents import AgentState
|
||||
@ -452,9 +452,15 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
|
||||
return content
|
||||
except Exception:
|
||||
logger.debug("Failed to trim summary prompt section with token counter; falling back to deterministic text cap", exc_info=True)
|
||||
if strategy == "last":
|
||||
omitted_marker = "\n...\n"
|
||||
if len(text) > max_tokens and max_tokens > len(omitted_marker):
|
||||
return omitted_marker + text[-(max_tokens - len(omitted_marker)) :]
|
||||
return text[-max_tokens:]
|
||||
return self._bound_text(text, max_tokens)
|
||||
|
||||
def _build_summary_input_text(self, formatted_messages: str, previous_summary: str | None = None) -> str | None:
|
||||
def _build_summary_input_text(self, formatted_messages: str, previous_summary: str | None = None, *, new_messages_strategy: Literal["first", "last"] = "first") -> str | None:
|
||||
"""Trim raw input sections before adding escaping and prompt overhead."""
|
||||
if self.trim_tokens_to_summarize is None:
|
||||
trimmed_new_messages = formatted_messages
|
||||
trimmed_previous_summary = previous_summary.strip() if previous_summary else ""
|
||||
@ -471,14 +477,14 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
|
||||
trimmed_new_messages = self._trim_summary_section_text(
|
||||
formatted_messages,
|
||||
new_message_tokens,
|
||||
strategy="first",
|
||||
strategy=new_messages_strategy,
|
||||
)
|
||||
else:
|
||||
trimmed_previous_summary = ""
|
||||
trimmed_new_messages = self._trim_summary_section_text(
|
||||
formatted_messages,
|
||||
max_tokens,
|
||||
strategy="first",
|
||||
strategy=new_messages_strategy,
|
||||
)
|
||||
|
||||
# Escape < > & before embedding into the <existing_summary>/<new_messages>
|
||||
@ -516,14 +522,24 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
|
||||
def _build_summary_prompt(self, messages_to_summarize: list[AnyMessage], previous_summary: str | None = None) -> str | None:
|
||||
"""Build the summary prompt, returning ``None`` when trimming leaves nothing."""
|
||||
trimmed_messages = self._trim_messages_for_summary(messages_to_summarize)
|
||||
new_messages_strategy: Literal["first", "last"] = "first"
|
||||
if not trimmed_messages:
|
||||
trimmed_messages = messages_to_summarize[-1:]
|
||||
if any(isinstance(message, HumanMessage) for message in messages_to_summarize):
|
||||
# The human anchor can fall outside the token-limited tail.
|
||||
# Preserve the existing final-message fallback for this case.
|
||||
trimmed_messages = messages_to_summarize[-1:]
|
||||
else:
|
||||
# Rescuing the current request can leave an AI/Tool-only window,
|
||||
# which the inherited human-anchored trimmer rejects even below
|
||||
# budget. Bound its raw text while favoring recent content.
|
||||
trimmed_messages = messages_to_summarize
|
||||
new_messages_strategy = "last"
|
||||
if not trimmed_messages:
|
||||
return None
|
||||
# Format messages to avoid token inflation from metadata when str() is called on
|
||||
# message objects.
|
||||
formatted_messages = get_buffer_string(trimmed_messages)
|
||||
formatted_messages = self._build_summary_input_text(formatted_messages, previous_summary=previous_summary)
|
||||
formatted_messages = self._build_summary_input_text(formatted_messages, previous_summary=previous_summary, new_messages_strategy=new_messages_strategy)
|
||||
if not formatted_messages:
|
||||
return None
|
||||
return self.summary_prompt.format(messages=formatted_messages).rstrip()
|
||||
@ -953,9 +969,8 @@ def create_summarization_middleware(
|
||||
"model": anchor_model,
|
||||
"trigger": trigger,
|
||||
"keep": keep_tuple,
|
||||
"trim_tokens_to_summarize": config.trim_tokens_to_summarize,
|
||||
}
|
||||
if config.trim_tokens_to_summarize is not None:
|
||||
kwargs["trim_tokens_to_summarize"] = config.trim_tokens_to_summarize
|
||||
if config.summary_prompt is not None:
|
||||
kwargs["summary_prompt"] = config.summary_prompt
|
||||
|
||||
|
||||
@ -1092,6 +1092,18 @@ def _factory_app_config(model_names, *, summary_model_name=None, summarization_k
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("trim_limit", [None, 80, 4000])
|
||||
def test_factory_preserves_explicit_summary_input_limit(monkeypatch, trim_limit):
|
||||
monkeypatch.setattr("deerflow.agents.middlewares.summarization_middleware.create_chat_model", _tracking_create_chat_model([]))
|
||||
config = _factory_app_config(("run-model",))
|
||||
config.summarization.trim_tokens_to_summarize = trim_limit
|
||||
|
||||
middleware = create_summarization_middleware(app_config=config, run_model_name="run-model", keep=("messages", 2))
|
||||
|
||||
assert middleware is not None
|
||||
assert middleware.trim_tokens_to_summarize == trim_limit
|
||||
|
||||
|
||||
def test_factory_null_case_anchor_is_run_model_not_models0(monkeypatch):
|
||||
"""model_name: null builds the summary model from ``run_model_name``, never
|
||||
config.models[0]. A run on a non-default model whose models[0] provider is broken
|
||||
|
||||
@ -1,9 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.messages import AIMessage, HumanMessage, RemoveMessage, SystemMessage
|
||||
from langchain_core.messages import AIMessage, HumanMessage, RemoveMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.outputs import ChatGeneration, ChatResult
|
||||
from pydantic import Field
|
||||
|
||||
@ -82,6 +84,153 @@ class TestSummaryFailureSafety:
|
||||
|
||||
|
||||
class TestSummaryWritesChannel:
|
||||
@pytest.mark.parametrize("async_mode", [False, True])
|
||||
@pytest.mark.asyncio
|
||||
async def test_rescued_user_does_not_drop_earlier_tool_exchanges(self, async_mode):
|
||||
model = _RecordingSummaryModel()
|
||||
middleware = DeerFlowSummarizationMiddleware(
|
||||
model=model,
|
||||
trigger=("messages", 4),
|
||||
keep=("messages", 2),
|
||||
token_counter=_char_count,
|
||||
trim_tokens_to_summarize=4000,
|
||||
)
|
||||
user = HumanMessage(content="CURRENT_REQUEST", id="user")
|
||||
history = [user]
|
||||
for i in range(3):
|
||||
history.extend(
|
||||
[
|
||||
AIMessage(content=f"PLAN_{i}", id=f"ai-{i}", tool_calls=[{"name": "bash", "args": {}, "id": f"call-{i}"}]),
|
||||
ToolMessage(content=f"RESULT_{i}", id=f"tool-{i}", tool_call_id=f"call-{i}"),
|
||||
]
|
||||
)
|
||||
runtime = SimpleNamespace(context={})
|
||||
if async_mode:
|
||||
result = await middleware.acompact_state({"messages": history}, runtime, force=True)
|
||||
else:
|
||||
result = middleware.compact_state({"messages": history}, runtime, force=True)
|
||||
|
||||
assert result is not None
|
||||
assert user in result.preserved_messages
|
||||
assert list(result.messages_to_summarize) == history[1:5]
|
||||
assert len(model.prompts) == 1
|
||||
for sentinel in ("PLAN_0", "RESULT_0", "PLAN_1", "RESULT_1"):
|
||||
assert sentinel in model.prompts[0]
|
||||
assert "CURRENT_REQUEST" not in model.prompts[0]
|
||||
assert "RESULT_2" not in model.prompts[0]
|
||||
|
||||
@pytest.mark.parametrize("async_mode", [False, True])
|
||||
@pytest.mark.parametrize("previous_summary", [None, "O" * 1000 + " OLD_END"], ids=["without-summary", "with-summary"])
|
||||
@pytest.mark.parametrize("trim_limit", [120, None], ids=["bounded", "untrimmed"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_oversized_rescued_user_window_keeps_recent_exchanges(self, async_mode, previous_summary, trim_limit):
|
||||
model = _RecordingSummaryModel()
|
||||
middleware = DeerFlowSummarizationMiddleware(
|
||||
model=model,
|
||||
trigger=("messages", 4),
|
||||
keep=("messages", 2),
|
||||
token_counter=_char_count,
|
||||
trim_tokens_to_summarize=trim_limit,
|
||||
)
|
||||
user = HumanMessage(content="CURRENT_REQUEST", id="user")
|
||||
history = [user]
|
||||
for i in range(7):
|
||||
history.extend(
|
||||
[
|
||||
AIMessage(content=f"PLAN_{i}", id=f"ai-{i}", tool_calls=[{"name": "bash", "args": {}, "id": f"call-{i}"}]),
|
||||
ToolMessage(content=f"RESULT_{i}", id=f"tool-{i}", tool_call_id=f"call-{i}"),
|
||||
]
|
||||
)
|
||||
state = {"messages": history, "summary_text": previous_summary}
|
||||
runtime = SimpleNamespace(context={})
|
||||
if async_mode:
|
||||
result = await middleware.acompact_state(state, runtime, force=True)
|
||||
else:
|
||||
result = middleware.compact_state(state, runtime, force=True)
|
||||
|
||||
assert result is not None
|
||||
assert user in result.preserved_messages
|
||||
assert list(result.messages_to_summarize) == history[1:13]
|
||||
assert len(model.prompts) == 1
|
||||
new_text = model.prompts[0].split("<new_messages>\n", 1)[1].split("\n</new_messages>", 1)[0]
|
||||
assert "RESULT_5" in new_text
|
||||
if trim_limit is None:
|
||||
for i in range(6):
|
||||
assert f"PLAN_{i}" in new_text
|
||||
assert f"RESULT_{i}" in new_text
|
||||
assert len(new_text) > 120
|
||||
if previous_summary:
|
||||
assert previous_summary in model.prompts[0]
|
||||
else:
|
||||
assert "RESULT_0" not in new_text
|
||||
assert "CURRENT_REQUEST" not in new_text
|
||||
assert "RESULT_6" not in new_text
|
||||
if previous_summary:
|
||||
assert "OLD_END" in model.prompts[0]
|
||||
else:
|
||||
assert "PLAN_5" in new_text
|
||||
|
||||
@pytest.mark.parametrize("async_mode", [False, True])
|
||||
@pytest.mark.asyncio
|
||||
async def test_mixed_history_empty_trim_preserves_recent_tool_result(self, async_mode):
|
||||
model = _RecordingSummaryModel()
|
||||
middleware = DeerFlowSummarizationMiddleware(
|
||||
model=model,
|
||||
trigger=("messages", 4),
|
||||
keep=("messages", 2),
|
||||
token_counter=_char_count,
|
||||
trim_tokens_to_summarize=120,
|
||||
)
|
||||
user = HumanMessage(content="CURRENT_REQUEST", id="user")
|
||||
history = [HumanMessage(content="OLD_REQUEST " + "x" * 1000, id="old-user"), user]
|
||||
for i in range(6):
|
||||
history.extend(
|
||||
[
|
||||
AIMessage(content=f"PLAN_{i}", id=f"ai-{i}", tool_calls=[{"name": "bash", "args": {}, "id": f"call-{i}"}]),
|
||||
ToolMessage(content=f"RESULT_{i}", id=f"tool-{i}", tool_call_id=f"call-{i}"),
|
||||
]
|
||||
)
|
||||
runtime = SimpleNamespace(context={})
|
||||
if async_mode:
|
||||
result = await middleware.acompact_state({"messages": history}, runtime, force=True)
|
||||
else:
|
||||
result = middleware.compact_state({"messages": history}, runtime, force=True)
|
||||
|
||||
assert result is not None
|
||||
assert list(result.messages_to_summarize) == [history[0], *history[2:12]]
|
||||
assert list(result.preserved_messages) == [user, *history[12:]]
|
||||
assert len(model.prompts) == 1
|
||||
new_text = model.prompts[0].split("<new_messages>\n", 1)[1].split("\n</new_messages>", 1)[0]
|
||||
assert new_text == "Tool: RESULT_4"
|
||||
assert "CURRENT_REQUEST" not in model.prompts[0]
|
||||
assert "RESULT_5" not in model.prompts[0]
|
||||
|
||||
def test_tool_only_fallback_applies_budget_before_escaping(self):
|
||||
middleware = DeerFlowSummarizationMiddleware(
|
||||
model=_StaticChatModel(),
|
||||
trigger=("messages", 4),
|
||||
keep=("messages", 2),
|
||||
token_counter=_char_count,
|
||||
trim_tokens_to_summarize=80,
|
||||
)
|
||||
prompt = middleware._build_summary_prompt(
|
||||
[ToolMessage(content="<" * 1000 + " TOOL_END", tool_call_id="call")],
|
||||
previous_summary="&" * 1000 + " OLD_END",
|
||||
)
|
||||
|
||||
assert prompt is not None
|
||||
new_text = prompt.split("<new_messages>\n", 1)[1].split("\n</new_messages>", 1)[0]
|
||||
old_text = prompt.split("<existing_summary>\n", 1)[1].split("\n</existing_summary>", 1)[0]
|
||||
# The raw input budget excludes escaping and the surrounding prompt.
|
||||
assert len(html.unescape(new_text)) + len(html.unescape(old_text)) <= 80
|
||||
assert len(new_text) + len(old_text) > 80
|
||||
assert "<" in new_text
|
||||
assert "&" in old_text
|
||||
assert "<" not in new_text
|
||||
assert "<" not in old_text
|
||||
assert "TOOL_END" in new_text
|
||||
assert "OLD_END" in old_text
|
||||
|
||||
def _middleware(self) -> DeerFlowSummarizationMiddleware:
|
||||
return DeerFlowSummarizationMiddleware(
|
||||
model=_StaticChatModel(text="COMPRESSED_SUMMARY"),
|
||||
@ -218,7 +367,8 @@ class TestSummaryWritesChannel:
|
||||
assert len(new_messages) <= 40
|
||||
assert "NEW_MESSAGE_SENTINEL" in new_messages
|
||||
|
||||
def test_summary_prompt_fallback_bound_respects_small_budget(self):
|
||||
@pytest.mark.parametrize(("strategy", "expected"), [("first", "ab"), ("last", "ef")])
|
||||
def test_summary_prompt_fallback_bound_respects_small_budget(self, strategy, expected):
|
||||
middleware = DeerFlowSummarizationMiddleware(
|
||||
model=_StaticChatModel(text="UPDATED_SUMMARY"),
|
||||
trigger=("messages", 4),
|
||||
@ -227,6 +377,18 @@ class TestSummaryWritesChannel:
|
||||
trim_tokens_to_summarize=2,
|
||||
)
|
||||
|
||||
text = middleware._trim_summary_section_text("abcdef", 2, strategy="first")
|
||||
text = middleware._trim_summary_section_text("abcdef", 2, strategy=strategy)
|
||||
|
||||
assert len(text) <= 2
|
||||
assert text == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("cap", "expected"),
|
||||
[(1, "i"), (2, "hi"), (5, "efghi"), (6, "\n...\ni"), (8, "\n...\nghi"), (9, "abcdefghi"), (20, "abcdefghi")],
|
||||
)
|
||||
def test_tail_fallback_marks_omitted_text_within_budget(self, cap, expected):
|
||||
middleware = DeerFlowSummarizationMiddleware(model=_StaticChatModel(), token_counter=_raising_count)
|
||||
|
||||
text = middleware._trim_summary_section_text("abcdefghi", cap, strategy="last")
|
||||
|
||||
assert text == expected
|
||||
assert len(text) <= cap
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user