fix(subagents): keep the subagent system prompt through context compaction (#5454)

* fix(subagents): keep the subagent system prompt through context compaction

A subagent carries its whole system prompt (role, report contract, skills,
deferred tools) as the leading SystemMessage in state, because its agent is
built with system_prompt=None. Summarization only rescued tagged reminders and
the latest user message, so the first compaction summarized the prompt away
and every later model call in that subagent ran without its instructions.

Rescue system messages as well. The lead agent's prompt is not in state, and
its in-state SystemMessages are the tagged reminders already rescued, so the
lead chain is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(compaction): clarify preservation contract and trim agent guidance

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

README.md

@@ -1325,7 +1325,7 @@ same optional field is supported in the agent's `config.yaml`.
 
 Sub-agents are an optimization, not the default response to a complex request.
 
-The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions — when delegation has clear net benefit from real parallel latency, specialist capability, or context isolation. It keeps interdependent scopes and overlapping side effects out of parallel dispatch; a bounded sequential chain can still run in one sub-agent when specialist or context-isolation benefit clearly wins. The lead uses the fewest useful sub-agents and re-evaluates later batches instead of fanning out solely because a task is large or multi-step. Sub-agents report back structured results, and the lead agent verifies and synthesizes them into a coherent output. Deterministic tool receipts cover both direct tool messages and state-updating `Command` results such as delegated `task` responses; when the receipt ledger reaches its context budget, it retains the newest actions and their original receipt IDs. Operators can disable this provenance layer with `verification.receipts_enabled: false`. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Reloaded thread history enforces the same boundary: callback-captured sub-agent AI responses remain available in run-event diagnostics but are excluded from the parent transcript, while the parent `task` result remains attached to its subtask card. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Concurrent parent runs also receive independent server-side sub-agent execution IDs, so a provider that reuses a tool-call ID cannot make one run poll, cancel, or clean up another run's background task. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is attributed back to the dispatching step from that run's terminal tool-message metadata rather than a process-global provider-ID cache.
+The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions — when delegation has clear net benefit from real parallel latency, specialist capability, or context isolation. It keeps interdependent scopes and overlapping side effects out of parallel dispatch; a bounded sequential chain can still run in one sub-agent when specialist or context-isolation benefit clearly wins. The lead uses the fewest useful sub-agents and re-evaluates later batches instead of fanning out solely because a task is large or multi-step. Sub-agents report back structured results, and the lead agent verifies and synthesizes them into a coherent output. Deterministic tool receipts cover both direct tool messages and state-updating `Command` results such as delegated `task` responses; when the receipt ledger reaches its context budget, it retains the newest actions and their original receipt IDs. Operators can disable this provenance layer with `verification.receipts_enabled: false`. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Reloaded thread history enforces the same boundary: callback-captured sub-agent AI responses remain available in run-event diagnostics but are excluded from the parent transcript, while the parent `task` result remains attached to its subtask card. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Their system instructions, including the role and report contract, survive compaction; if only those instructions and the current request would be summarized, compaction is skipped. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Concurrent parent runs also receive independent server-side sub-agent execution IDs, so a provider that reuses a tool-call ID cannot make one run poll, cancel, or clean up another run's background task. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is attributed back to the dispatching step from that run's terminal tool-message metadata rather than a process-global provider-ID cache.
 
 An ordinary `task` also receives a defensive snapshot of the dispatching run's current uploads. This lets eligible sub-agents use `list_uploaded_files` to find earlier-turn files without returning same-turn attachments as historical. Delayed or recovered `batch_task` workers leave this tool disabled because they have no valid turn-local upload boundary.
 

backend/AGENTS.md

@@ -84,23 +84,17 @@ regression exercises the production extractor under a generous process deadline.
 ## Important Development Guidelines
 
 ### Documentation Update Policy
-**CRITICAL: Always update README.md and AGENTS.md after every code change**
-
-When making code changes, you MUST update the relevant documentation:
-- Update `README.md` for user-facing changes (features, setup, usage instructions)
-- Update `AGENTS.md` for development changes (architecture, commands, workflows, internal systems). `CLAUDE.md` imports it via `@AGENTS.md`, so editing `AGENTS.md` updates both.
-- Keep documentation synchronized with the codebase at all times
-- Ensure accuracy and timeliness of all documentation
+Every code change must keep docs accurate and current: update `README.md` for
+user-facing behavior and the relevant `AGENTS.md` for development changes.
+`CLAUDE.md` imports `AGENTS.md`; do not edit the shim.
 
 ### Backend Benchmarks
 
 `scripts/benchmark/context_snapshot/`: explicit `run-live` needs provider env
 vars; `summarize` and pytest are offline. See its README for the protocol.
 
-`scripts/benchmark/` contains standalone, reproducible measurements and
-evaluations of production backend behavior. A benchmark may import the
-production function it measures, but it must not duplicate or introduce an
-alternative runtime implementation.
+Benchmarks in `scripts/benchmark/` must be standalone and reproducible. Import
+production functions; never duplicate them or introduce an alternative runtime.
 
 - Pin every external dataset by immutable revision and SHA-256. Callers provide
   the local dataset path; evaluation commands must not silently download data.
@@ -280,12 +274,7 @@ InfoQuest connect/read timeout is 30s, separate from crawl timeouts (`tests/test
 
 ### Running the Full Application
 
-From the **project root** directory:
-```bash
-make dev
-```
-
-This starts all services and makes the application available at `http://localhost:2026`.
+Run `make dev` from the repo root to start all services at `http://localhost:2026`.
 
 **All startup modes:**
 

backend/packages/harness/deerflow/agents/middlewares/AGENTS.md

@@ -1,5 +1,9 @@
 ### Middleware Chain
 
+Compaction preserves all state-level `SystemMessage`s as framework instructions,
+including untagged legacy reminders. Transient instructions belong in request
+wrappers. A fully rescued partition skips compaction.
+
 After latest-user rescue, if the inherited trimmer empties an AI/Tool-only
 window, format it and use `_build_summary_input_text(strategy="last")`.
 Keep normal human-anchored trimming and the final-message fallback for mixed

backend/packages/harness/deerflow/agents/middlewares/summarization_middleware.py

@@ -11,7 +11,7 @@ from typing import Any, Literal, Protocol, override, runtime_checkable
 from deerflow_extension_api import CompactionEvent, canonical_hash
 from langchain.agents import AgentState
 from langchain.agents.middleware import SummarizationMiddleware
-from langchain_core.messages import AnyMessage, HumanMessage, RemoveMessage, get_buffer_string, trim_messages
+from langchain_core.messages import AnyMessage, HumanMessage, RemoveMessage, SystemMessage, get_buffer_string, trim_messages
 from langgraph.config import get_config
 from langgraph.constants import TAG_NOSTREAM
 from langgraph.graph.message import REMOVE_ALL_MESSAGES
@@ -575,7 +575,7 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
             return None
 
         # The latest real user message (the current request) must survive: peer
-        # rescue no longer covers it (see _preserve_dynamic_context_reminders), so
+        # rescue no longer covers it (see _preserve_required_context), so
         # lock its id here and rescue by exact id. This keeps the current request
         # without "moving cutoff" — which would also retain early AI/Tool turns and
         # never compress a first-turn long analysis. A Human Input Card reply is
@@ -587,7 +587,7 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
                 break
 
         messages_to_summarize, preserved_messages = self._partition_messages(messages, cutoff_index)
-        messages_to_summarize, preserved_messages = self._preserve_dynamic_context_reminders(messages_to_summarize, preserved_messages, latest_user_id=latest_user_id)
+        messages_to_summarize, preserved_messages = self._preserve_required_context(messages_to_summarize, preserved_messages, latest_user_id=latest_user_id)
         if not messages_to_summarize:
             return None
         return messages_to_summarize, preserved_messages, previous_summary, total_tokens
@@ -760,28 +760,36 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
             **({"task_history": result.task_history} if result.task_history is not None else {}),
         }
 
-    def _preserve_dynamic_context_reminders(
+    def _preserve_required_context(
         self,
         messages_to_summarize: list[AnyMessage],
         preserved_messages: list[AnyMessage],
         *,
         latest_user_id: str | None = None,
     ) -> tuple[list[AnyMessage], list[AnyMessage]]:
-        """Keep tagged dynamic-context reminders and the current user request out of compression.
+        """Keep system messages, tagged dynamic-context reminders and the current user request out of compression.
 
-        Only tagged reminders (date ``SystemMessage`` + optional ``__memory`` peer,
-        both carrying ``dynamic_context_reminder=True``) and the latest real user
-        message are rescued. The untagged ``__user`` peer is deliberately NOT
+        State-level SystemMessages are framework-owned instructions and must
+        survive compaction, including legacy untagged reminders and extension
+        instructions. Transient instructions should be injected into requests,
+        not state. Tagged reminders (including their ``__memory`` peers) and the
+        latest real user message are also rescued. A subagent keeps its whole system
+        prompt as the leading ``SystemMessage`` in state (``create_agent`` is built
+        with ``system_prompt=None``), so compressing it would leave every later
+        call without its instructions. The untagged ``__user`` peer is deliberately NOT
         rescued by ID-swap prefix: it is a stale historical request that must be
         allowed to compress — the source of cross-turn prompt contamination. The
         *current* request is instead identified by ``latest_user_id``, so a
         first-turn long analysis keeps its ``__user`` request while its early
         AI/Tool turns still compress.
+
+        Rescuing the whole partition is legitimate: ``_prepare_compaction``
+        skips compaction when there is no history left to summarize.
         """
         rescued: list[AnyMessage] = []
         remaining: list[AnyMessage] = []
         for msg in messages_to_summarize:
-            if is_dynamic_context_reminder(msg) or (latest_user_id is not None and msg.id == latest_user_id):
+            if isinstance(msg, SystemMessage) or is_dynamic_context_reminder(msg) or (latest_user_id is not None and msg.id == latest_user_id):
                 rescued.append(msg)
             else:
                 remaining.append(msg)

backend/tests/test_dynamic_context_middleware.py

@@ -720,7 +720,7 @@ def test_memory_message_carries_reminder_key_for_title_eligibility():
 
     Without it, title_middleware._is_user_message_for_title counts the memory
     block as a second user message and skips title generation entirely.
-    Similarly, summarization_middleware._preserve_dynamic_context_reminders
+    Similarly, summarization_middleware._preserve_required_context
     would not rescue the memory block from summary compression.
     """
     from deerflow.agents.middlewares.dynamic_context_middleware import is_dynamic_context_reminder

backend/tests/test_summarization_middleware.py

@@ -109,6 +109,54 @@ def test_before_summarization_hook_receives_messages_before_compression() -> Non
     assert [message.content for message in result["messages"][1:]] == ["user-2", "assistant-2"]
 
 
+def test_compaction_preserves_all_state_system_messages_in_order() -> None:
+    """State-level instructions, including legacy untagged reminders, stay authoritative."""
+    captured: list[SummarizationEvent] = []
+    middleware = _middleware(before_summarization=[captured.append])
+    prompt = SystemMessage(content="subagent role and report contract", id="prompt")
+    legacy_reminder = SystemMessage(content="<current_date>2026-05-08</current_date>", id="legacy-date")
+    extension_instructions = SystemMessage(content="extension instructions", id="extension")
+    current_request = HumanMessage(content="current request", id="current")
+    tail = [AIMessage(content="recent analysis", id="recent"), AIMessage(content="recent result", id="result")]
+    state = {"messages": [prompt, HumanMessage(content="old request", id="old-user"), AIMessage(content="old answer", id="old-ai"), legacy_reminder, extension_instructions, current_request, *tail]}
+
+    result = middleware.compact_state(state, _runtime())
+
+    assert result is not None
+    assert [message.id for message in result.messages_to_summarize] == ["old-user", "old-ai"]
+    assert [message.id for message in result.preserved_messages] == ["prompt", "legacy-date", "extension", "current", "recent", "result"]
+    assert captured[0].messages_to_summarize == result.messages_to_summarize
+    summary_input = middleware.model.invoke.call_args.args[0]
+    assert "subagent role and report contract" not in str(summary_input)
+    assert "extension instructions" not in str(summary_input)
+    assert "2026-05-08" not in str(summary_input)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("asynchronous", [False, True])
+async def test_compaction_skips_a_fully_rescued_partition(asynchronous: bool) -> None:
+    captured: list[SummarizationEvent] = []
+    middleware = _middleware(before_summarization=[captured.append])
+    messages = [
+        SystemMessage(content="subagent instructions", id="prompt"),
+        HumanMessage(content="current request", id="current"),
+        AIMessage(content="searching", tool_calls=[{"name": "search", "id": "call", "args": {}}], id="assistant"),
+        ToolMessage(content="search result", tool_call_id="call", id="tool"),
+    ]
+    state = {"messages": messages, "summary_text": "previous summary"}
+
+    # The trigger is met, but rescuing the prompt and request leaves no history
+    # to compress. Repeated checks must not invoke the summary model or hooks.
+    for _ in range(2):
+        result = await middleware.acompact_state(state, _runtime()) if asynchronous else middleware.compact_state(state, _runtime())
+        assert result is None
+    middleware.model.invoke.assert_not_called()
+    middleware.model.ainvoke.assert_not_called()
+    assert captured == []
+    assert state["messages"] == messages
+    assert state["summary_text"] == "previous summary"
+
+
 def test_summarization_middleware_emits_frontend_update_key_in_agent_stream() -> None:
     middleware = DeerFlowSummarizationMiddleware(
         model=_StaticChatModel(text="compressed summary"),

backend/tests/test_tool_error_handling_middleware.py

@@ -844,9 +844,10 @@ def test_subagent_compaction_injects_summary_before_assistant_tool_tail(monkeypa
     """A three-tool turn with ``keep=4`` must remain provider-valid.
 
     This reproduces the production failure shape: compaction preserves an
-    assistant tool-call plus three tool results while removing the original
-    system/user messages. The subagent chain must inject the generated summary
-    as durable human context before that tail reaches the model.
+    assistant tool-call plus three tool results and compresses the turn before
+    them. The subagent chain must inject the generated summary as durable human
+    context before that tail reaches the model, and keep the subagent's own
+    system prompt, which lives in state rather than in ``create_agent``.
     """
     from langchain.agents import create_agent
     from langchain_core.language_models import BaseChatModel
@@ -883,6 +884,7 @@ def test_subagent_compaction_injects_summary_before_assistant_tool_tail(monkeypa
                 # outgoing request is provider-valid: a single leading SystemMessage.
                 system_indices = [i for i, message in enumerate(messages) if isinstance(message, SystemMessage)]
                 assert system_indices == [0], f"request must have exactly one leading SystemMessage, got {system_indices}"
+                assert "subagent instructions" in messages[0].content, "the subagent system prompt must survive compaction"
             return ChatResult(generations=[ChatGeneration(message=AIMessage(content=self.text))])
 
     summary_model = _StaticModel(text="COMPRESSED_SUBAGENT_HISTORY")
@@ -918,6 +920,8 @@ def test_subagent_compaction_injects_summary_before_assistant_tool_tail(monkeypa
     seed = [
         SystemMessage(content="subagent instructions", id="system"),
         HumanMessage(content="research three regions", id="human"),
+        AIMessage(content="planning", tool_calls=[{"name": "web_search", "args": {"query": "overview"}, "id": "call_plan", "type": "tool_call"}], id="plan"),
+        ToolMessage(content="overview result", tool_call_id="call_plan", id="tool_plan"),
         AIMessage(content="searching", tool_calls=tool_calls, id="assistant"),
         *[ToolMessage(content=f"result {i}", tool_call_id=f"call_{i}", id=f"tool_{i}") for i in range(3)],
     ]
@@ -925,6 +929,7 @@ def test_subagent_compaction_injects_summary_before_assistant_tool_tail(monkeypa
     result = agent.invoke({"messages": seed})
 
     assert result["summary_text"] == "COMPRESSED_SUBAGENT_HISTORY"
+    assert result["messages"][0].id == "system"
     assert result["messages"][-1].content == "final answer"