fix(compaction): reject manual compaction of threads with pending execution (#6929)

* fix(compaction): reject manual compaction of threads with pending execution

A thread paused at a static breakpoint (interrupt_before=["tools"]) keeps its
pending node send in the checkpoint's __pregel_tasks channel. The compaction
write goes through the state-mutation graph, which schedules no tasks, so
LangGraph drops the pending send ('Ignoring unknown node name tools in pending
sends') and a resumed run finishes success without ever executing the tool.

Fail closed: detect a non-empty __pregel_tasks in the raw checkpoint tuple
inside compact_thread_context and raise ContextCompactionPendingWork, mapped
to HTTP 409 by the compact route. The reservation (reserve_checkpoint_write)
already serializes this check against runs and other checkpoint writers, and
the write still forks from the exact head that was read.

Adds dual-mode (full/delta) regression tests: the rejected compaction leaves
checkpoint id, next and pending task ids untouched, and resuming afterwards
executes the pending tool exactly once; idle threads still compact.

* fix(compaction): reject unavailable checkpoints before summarizing

* docs: keep compaction guidance within merged size budget
This commit is contained in:
CSXizhang 2026-10-16 08:59:44 +08:00 committed by GitHub
parent 0f2c2f788a
commit 3fd03ec2eb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 277 additions and 2 deletions

View File

@ -2091,6 +2091,8 @@ If the same rendered summary prompt and model candidates recur, DeerFlow reuses
Use `/compact` in the Web UI composer to summarize older context for the current thread. DeerFlow keeps the full chat visible, but future model calls use the compacted summary plus recent messages. The command is ignored when there is not enough history to compact, and it is blocked while the thread has a run in flight, including when that run is owned by another Gateway worker. If a multi-worker reservation loses its lease, DeerFlow cancels the checkpoint writer before the replacing run proceeds and returns a retryable conflict after cleanup. Thread-title edits are serialized through the same state-write boundary and show a conflict without closing the rename dialog when a run is active.
Manual compaction also returns HTTP 409 when execution is pending or the checkpoint is unavailable, so it cannot confirm the thread is idle.
The chat header also shows a context-window gauge when the selected model has a positive `context_window` configured. It estimates the latest materialized checkpoint's message tokens and keeps the previous same-thread percentage visible while data refetches, independently of the cumulative token-usage setting.
When `token_budget.enabled` and `token_usage.enabled` are both enabled, each lead-agent run budget includes completed subagent usage, including the final batch. Hidden goal continuations share the budget; a new user run starts fresh.

View File

@ -128,7 +128,7 @@ owner-scoped assistant version selection remains enabled.
| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_probed` / `sandbox_runtime_ready` (whether `lark-cli` will be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/config/credentials` - atomically switch the caller's per-user Lark app after validating the new `app_id`/`app_secret` through the official CLI's live tenant-token probe, revoke/remove the previous OAuth tokens, and restore the prior credential tree if the switch fails; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants. Config and auth flows carry a server-issued, per-user generation persisted under the credential lock; a rejected direct switch leaves the current generation unchanged, stale completions return 409, and browser re-registration uses the same token-clearing/revocation transaction as direct credential switches. |
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload/convert PDF/PPT/Excel/Word; upload/shelf names share `normalize_filename` before staging; non-mounted sync lease ends when its last holder exits; `GET /list`; `DELETE /{filename}` - regular files, symlinks 404, keeps converted `.md` (#5672) |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed: only a branch from the **latest** turn best-effort copies the workspace (`workspace_clone_mode="current_thread_best_effort"`); older turns skip it (`"skipped_historical_turn"`), so no later-timeline file leaks in. `sandbox`/`thread_data` channels are not copied (parent `sandbox_id` binds paths/release); the branch lazily acquires its own sandbox. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows get one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn at every persisted human message, including an allowlisted hidden `ask_clarification` reply): `run_id` is a turn identity, and regenerating an inherited answer supersedes its whole `run_id` in `GET /messages/page`, so one shared seed id deleted all inherited history (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request `agent_name`, and block while a run is in flight; unexpected failures return a generic 500 detail |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed: only a branch from the **latest** turn best-effort copies the workspace (`workspace_clone_mode="current_thread_best_effort"`); older turns skip it (`"skipped_historical_turn"`), so no later-timeline file leaks in. `sandbox`/`thread_data` channels are not copied (parent `sandbox_id` binds paths/release); the branch lazily acquires its own sandbox. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows get one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn at every persisted human message, including an allowlisted hidden `ask_clarification` reply): `run_id` is a turn identity, and regenerating an inherited answer supersedes its whole `run_id` in `GET /messages/page`, so one shared seed id deleted all inherited history (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request `agent_name`, and block while a run is in flight; 409 unless idle; unexpected failures return a generic 500 detail |
| **Artifacts** (`/api/threads/{id}/artifacts`) | GET streams byte ranges; HTML/XML/XSL/`+xml` (also `.skill` members) and `?download=true` force attachments. PUT atomically replaces existing UTF-8 outputs on matching SHA-256; active runs conflict. Non-mounted sync uses a request lease and drains remote/local commit-or-rollback across cancellation before releasing reservations/leases. `resolve_outputs_confined_path` normalizes `..` and confines host paths against traversal/symlinks. Preserve POSIX ownership/mode where supported; Windows keeps temp-file permissions. Digest cache: path/dev/ino/ctime_ns/mtime_ns/size; identity invalidates same-size/mtime replacements. Large regular files use stat-prefixed ETags; If-Range requires matching ETags. Windows ctime is creation time. Keep cache reuse, size limits and worker hashing. |
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |

View File

@ -58,6 +58,7 @@ from deerflow.runtime.checkpoint_state import graph_reducer_channels, graph_stat
from deerflow.runtime.context_compaction import (
ContextCompactionDisabled,
ContextCompactionFailed,
ContextCompactionPendingWork,
ThreadCompactionResult,
compact_thread_context,
)
@ -1557,6 +1558,8 @@ async def compact_thread(thread_id: ThreadId, body: ThreadCompactRequest, reques
)
except ConflictError:
raise HTTPException(status_code=409, detail="Thread has a run in flight. Compact after the run finishes.") from None
except ContextCompactionPendingWork as exc:
raise HTTPException(status_code=409, detail=str(exc)) from None
except _CHECKPOINT_MODE_ERRORS as exc:
raise _checkpoint_mode_http_error(exc, thread_id) from exc
except ContextCompactionDisabled:

View File

@ -46,6 +46,20 @@ class ContextCompactionFailed(RuntimeError):
"""Raised when a compressible thread cannot be summarized."""
class ContextCompactionPendingWork(RuntimeError):
"""Raised when pending execution cannot be ruled out before compaction.
A thread paused at a static breakpoint (e.g. ``interrupt_before=["tools"]``)
keeps the pending node send in the checkpoint's ``__pregel_tasks`` channel.
The state-mutation write below schedules no tasks and LangGraph drops
pending sends whose target node is unknown to the mutation graph, so
compacting such a thread would silently discard the pending send and a
resumed run would finish without ever executing it. Fail closed instead.
An unavailable raw checkpoint also fails closed: its pending sends cannot
be checked.
"""
@dataclass(frozen=True)
class ThreadCompactionResult:
"""Result returned after a manual context-compaction attempt."""
@ -148,6 +162,12 @@ async def compact_thread_context(
checkpoint_id = snapshot_config.get("configurable", {}).get("checkpoint_id")
if not checkpoint_id:
raise LookupError(f"Thread {thread_id} checkpoint not found")
checkpoint_tuple = await accessor.aget_tuple(read_config)
if checkpoint_tuple is None:
raise ContextCompactionPendingWork("Thread checkpoint is unavailable. Cannot verify pending execution before compacting.")
pending_sends = (checkpoint_tuple.checkpoint.get("channel_values") or {}).get("__pregel_tasks")
if pending_sends:
raise ContextCompactionPendingWork("Thread has pending execution. Resume the interrupted run before compacting.")
binding_known, checkpoint_agent_name = _checkpoint_agent_binding(getattr(snapshot, "metadata", None))
# The body hint may still select a compatible summarization model for

View File

@ -0,0 +1,225 @@
"""Manual compaction must refuse a thread paused at a static breakpoint.
A thread interrupted via ``interrupt_before=["tools"]`` still owns a pending
tool task: the Send for the tools node sits in the checkpoint's
``__pregel_tasks`` channel. Manual compaction writes the new checkpoint head
through the state-mutation graph, which schedules no tasks and does not carry
that pending Send across — a resumed run then finishes ``success`` without
ever executing the tool or producing an answer. Until compaction can preserve
the pending position, ``POST /api/threads/{id}/compact`` must fail closed
with 409 instead of silently dropping the pending execution.
The graph, checkpointer, static-interrupt mechanism, compaction route and
state accessor are all real; only the external model service is faked.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from _router_auth_helpers import make_authed_test_app
from fastapi.testclient import TestClient
from langchain.agents import create_agent
from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_core.outputs import ChatGeneration, ChatResult
from langchain_core.runnables import Runnable
from langchain_core.tools import tool
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from langgraph.types import Command
from app.gateway.auth.models import User
from app.gateway.routers import threads
from deerflow.agents.thread_state import get_thread_state_schema
from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config
from deerflow.runtime import RunManager, context_compaction
from deerflow.runtime.checkpoint_mode import inject_checkpoint_mode
from deerflow.runtime.checkpoint_state import CheckpointStateAccessor
THREAD_ID = "thread-compact-interrupted"
TOOL_CALL_ID = "call_probe_1"
FINAL_ANSWER = "probe observed"
class _ScriptedToolModel(FakeMessagesListChatModel):
"""State-aware fake lead model: plain answer on turn 1, one tool call on
turn 2, final answer once a ToolMessage exists."""
invocations: int = 0
def bind_tools(self, tools: Any, *, tool_choice: Any = None, **kwargs: Any) -> Runnable:
return self
def _generate(self, messages, stop=None, run_manager=None, **kwargs):
self.invocations += 1
if any(isinstance(m, ToolMessage) for m in messages):
return ChatResult(generations=[ChatGeneration(message=AIMessage(content=FINAL_ANSWER, id="a-final"))])
if sum(1 for m in messages if isinstance(m, HumanMessage)) >= 2:
return ChatResult(
generations=[
ChatGeneration(
message=AIMessage(
content="",
id="a-call",
tool_calls=[{"name": "probe_tool", "args": {"query": "probe-v1"}, "id": TOOL_CALL_ID, "type": "tool_call"}],
)
)
]
)
return ChatResult(generations=[ChatGeneration(message=AIMessage(content="turn one answer", id="a1"))])
def _build_agent_graph(mode: str, checkpointer: Any, tool_calls: list[str]):
@tool
def probe_tool(query: str) -> str:
"""Record the invocation and return a fixed result."""
tool_calls.append(query)
return f"probe result: {query}"
return create_agent(
model=_ScriptedToolModel(responses=[]),
tools=[probe_tool],
state_schema=get_thread_state_schema(mode),
checkpointer=checkpointer,
)
@pytest.fixture
def _summarization_app_config():
set_app_config(
AppConfig.model_validate(
{
"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"},
"summarization": {"enabled": True},
}
)
)
yield
reset_app_config()
@pytest.fixture
def _fake_summary_model(monkeypatch: pytest.MonkeyPatch):
def fake_create_chat_model(*args: Any, **kwargs: Any) -> FakeMessagesListChatModel:
return FakeMessagesListChatModel(responses=[AIMessage(content="Summary of prior turns.")])
monkeypatch.setattr("deerflow.agents.middlewares.summarization_middleware.create_chat_model", fake_create_chat_model)
def _make_compact_app(mode: str, checkpointer: Any) -> Any:
user = User(email="compact-guard@example.com", password_hash="x", system_role="user", id="3f1c2a10-7b8e-4d2a-9c1f-5e6a7b8c9d0e")
app = make_authed_test_app(user_factory=lambda: user, bind_current_user=True)
app.state.store = InMemoryStore()
app.state.checkpointer = checkpointer
app.state.checkpoint_channel_mode = mode
app.state.run_manager = RunManager()
app.include_router(threads.router)
return app
def _config(mode: str) -> dict[str, Any]:
config: dict[str, Any] = {"configurable": {"thread_id": THREAD_ID}}
inject_checkpoint_mode(config, mode)
return config
def _run_two_turns_to_static_pause(graph: Any, config: dict[str, Any]) -> None:
"""Turn 1 completes; turn 2 pauses statically before the tools node."""
graph.invoke({"messages": [HumanMessage(content="hello", id="h1")]}, config)
graph.interrupt_before_nodes = ["tools"]
try:
graph.invoke({"messages": [HumanMessage(content="please probe", id="h2")]}, config)
finally:
graph.interrupt_before_nodes = None
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_compact_rejects_thread_paused_before_tools(mode: str, _summarization_app_config, _fake_summary_model):
"""A statically interrupted thread owns a pending tool task; compaction must
fail closed (409) and leave the pending execution intact."""
checkpointer = InMemorySaver()
tool_calls: list[str] = []
graph = _build_agent_graph(mode, checkpointer, tool_calls)
config = _config(mode)
_run_two_turns_to_static_pause(graph, config)
paused = graph.get_state(config)
assert list(paused.next) == ["tools"], f"expected pending tools task, got {paused.next!r}"
assert tool_calls == []
paused_checkpoint_id = paused.config["configurable"]["checkpoint_id"]
paused_task_ids = [task.id for task in paused.tasks]
assert paused_task_ids, "expected the paused tools task in the snapshot"
app = _make_compact_app(mode, checkpointer)
with TestClient(app) as client:
response = client.post(f"/api/threads/{THREAD_ID}/compact", json={"force": True, "keep": {"type": "messages", "value": 1}})
assert response.status_code == 409, response.text
assert response.json()["detail"] == "Thread has pending execution. Resume the interrupted run before compacting."
# The rejected compaction wrote nothing: head checkpoint, pending position
# and pending task identity are exactly as before the attempt.
after_reject = graph.get_state(config)
assert after_reject.config["configurable"]["checkpoint_id"] == paused_checkpoint_id
assert list(after_reject.next) == ["tools"]
assert [task.id for task in after_reject.tasks] == paused_task_ids
# The thread is still resumable: the pending tool executes exactly once and
# the model then produces the final answer.
graph.invoke(Command(resume="go"), config)
assert tool_calls == ["probe-v1"]
final_messages = graph.get_state(config).values["messages"]
assert any(isinstance(m, ToolMessage) and m.tool_call_id == TOOL_CALL_ID for m in final_messages)
assert any(isinstance(m, AIMessage) and m.content == FINAL_ANSWER for m in final_messages)
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_compact_rejects_unavailable_checkpoint(mode: str, monkeypatch, _summarization_app_config, _fake_summary_model):
"""A valid snapshot alone cannot prove the raw checkpoint has no pending work."""
checkpointer = InMemorySaver()
graph = _build_agent_graph(mode, checkpointer, [])
config = _config(mode)
graph.invoke({"messages": [HumanMessage(content="hello", id="h1")]}, config)
graph.invoke({"messages": [HumanMessage(content="again", id="h2")]}, config)
before = graph.get_state(config)
checkpoint_ids = [item.config["configurable"]["checkpoint_id"] for item in checkpointer.list(config)]
missing_tuple = AsyncMock(return_value=None)
monkeypatch.setattr(CheckpointStateAccessor, "aget_tuple", missing_tuple)
create_middleware = MagicMock(wraps=context_compaction._create_compaction_middleware)
monkeypatch.setattr(context_compaction, "_create_compaction_middleware", create_middleware)
write_checkpoint = AsyncMock(wraps=checkpointer.aput)
monkeypatch.setattr(checkpointer, "aput", write_checkpoint)
app = _make_compact_app(mode, checkpointer)
with TestClient(app) as client:
response = client.post(f"/api/threads/{THREAD_ID}/compact", json={"force": True, "keep": {"type": "messages", "value": 1}})
assert response.status_code == 409, response.text
assert response.json()["detail"] == "Thread checkpoint is unavailable. Cannot verify pending execution before compacting."
missing_tuple.assert_awaited_once()
create_middleware.assert_not_called()
write_checkpoint.assert_not_awaited()
assert graph.get_state(config) == before
assert [item.config["configurable"]["checkpoint_id"] for item in checkpointer.list(config)] == checkpoint_ids
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_compact_still_accepts_idle_completed_thread(mode: str, _summarization_app_config, _fake_summary_model):
"""The guard must not overreach: a thread with no pending tasks compacts."""
checkpointer = InMemorySaver()
tool_calls: list[str] = []
graph = _build_agent_graph(mode, checkpointer, tool_calls)
config = _config(mode)
graph.invoke({"messages": [HumanMessage(content="hello", id="h1")]}, config)
graph.invoke({"messages": [HumanMessage(content="again", id="h2")]}, config)
idle = graph.get_state(config)
assert list(idle.next) == []
app = _make_compact_app(mode, checkpointer)
with TestClient(app) as client:
response = client.post(f"/api/threads/{THREAD_ID}/compact", json={"force": True, "keep": {"type": "messages", "value": 1}})
assert response.status_code == 200, response.text
assert response.json()["compacted"] is True

View File

@ -2,6 +2,7 @@ from __future__ import annotations
from types import SimpleNamespace
from typing import Annotated, NotRequired, TypedDict
from unittest.mock import AsyncMock, MagicMock
import pytest
from langchain_core.messages import AIMessage, HumanMessage
@ -13,7 +14,7 @@ from app.gateway import services as gateway_services
from deerflow.agents.middlewares.summarization_middleware import SummaryGenerationError
from deerflow.runtime import context_compaction
from deerflow.runtime.checkpoint_state import CheckpointStateAccessor
from deerflow.runtime.context_compaction import ContextCompactionFailed, compact_thread_context
from deerflow.runtime.context_compaction import ContextCompactionFailed, ContextCompactionPendingWork, compact_thread_context
from deerflow.runtime.context_keys import CHECKPOINT_AGENT_NAME_METADATA_KEY, DEFAULT_AGENT_NAME_METADATA_VALUE
@ -35,6 +36,11 @@ class _FakeAccessor:
async def aget(self, _config):
return self.snapshot
async def aget_tuple(self, _config):
# Mirror CheckpointStateAccessor.aget_tuple: raw checkpoint tuple with
# channel_values. The stub thread is idle, so no __pregel_tasks entry.
return SimpleNamespace(checkpoint={"channel_values": dict(self.snapshot.values)})
async def aupdate(self, config, values, *, as_node=None):
self.update_args = (config, values, as_node)
return {
@ -124,6 +130,25 @@ async def test_compact_thread_context_reads_materialized_state_and_overwrites_me
]
@pytest.mark.asyncio
async def test_compact_thread_context_rejects_unavailable_checkpoint_before_summarizing(monkeypatch):
accessor = _FakeAccessor({"messages": [HumanMessage(content="question"), AIMessage(content="answer")]})
accessor.aget_tuple = AsyncMock(return_value=None)
accessor.aupdate = AsyncMock(wraps=accessor.aupdate)
middleware = _FakeCompactionMiddleware()
create_middleware = MagicMock(return_value=middleware)
monkeypatch.setattr(context_compaction, "_create_compaction_middleware", create_middleware)
with pytest.raises(ContextCompactionPendingWork, match="checkpoint is unavailable.*Cannot verify pending execution"):
await compact_thread_context(accessor, "thread-1", app_config=SimpleNamespace())
accessor.aget_tuple.assert_awaited_once()
create_middleware.assert_not_called()
assert middleware.prepare_calls == 0
assert middleware.runtime_contexts == []
accessor.aupdate.assert_not_awaited()
@pytest.mark.asyncio
async def test_compact_thread_context_real_mutation_graph_finishes_without_scheduling(monkeypatch):
messages = [