github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-14 16:08:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/backend/tests/test_thread_run_idempotency.py
Shxiao 572744975d
fix(tools): run tool assembly off-loop at async entry points (#5224) 
* fix(tools): run tool assembly off-loop at async entry points

get_available_tools() may block on MCP cache initialization while it is
called on async agent-assembly paths (task_tool, durable batch execution),
stalling the calling event loop for the full discovery duration.

Dispatch the (unchanged, synchronous) assembly call to a worker thread via
asyncio.to_thread at the two async entry points so the loop keeps processing
requests, SSE frames, cancellations, and timers.

Fixes #5172

* fix(tools): offload lead-agent assembly off-loop and pin with blocking-io anchors

Review follow-up for #5224:

- run_agent now dispatches agent_factory(...) through asyncio.to_thread, so
  lead-agent assembly (including both get_available_tools call sites in
  _assemble_lead_agent) runs off the event loop — the Gateway headline
  scenario from issue #5172.
- _ensure_sync_invocable_tool takes a double-checked threading.Lock, making
  the in-place tool.func wrap on the shared tool singletons explicitly
  single-shot now that assembly can run concurrently on worker threads.
- Add backend/tests/blocking_io/test_tool_assembly_offloop.py: blocking-probe
  anchors for task_tool and SubagentBatchService._execute_item under the
  strict Blockbuster gate, plus a meta-check proving the gate trips on the
  exact syscall class (ExtensionsConfig.from_file on the loop). Verified the
  anchor goes red when the offload is flattened back to a plain call.

* fix(gateway): build checkpoint state accessor off-loop; anchor run_agent offload

Review follow-up for #5224:

- Add abuild_checkpoint_state_accessor (asyncio.to_thread around the
  unchanged sync builder) and switch every async call site to it: the
  stateless_wait route, thread_runs, both threads call sites, and the
  build_thread_checkpoint_state_accessor boundary. The agent-factory
  assembly re-enters get_available_tools() and may block on MCP cache
  initialization; repeat calls hit _state_accessor_graph_cache and only
  pay the thread hop.
- Add a third blocking-io anchor driving the real run_agent with minimal
  RunManager/bridge stubs; the factory performs a real production blocking
  read (ExtensionsConfig.from_file()) and the test asserts assembly never
  runs on the main thread. Verified the anchor goes red when the run_agent
  offload is flattened back to a plain call.
- Adapt the test_threads_router checkpoint-builder patch sites to the new
  async name.

* refactor(tools): carry assembly offloads on a dedicated bounded pool

Review follow-up for #5224:

- Add utils/assembly_io.py: a dedicated ThreadPoolExecutor (default 8
  workers, DEER_FLOW_ASSEMBLY_WORKERS-overridable, mirroring
  utils/file_io.py and tools/sync.py) with run_assembly(), which copies
  contextvars explicitly. A hung stdio MCP server parks its worker for
  the full MCP timeout; carrying assembly hops on the loop's default
  executor would let a few parked assemblies queue every other
  to_thread/run_in_executor(None, ...) caller behind them.
- Switch all four offloads (run_agent, task_tool, batch _execute_item,
  abuild_checkpoint_state_accessor) to run_assembly().
- State the cold-path behavior in the accessor docstring: the graph
  cache validates factory identity, so non-identity-stable factories may
  duplicate lead-agent assembly across concurrent readers (MCP discovery
  stays process-wide single-flight); the pool bounds the duplicates.
- Add a fourth blocking-io anchor driving build_thread_checkpoint_state_
  accessor with a per-resolution fresh factory (always a cache miss) and
  the real production blocking read; enumerate all four offloads in the
  gate's module docstring. Verified the anchor goes red when
  abuild_checkpoint_state_accessor is flattened back to a plain call.

* fix(subagents): revalidate batch item before launch; make assembly pool observable

Review follow-up for #5224:

- _execute_item() revalidates the durable state right after assembly and
  before executor.execute_async(): renew_item_lease() returns valid=False
  when cancel_batch() terminalized the item or the lease was lost while
  assembly was parked, and the launch is skipped (the canceller already
  finalized the item). Previously the launch was unconditional and the
  poll loop's cancellation checks only started after execution began.
- Regression test driving the real SQLite repository: a blocking assembly
  probe parks _execute_item, cancel_batch() lands, and the launch is
  skipped with the item staying cancelled. Verified the test goes red
  when the revalidation is removed.
- run_assembly() tracks pending assemblies and logs a throttled WARNING
  once the pending count exceeds the worker count, so assembly starvation
  (workers parked on a hung MCP server) is distinguishable from idle.
- The run_agent blocking-io anchor now binds a sentinel extension
  snapshot via ctx.extensions and asserts the factory observed it through
  get_agent_build_extensions(), pinning run_assembly()'s ContextVar
  propagation. Verified red when ctx.run is dropped.
- Document the assembly pool in backend/AGENTS.md.

* fix(utils): decrement the assembly pending count on the pool thread

The pending-assembly counter behind the starvation warning decremented
from the asyncio future's done callback, which never fires once the
submitting loop is closed while its worker is still running: the count
ratcheted up permanently and eventually fired the starvation warning
with no starvation behind it (reproduced at 97dc9bec by review).

Decrement instead from the dispatched work item: run_assembly() wraps
func so a finally drops the count under the pending lock on the pool
thread, and the done callback is gone.

Pin the counter with tests/test_assembly_io.py: a healthy call returns
the count to zero, and an abandoned loop (stopped while the worker is
parked) does not wedge it — the abandoned case goes red against the old
done-callback decrement.

* docs(utils): fix the pending-counter comment after the decrement move

The comment still described the removed done-callback decrement,
contradicting _work()'s own comment; state the actual mechanism
(increment on the loop before dispatch, decrement from the dispatched
work item's finally on a pool thread).

* test(gateway): retarget checkpoint-accessor stubs to the services seam

thread_runs and runs now call abuild_checkpoint_state_accessor, so the
upstream wait-reader, regenerate-prepare, and idempotency tests must stub
the sync builder where abuild resolves it (app.gateway.services); stubbing
the removed router re-exports fails with AttributeError at setup. The
async seam semantics are unchanged: run_assembly invokes the stubbed
sync builder off-loop and propagates its return values and exceptions.

Move the agent/tool assembly off-load note from backend/AGENTS.md to
deerflow/utils/AGENTS.md (next to assembly_io.py) so the effective
instruction chain for agents/middlewares no longer grows past the AG002
hard limit.

* fix(runtime): serialize same-key accessor assembly and release queued-cancel slots

Address the three review follow-ups on the assembly off-load:

- assembly_io: a job cancelled while still queued never runs its work
  item, so the dispatched finally never fired and _pending_assemblies
  stayed elevated until a false starvation warning. Exactly-once cleanup
  now rides the concurrent future's cancelled() state — cancel() only
  succeeds before the executor starts the item, so cancelled() is true
  precisely when the finally will never run — plus a submit-failure
  release; the one-worker queued-cancellation case is pinned red/green.
- services: overlapping cold readers sharing one cache key could both
  run full agent assembly. _state_accessor_graph now serializes per key
  through a thread-side KeyedLockTable (pool threads, no running loop)
  and re-validates factory/app-config identity under the lock, so the
  factory runs exactly once while identity changes still rebuild. Cache
  dict access is lock-guarded now that construction runs off-loop.
- guidance inventory: register deerflow/utils/AGENTS.md in
  EXPECTED_GUIDANCE_PATHS so test_repository_has_the_approved_scoped_
  guidance_shape matches the relocated assembly note (CI shard 4).

* test(keyed-lock): pin KeyedLockTable reclamation and waiter bypass directly

Thread-side counterparts of the async table's own tests: overlapping
hold() calls serialize (a late arrival joins the live entry instead of
creating a second lock that bypasses a queued waiter), the last check-in
pops the entry, and many unique keys leave the registry empty. Both
regressions verified red — popping unconditionally trips the late-arrival
test, never reclaiming trips the many-keys test.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-12 07:41:24 +08:00

794 lines
28 KiB
Python
Raw Blame History

"""HTTP contract tests for idempotent thread-run creation (issue #5257)."""
from __future__ import annotations
import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
from uuid import uuid4
import pytest
from _router_auth_helpers import call_unwrapped, make_authed_test_app
from fastapi import HTTPException
from fastapi.testclient import TestClient
from app.gateway import services
from app.gateway.auth.models import User
from app.gateway.routers import thread_runs
from app.gateway.run_models import RunCreateRequest
from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config
from deerflow.runtime import DisconnectMode, RunManager, RunRecord, RunStatus
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.runs.store.memory import MemoryRunStore
def _user(email: str) -> User:
return User(email=email, password_hash="x", system_role="user", id=uuid4())
def _run(run_id: str, thread_id: str) -> RunRecord:
return RunRecord(
run_id=run_id,
thread_id=thread_id,
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
error=run_id,
)
def _make_client(monkeypatch, user: User, admissions: dict[str, RunRecord]) -> TestClient:
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, request, require_existing_thread
if idempotency_key is not None and idempotency_key in admissions:
record = admissions[idempotency_key]
record.idempotency_reused = True
return record
record = _run(f"run-{len(admissions) + 1}", thread_id)
admissions[idempotency_key or f"unkeyed-{record.run_id}"] = record
return record
monkeypatch.setattr(thread_runs, "start_run", fake_start_run)
app = make_authed_test_app(user_factory=lambda: user)
app.include_router(thread_runs.router)
app.state.stream_bridge = MagicMock(stream_exists=AsyncMock(return_value=False))
app.state.run_manager = MagicMock()
return TestClient(app)
def test_same_idempotency_key_reuses_thread_run(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
url = "/api/threads/thread-1/runs"
headers = {"Idempotency-Key": "send-message-1"}
first = client.post(url, json={"input": {"messages": []}}, headers=headers)
retry = client.post(url, json={"input": {"messages": []}}, headers=headers)
assert first.status_code == 200, first.text
assert retry.status_code == 200, retry.text
assert retry.json()["run_id"] == first.json()["run_id"]
def test_same_idempotency_key_reuses_stream_run(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
url = "/api/threads/thread-1/runs/stream"
headers = {"Idempotency-Key": "send-message-1"}
first = client.post(url, json={"input": {"messages": []}}, headers=headers)
retry = client.post(url, json={"input": {"messages": []}}, headers=headers)
assert first.status_code == 200, first.text
assert retry.status_code == 200, retry.text
assert retry.headers["Content-Location"] == first.headers["Content-Location"]
assert "event: end" in first.text
assert "event: gap" not in first.text
assert "event: gap" in retry.text
assert "stream_replay_gap" in retry.text
assert "reload_durable_state" in retry.text
assert "event: end" not in retry.text
def test_same_idempotency_key_reuses_wait_run(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
url = "/api/threads/thread-1/runs/wait"
headers = {"Idempotency-Key": "send-message-1"}
first = client.post(url, json={"input": {"messages": []}}, headers=headers)
retry = client.post(url, json={"input": {"messages": []}}, headers=headers)
assert first.status_code == 200, first.text
assert retry.status_code == 200, retry.text
assert retry.json()["error"] == first.json()["error"]
def test_idempotency_key_is_scoped_to_thread(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
headers = {"Idempotency-Key": "send-message-1"}
first = client.post("/api/threads/thread-1/runs", json={}, headers=headers)
second = client.post("/api/threads/thread-2/runs", json={}, headers=headers)
assert first.status_code == 200, first.text
assert second.status_code == 200, second.text
assert second.json()["run_id"] != first.json()["run_id"]
def test_idempotency_key_is_scoped_to_authenticated_user(monkeypatch):
admissions: dict[str, RunRecord] = {}
alice = _make_client(monkeypatch, _user("alice@example.com"), admissions)
bob = _make_client(monkeypatch, _user("bob@example.com"), admissions)
url = "/api/threads/thread-1/runs"
headers = {"Idempotency-Key": "send-message-1"}
first = alice.post(url, json={}, headers=headers)
second = bob.post(url, json={}, headers=headers)
assert first.status_code == 200, first.text
assert second.status_code == 200, second.text
assert second.json()["run_id"] != first.json()["run_id"]
def test_missing_idempotency_key_keeps_creating_runs(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
url = "/api/threads/thread-1/runs"
first = client.post(url, json={})
second = client.post(url, json={})
assert first.status_code == 200, first.text
assert second.status_code == 200, second.text
assert second.json()["run_id"] != first.json()["run_id"]
def test_different_idempotency_keys_create_different_runs(monkeypatch):
admissions: dict[str, RunRecord] = {}
client = _make_client(monkeypatch, _user("alice@example.com"), admissions)
url = "/api/threads/thread-1/runs"
first = client.post(url, json={}, headers={"Idempotency-Key": "send-message-1"})
second = client.post(url, json={}, headers={"Idempotency-Key": "send-message-2"})
assert first.status_code == 200, first.text
assert second.status_code == 200, second.text
assert second.json()["run_id"] != first.json()["run_id"]
def test_blank_idempotency_key_is_rejected(monkeypatch):
client = _make_client(monkeypatch, _user("alice@example.com"), {})
response = client.post(
"/api/threads/thread-1/runs",
json={},
headers={"Idempotency-Key": " "},
)
assert response.status_code == 422
def test_oversized_idempotency_key_is_rejected(monkeypatch):
client = _make_client(monkeypatch, _user("alice@example.com"), {})
response = client.post(
"/api/threads/thread-1/runs",
json={},
headers={"Idempotency-Key": "x" * 256},
)
assert response.status_code == 422
class _LocalBridge:
supports_cross_process = False
async def stream_exists(self, run_id):
del run_id
return False
class _StaleSnapshot:
config = {"configurable": {"checkpoint_id": "cp-previous"}}
values = {"messages": [{"type": "ai", "content": "PREVIOUS_TURN"}]}
def test_wait_reused_store_only_run_does_not_return_stale_checkpoint(monkeypatch):
"""A reused running record has no local task; /wait must not serialize the current checkpoint."""
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, request, idempotency_key, require_existing_thread
return RunRecord(
run_id="run-live",
thread_id=thread_id,
assistant_id=None,
status=RunStatus.running,
on_disconnect=DisconnectMode.continue_,
store_only=True,
idempotency_reused=True,
)
async def fake_aget(config):
del config
return _StaleSnapshot()
monkeypatch.setattr(thread_runs, "start_run", fake_start_run)
monkeypatch.setattr(
services,
"build_checkpoint_state_accessor",
lambda *args, **kwargs: (SimpleNamespace(aget=fake_aget), {}),
)
monkeypatch.setattr(thread_runs, "serialize_channel_values_for_api", lambda values: values)
app = make_authed_test_app(user_factory=lambda: _user("alice@example.com"))
app.include_router(thread_runs.router)
app.state.stream_bridge = _LocalBridge()
app.state.run_manager = MagicMock()
with TestClient(app) as client:
response = client.post(
"/api/threads/thread-1/runs/wait",
json={"input": {"messages": []}},
headers={"Idempotency-Key": "send-message-1"},
)
assert response.status_code == 200, response.text
assert response.json() == {"status": "running", "error": None}
assert "PREVIOUS_TURN" not in response.text
def test_wait_reused_completed_run_does_not_return_later_checkpoint(monkeypatch):
"""A locally cached completed reuse must not serialize a later thread head."""
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, request, idempotency_key, require_existing_thread
return RunRecord(
run_id="run-a",
thread_id=thread_id,
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
store_only=False,
idempotency_reused=True,
)
async def fake_aget(config):
del config
return SimpleNamespace(
config={"configurable": {"checkpoint_id": "cp-later"}},
values={"messages": [{"type": "ai", "content": "LATER_RUN_RESULT"}]},
)
monkeypatch.setattr(thread_runs, "start_run", fake_start_run)
monkeypatch.setattr(
services,
"build_checkpoint_state_accessor",
lambda *args, **kwargs: (SimpleNamespace(aget=fake_aget), {}),
)
monkeypatch.setattr(thread_runs, "serialize_channel_values_for_api", lambda values: values)
app = make_authed_test_app(user_factory=lambda: _user("alice@example.com"))
app.include_router(thread_runs.router)
app.state.stream_bridge = _LocalBridge()
app.state.run_manager = MagicMock()
with TestClient(app) as client:
response = client.post(
"/api/threads/thread-1/runs/wait",
json={"input": {"messages": []}},
headers={"Idempotency-Key": "send-message-1"},
)
assert response.status_code == 200, response.text
assert response.json() == {"status": "success", "error": None}
assert "LATER_RUN_RESULT" not in response.text
@pytest.mark.anyio
async def test_wait_original_request_keeps_checkpoint_when_retry_overlaps():
"""An overlapping retry must not suppress the original creating /wait result."""
from deerflow.runtime.stream_bridge.memory import MemoryStreamBridge
bridge = MemoryStreamBridge()
record = RunRecord(
run_id="run-a",
thread_id="thread-1",
assistant_id=None,
status=RunStatus.running,
on_disconnect=DisconnectMode.continue_,
store_only=False,
idempotency_reused=False,
)
record.task = asyncio.create_task(asyncio.Event().wait())
snapshot = SimpleNamespace(
config={"configurable": {"checkpoint_id": "cp-a"}},
values={"messages": [{"type": "ai", "content": "FIRST_RUN_RESULT"}]},
)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, thread_id, request, idempotency_key, require_existing_thread
return record
async def fake_aget(config):
del config
return snapshot
with (
patch.object(thread_runs, "start_run", fake_start_run),
patch.object(thread_runs, "get_stream_bridge", return_value=bridge),
patch.object(thread_runs, "get_run_manager", return_value=MagicMock()),
patch.object(
services,
"build_checkpoint_state_accessor",
lambda *args, **kwargs: (SimpleNamespace(aget=fake_aget), {}),
),
patch.object(thread_runs, "serialize_channel_values_for_api", lambda values: values),
):
wait_task = asyncio.create_task(
call_unwrapped(
thread_runs.wait_run,
"thread-1",
RunCreateRequest(input={"messages": []}),
request,
)
)
await asyncio.sleep(0.05)
record.idempotency_reused = True
record.status = RunStatus.success
await bridge.publish_end(record.run_id)
result = await asyncio.wait_for(wait_task, timeout=2)
record.task.cancel()
with pytest.raises(asyncio.CancelledError):
await record.task
assert result["messages"][0]["content"] == "FIRST_RUN_RESULT"
@pytest.mark.anyio
async def test_wait_peer_refreshes_status_after_owner_completes():
"""A cross-worker reuse must not keep admission-time running after END."""
from deerflow.runtime.stream_bridge.memory import MemoryStreamBridge
store = MemoryRunStore()
owner = RunManager(store=store, worker_id="worker-a")
peer = RunManager(store=store, worker_id="worker-b")
bridge = MemoryStreamBridge()
bridge.supports_cross_process = True
input_payload = {"messages": [{"role": "user", "content": "hello"}]}
first = await owner.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
kwargs={"input": input_payload, "config": None},
)
await owner.set_status(first.run_id, RunStatus.running)
reused = await peer.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
kwargs={"input": input_payload, "config": None},
)
assert reused.run_id == first.run_id
assert reused.store_only is True
assert reused.idempotency_reused is True
assert reused.status == RunStatus.running
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, thread_id, request, idempotency_key, require_existing_thread
return reused
with (
patch.object(thread_runs, "start_run", fake_start_run),
patch.object(thread_runs, "get_stream_bridge", return_value=bridge),
patch.object(thread_runs, "get_run_manager", return_value=peer),
patch.object(
services,
"build_checkpoint_state_accessor",
side_effect=AssertionError("reused wait must not read latest checkpoint"),
),
):
wait_task = asyncio.create_task(
call_unwrapped(
thread_runs.wait_run,
"thread-1",
RunCreateRequest(input=input_payload),
request,
)
)
await asyncio.sleep(0.05)
await owner.set_status(first.run_id, RunStatus.success)
await bridge.publish_end(first.run_id)
result = await asyncio.wait_for(wait_task, timeout=2)
assert result == {"status": "success", "error": None}
assert reused.status == RunStatus.success
def test_scope_http_run_idempotency_key_ignores_header_default():
"""Direct handler calls pass FastAPI's Header() object, not None."""
from fastapi.params import Header as HeaderParam
request = SimpleNamespace(state=SimpleNamespace(user=None))
assert thread_runs._scope_http_run_idempotency_key(request, "thread-1", HeaderParam(default=None)) is None
assert thread_runs._scope_http_run_idempotency_key(request, "thread-1", None) is None
def test_stream_reused_store_only_running_run_returns_409(monkeypatch):
"""A reused running record on a process-local bridge must not hang on an empty stream."""
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, request, idempotency_key, require_existing_thread
return RunRecord(
run_id="run-live",
thread_id=thread_id,
assistant_id=None,
status=RunStatus.running,
on_disconnect=DisconnectMode.continue_,
store_only=True,
idempotency_reused=True,
)
monkeypatch.setattr(thread_runs, "start_run", fake_start_run)
app = make_authed_test_app(user_factory=lambda: _user("alice@example.com"))
app.include_router(thread_runs.router)
app.state.stream_bridge = _LocalBridge()
app.state.run_manager = MagicMock()
with TestClient(app) as client:
response = client.post(
"/api/threads/thread-1/runs/stream",
json={"input": {"messages": []}},
headers={"Idempotency-Key": "send-message-1"},
)
assert response.status_code == 409, response.text
assert "not active on this worker" in response.json()["detail"]
@pytest.mark.anyio
async def test_sse_consumer_reused_terminal_missing_stream_yields_gap():
from app.gateway.services import sse_consumer
record = RunRecord(
run_id="run-done",
thread_id="thread-1",
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
store_only=True,
idempotency_reused=True,
)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
frames = [
frame
async for frame in sse_consumer(
_LocalBridge(),
record,
request,
MagicMock(),
emit_gap_on_missing_stream=True,
)
]
assert len(frames) == 1
assert frames[0].startswith("event: gap\n")
assert "stream_replay_gap" in frames[0]
assert "reload_durable_state" in frames[0]
assert "event: end" not in frames[0]
@pytest.mark.anyio
async def test_sse_consumer_observer_join_keeps_end_after_sticky_reuse_flag():
"""Observer joins must not inherit create_or_reject's sticky reuse flag."""
from app.gateway.services import sse_consumer
record = RunRecord(
run_id="run-done",
thread_id="thread-1",
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
idempotency_reused=True,
)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
frames = [frame async for frame in sse_consumer(_LocalBridge(), record, request, MagicMock(), apply_on_disconnect=False)]
assert len(frames) == 1
assert frames[0].startswith("event: end\n")
assert "event: gap" not in frames[0]
@pytest.mark.anyio
async def test_sse_consumer_default_path_keeps_end_after_sticky_reuse_flag():
"""Default sse_consumer, including stateless /api/runs/stream, must not emit gap
just because create_or_reject left idempotency_reused set, or because
apply_on_disconnect still defaults to True.
"""
from app.gateway.services import sse_consumer
record = RunRecord(
run_id="run-done",
thread_id="thread-1",
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
store_only=True,
idempotency_reused=True,
)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
frames = [frame async for frame in sse_consumer(_LocalBridge(), record, request, MagicMock())]
assert len(frames) == 1
assert frames[0].startswith("event: end\n")
assert "event: gap" not in frames[0]
@pytest.mark.anyio
async def test_sse_consumer_missing_stream_gap_requires_explicit_flag():
"""apply_on_disconnect must not select gap vs end by itself."""
from app.gateway.services import sse_consumer
record = RunRecord(
run_id="run-done",
thread_id="thread-1",
assistant_id=None,
status=RunStatus.success,
on_disconnect=DisconnectMode.continue_,
store_only=True,
)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
default_frames = [frame async for frame in sse_consumer(_LocalBridge(), record, request, MagicMock(), apply_on_disconnect=True)]
gap_frames = [
frame
async for frame in sse_consumer(
_LocalBridge(),
record,
request,
MagicMock(),
apply_on_disconnect=False,
emit_gap_on_missing_stream=True,
)
]
assert default_frames[0].startswith("event: end\n")
assert gap_frames[0].startswith("event: gap\n")
@pytest.mark.anyio
async def test_observer_join_stays_end_after_real_manager_reuse():
"""Join of a terminal missing stream stays `end` after a later key reuse.
``create_or_reject`` sets ``idempotency_reused`` on the cached record that
``RunManager.get()`` returns. Observer joins read that same object; the
missing-stream branch must still follow ``emit_gap_on_missing_stream``,
not the sticky flag or ``apply_on_disconnect``.
"""
from app.gateway.services import sse_consumer
store = MemoryRunStore()
manager = RunManager(store=store, worker_id="worker-a")
first = await manager.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
)
await manager.set_status(first.run_id, RunStatus.success)
request = SimpleNamespace(headers={}, is_disconnected=AsyncMock(return_value=False))
async def _frames(*, apply_on_disconnect: bool = True, emit_gap_on_missing_stream: bool = False):
record = await manager.get(first.run_id)
assert record is not None
return [
frame
async for frame in sse_consumer(
_LocalBridge(),
record,
request,
manager,
apply_on_disconnect=apply_on_disconnect,
emit_gap_on_missing_stream=emit_gap_on_missing_stream,
)
]
before = await _frames(apply_on_disconnect=False)
assert before[0].startswith("event: end\n")
reused = await manager.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
)
assert reused.run_id == first.run_id
assert reused.idempotency_reused is True
after = await _frames(apply_on_disconnect=False)
assert after[0].startswith("event: end\n")
assert "event: gap" not in after[0]
after_default = await _frames()
assert after_default[0].startswith("event: end\n")
assert "event: gap" not in after_default[0]
creating = await _frames(emit_gap_on_missing_stream=True)
assert creating[0].startswith("event: gap\n")
assert "event: end" not in creating[0]
def _make_start_run_request(run_manager):
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
store = InMemoryStore()
return SimpleNamespace(
headers={},
state=SimpleNamespace(auth_source=None, user=None),
app=SimpleNamespace(
state=SimpleNamespace(
stream_bridge=SimpleNamespace(),
run_manager=run_manager,
checkpointer=InMemorySaver(),
store=store,
run_event_store=MemoryRunEventStore(),
run_events_config=None,
thread_store=MemoryThreadMetaStore(store),
)
),
)
@pytest.fixture
def _stub_app_config():
set_app_config(AppConfig.model_validate({"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}}))
yield
reset_app_config()
@pytest.mark.anyio
async def test_start_run_reuses_store_backed_running_row_without_attaching_worker(_stub_app_config):
from app.gateway.services import start_run
input_payload = {"messages": [{"role": "user", "content": "hello"}]}
store = MemoryRunStore()
owner = RunManager(store=store, worker_id="worker-a")
peer = RunManager(store=store, worker_id="worker-b")
first = await owner.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
kwargs={"input": input_payload, "config": None},
)
attached = False
async def fake_run_agent(*args, **kwargs):
del args, kwargs
nonlocal attached
attached = True
with (
patch("app.gateway.services.resolve_agent_factory", return_value=object()),
patch("app.gateway.services.run_agent", side_effect=fake_run_agent),
):
record = await start_run(
RunCreateRequest(input=input_payload),
"thread-1",
_make_start_run_request(peer),
idempotency_key="http-run:same",
)
assert record.run_id == first.run_id
assert record.idempotency_reused is True
assert record.store_only is True
assert record.task is None
assert attached is False
@pytest.mark.anyio
async def test_start_run_rejects_reused_key_with_different_input(_stub_app_config):
from app.gateway.services import start_run
store = MemoryRunStore()
owner = RunManager(store=store, worker_id="worker-a")
peer = RunManager(store=store, worker_id="worker-b")
await owner.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:same",
kwargs={"input": {"messages": [{"role": "user", "content": "summarize"}]}, "config": None},
)
with (
patch("app.gateway.services.resolve_agent_factory", return_value=object()),
patch("app.gateway.services.run_agent", side_effect=AssertionError("worker must not attach")),
pytest.raises(HTTPException) as excinfo,
):
await start_run(
RunCreateRequest(input={"messages": [{"role": "user", "content": "translate"}]}),
"thread-1",
_make_start_run_request(peer),
idempotency_key="http-run:same",
)
assert excinfo.value.status_code == 409
assert "different request" in str(excinfo.value.detail)
@pytest.mark.anyio
async def test_wait_retry_after_later_run_does_not_return_later_checkpoint(monkeypatch):
"""Complete two runs, then retry the first key: /wait must not return run B."""
first_input = {"messages": [{"role": "user", "content": "one"}]}
later_input = {"messages": [{"role": "user", "content": "two"}]}
store = MemoryRunStore()
manager = RunManager(store=store, worker_id="worker-a")
first = await manager.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:first",
kwargs={"input": first_input, "config": None},
)
await manager.set_status(first.run_id, RunStatus.success)
later = await manager.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:later",
kwargs={"input": later_input, "config": None},
)
await manager.set_status(later.run_id, RunStatus.success)
reused = await manager.create_or_reject(
"thread-1",
user_id=None,
idempotency_key="http-run:first",
kwargs={"input": first_input, "config": None},
)
assert reused.run_id == first.run_id
assert reused.idempotency_reused is True
assert reused.store_only is False
assert reused.status == RunStatus.success
async def fake_start_run(body, thread_id, request, *, idempotency_key=None, require_existing_thread=False):
del body, thread_id, request, idempotency_key, require_existing_thread
return reused
async def fake_aget(config):
del config
return SimpleNamespace(
config={"configurable": {"checkpoint_id": "cp-later"}},
values={"messages": [{"type": "ai", "content": "LATER_RUN_RESULT"}]},
)
monkeypatch.setattr(thread_runs, "start_run", fake_start_run)
monkeypatch.setattr(
services,
"build_checkpoint_state_accessor",
lambda *args, **kwargs: (SimpleNamespace(aget=fake_aget), {}),
)
monkeypatch.setattr(thread_runs, "serialize_channel_values_for_api", lambda values: values)
app = make_authed_test_app(user_factory=lambda: _user("alice@example.com"))
app.include_router(thread_runs.router)
app.state.stream_bridge = _LocalBridge()
app.state.run_manager = manager
with TestClient(app) as client:
response = client.post(
"/api/threads/thread-1/runs/wait",
json={"input": first_input},
headers={"Idempotency-Key": "first"},
)
assert response.status_code == 200, response.text
assert response.json() == {"status": "success", "error": None}
assert "LATER_RUN_RESULT" not in response.text
Reference in New Issue View Git Blame Copy Permalink