mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-25 22:16:19 +00:00
906c3d4554
* feat(mcp): re-scope to MCP task claim lifecycle only Keep PR #4966 a small, closed MCP lease/cancellation state-machine change and move RunJournal and Run lifecycle work into dedicated follow-ups. This branch contains only the MCP task claim lifecycle: - mcp task release/snapshot fencing by owner + per-claim lease token - phase-level single-flight poll/cancel/notification owners with retained handoff - routine cancellation no longer persisted as a task failure diagnostic - bounded ordinary release ownership retention past the drain deadline - 0018_mcp_task_lease_tokens migration + migration/bootstrap head assertions - wait_for_task_until helper (MCP uses it); worker-specific capture helper moved to the run-finalization follow-up RunJournal (journal.py + test_run_journal.py) and run lifecycle (manager/worker/store/run sql + run tests) are preserved on backup/cancellation-safety-full and will be raised as separate follow-ups. * fix(mcp): unblock claims after ambiguous handoff resolves A phase-level single-flight owner only guards an ambiguous claim outcome. Once the claim resolves, the phase owner is released immediately; the handoff may continue releasing returned rows as bounded, service-owned background work (transferred to _compensation_tasks on timeout). Per-claim token fencing rejects a late release against a newer claim generation, so a stuck release no longer locks the whole phase until process restart. - README: drop the stale progress-snapshot sentence from the bounded ordinary release description. - service: pop the identity-checked phase owner as soon as the claim outcome is known, then release returned rows with the bounded path; carry the release in _compensation_tasks if it exceeds the drain deadline. - mcp/AGENTS.md: document that only an unresolved claim outcome (not the handoff) blocks later phase scans, and that returned-row releases may continue in the background once the owner is released. - tests: pin that the phase owner is released before a stuck release finishes while the release stays service strong-owned. * refactor(mcp): remove unused single-record claim wrappers _poll_one, _cancel_one, and _notify_one are unreachable in production: the worker always processes claimed records through _run_claimed_batch, so these wrappers preserved a second, dead single-record lifecycle (state is None) whose only observable behavior was a wrapper-specific cancellation release. Remove the three wrappers and migrate the regressions that guarded their cancel/release invariants to exercise the production _run_claimed_batch path (operation=_*_one_claimed, release=_release_*_after_cancellation). The single wrapper-only "state is None" contract (test_poll_release_hang_without_batch) is deleted; all 11 remaining invariants (CancelledError preservation, repeated cancellation, poll-only token-fenced lease release, notification claimed vs dispatched phase release, hung compensation -> service ownership, and background compensation exactly-once observation) are now covered through the real batch lifecycle. * fix(mcp): fence claim-owned mutations against stale generations The per-claim token check in the ORM release/apply paths was only in the SELECT; the final write went out by primary key. On SQLite (where with_for_update() is a no-op) a mutation from an older claim generation could therefore clear a claim that a newer generation had reclaimed after lease expiry — the exact distributed lease-fencing failure the per-claim token was meant to prevent. Make every claim-owned mutation a single atomic conditional UPDATE with the owner and per-claim token in the WHERE clause (rowcount 0 => stale, return False, no mutation): - release_claim: atomic fence; record the poll-failure event after the fence wins (same transaction, holding the write lock). - apply_snapshot / apply_cancel_snapshot: atomic fence; record the event after. - finish_notification_run: atomic fence; use a CASE on event_version >> dispatch_version to keep a newer event pending for redelivery instead of swallowing it as delivered. Add one regression per path: a stale generation's release/apply/finish after a same-worker reclaim is rejected and never clears the newer claim. * test(mcp): pin the migration chain head to the lease-token revision 0026_mcp_task_lease_tokens becomes the alembic head, so the chain-head pin in the 0025 repair test had to move on. Follow the 0023 precedent there (single head plus expected predecessor) instead of pinning a literal head, and give the new revision its own migration test, which owns the pin and covers the nullable claim-token columns on upgrade and their removal on downgrade. * refactor(mcp): close cancellation cleanup leftovers * fix(mcp): retain cancelled release diagnostics * test(mcp): remove obsolete settled compensation case * test(mcp): cover interleaved lease reclaim races --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
92 lines
2.6 KiB
Python
92 lines
2.6 KiB
Python
import asyncio
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
import deerflow.runtime.cancellation as cancellation
|
|
from deerflow.runtime.cancellation import wait_for_task_until
|
|
|
|
|
|
@pytest.mark.anyio
|
|
async def test_wait_for_task_until_reports_completion():
|
|
child = asyncio.create_task(asyncio.sleep(0, result="done"))
|
|
|
|
completed = await wait_for_task_until(child, deadline=asyncio.get_running_loop().time() + 1)
|
|
|
|
assert completed is True
|
|
assert child.result() == "done"
|
|
|
|
|
|
@pytest.mark.anyio
|
|
async def test_wait_for_task_until_times_out_without_cancelling_child():
|
|
event = asyncio.Event()
|
|
child = asyncio.create_task(event.wait())
|
|
|
|
completed = await wait_for_task_until(child, deadline=asyncio.get_running_loop().time() + 0.01)
|
|
|
|
assert completed is False
|
|
assert child.done() is False
|
|
event.set()
|
|
await child
|
|
|
|
|
|
@pytest.mark.anyio
|
|
async def test_wait_for_task_until_zero_budget_returns_immediately():
|
|
event = asyncio.Event()
|
|
child = asyncio.create_task(event.wait())
|
|
|
|
completed = await wait_for_task_until(child, deadline=asyncio.get_running_loop().time())
|
|
|
|
assert completed is False
|
|
assert child.done() is False
|
|
child.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await child
|
|
|
|
|
|
@pytest.mark.anyio
|
|
async def test_wait_for_task_until_repeated_cancellation_keeps_original_deadline(monkeypatch):
|
|
clock = iter((0.0, 0.01, 0.02, 0.05))
|
|
clock_loop = SimpleNamespace(time=lambda: next(clock, 0.05))
|
|
|
|
wait_timeouts = []
|
|
entered_first_wait = asyncio.Event()
|
|
entered_second_wait = asyncio.Event()
|
|
|
|
async def fake_wait(tasks, *, timeout):
|
|
del tasks
|
|
wait_timeouts.append(timeout)
|
|
if len(wait_timeouts) == 1:
|
|
entered_first_wait.set()
|
|
await asyncio.Future()
|
|
if len(wait_timeouts) == 2:
|
|
entered_second_wait.set()
|
|
await asyncio.Future()
|
|
return set(), set()
|
|
|
|
monkeypatch.setattr(
|
|
cancellation,
|
|
"asyncio",
|
|
SimpleNamespace(
|
|
CancelledError=asyncio.CancelledError,
|
|
get_running_loop=lambda: clock_loop,
|
|
wait=fake_wait,
|
|
),
|
|
)
|
|
event = asyncio.Event()
|
|
child = asyncio.create_task(event.wait())
|
|
waiter = asyncio.create_task(wait_for_task_until(child, deadline=0.05))
|
|
|
|
await entered_first_wait.wait()
|
|
waiter.cancel()
|
|
await entered_second_wait.wait()
|
|
waiter.cancel()
|
|
assert waiter.cancelling() == 2
|
|
completed = await waiter
|
|
|
|
assert completed is False
|
|
assert wait_timeouts == pytest.approx([0.05, 0.04, 0.03])
|
|
assert child.done() is False
|
|
event.set()
|
|
await child
|