From 39788402ec106d06c87af486ab8d04eb359e28ad Mon Sep 17 00:00:00 2001 From: ordinary being Date: Tue, 17 Nov 2026 10:36:27 +0800 Subject: [PATCH] fix(scheduler): fence late launch bookkeeping by claim generation (#8268) * fix(scheduler): fence late launch bookkeeping by claim generation An expired launch claim can be requeued and immediately reclaimed by the same scheduler process. Because requeue_claimed_run and fail_launching_run only matched status == launching and the lease owner, the original attempt's late busy-thread requeue or launch failure could still clear the fresh claim's lease and release its global execution-budget slot. Another queued occurrence could then claim that slot while the fresh launch was still in flight. attempt_count already increments on every claim. Capture it from claim_queued_run and pass it to both slot-releasing writes, which now match on the generation as well. The lease-owner check remains; a stale attempt becomes a no-op, while the current generation can still release its own claim. The fence is optional so legacy/custom repositories that do not expose attempt_count keep the previous behavior. Tests: a SQLite/Postgres repository test recovers an expired claim, reclaims it with the same owner, then replays the stale requeue and failure and asserts the fresh row plus its global slot survive. Service tests pin that the claim attempt is passed on both the busy-thread and pre-launch-failure paths. * fix(scheduler): log lost requeue claims after a newer generation wins The busy-thread requeue path discarded requeue_claimed_run()'s boolean, so a generation-guard rejection was indistinguishable from a normal requeue in logs while the attempt no longer owned the occurrence. Capture the result and emit the same lost-launch-claim warning used by the failure path; add a regression test that simulates a newer claim winning. * docs(scheduler): keep persistence guidance within its soft limit Compress the generation-fence note so the shared persistence AGENTS.md chain stays below the soft size limit. * fix(scheduler): omit generation kwarg for legacy claim repos Both late-bookkeeping calls passed expected_attempt_count=None even when the claim had no generation. Custom repositories that still implement the pre-fence signatures reject that keyword with TypeError. Build the kwargs conditionally so the argument is only sent when the claim actually carries a generation; add a legacy repo regression for both the busy-requeue and pre-launch-failure paths. --------- Co-authored-by: Tong-bit-art --- backend/app/scheduler/service.py | 38 +++- .../harness/deerflow/persistence/AGENTS.md | 1 + .../persistence/scheduled_task_runs/sql.py | 24 ++- .../test_scheduled_launch_recovery_fence.py | 83 ++++++++ backend/tests/test_scheduled_task_service.py | 182 ++++++++++++++++++ docs/guide/scheduled-tasks.md | 1 + 6 files changed, 317 insertions(+), 12 deletions(-) diff --git a/backend/app/scheduler/service.py b/backend/app/scheduler/service.py index 53e75b355d..440447965b 100644 --- a/backend/app/scheduler/service.py +++ b/backend/app/scheduler/service.py @@ -754,6 +754,12 @@ class ScheduledTaskService: if claimed is None: return self._queued_result(task_run_id, execution_thread_id) + # Every claim increments attempt_count. Late bookkeeping from this + # attempt must carry that generation: after lease expiry the same + # process can reclaim the occurrence, so the lease owner alone cannot + # distinguish the two attempts. + claim_attempt = int(claimed["attempt_count"]) if claimed.get("attempt_count") is not None else None + # Track whether _launch_run has produced a live run. A bookkeeping # failure after launch must retain the non-terminal slot so a later # poll cannot start the same occurrence twice. @@ -827,11 +833,24 @@ class ScheduledTaskService: } except Exception as exc: if not launch_succeeded and self._is_overlap_conflict(exc): - await self._task_run_repo.requeue_claimed_run( + requeue_kwargs = { + "lease_owner": self._lease_owner, + "error": str(exc), + } + # Omit the generation keyword entirely for legacy/custom repos: + # passing None would break implementations that predate the + # optional fence and do not accept the argument. + if claim_attempt is not None: + requeue_kwargs["expected_attempt_count"] = claim_attempt + requeued = await self._task_run_repo.requeue_claimed_run( task_run_id, - lease_owner=self._lease_owner, - error=str(exc), + **requeue_kwargs, ) + if not requeued: + logger.warning( + "Scheduled task-run %s lost its launch claim before requeue bookkeeping; leaving recovery-owned state unchanged", + task_run_id, + ) return self._queued_result(task_run_id, execution_thread_id, error=str(exc)) if launch_succeeded: @@ -901,12 +920,17 @@ class ScheduledTaskService: # _launch_run itself failed (or a step before it did): no live run # was created, so it is safe to release the active slot. + fail_kwargs = { + "task_id": task["id"], + "lease_owner": self._lease_owner, + "error": str(exc), + "now": now, + } + if claim_attempt is not None: + fail_kwargs["expected_attempt_count"] = claim_attempt finalized = await self._task_run_repo.fail_launching_run( task_run_id, - task_id=task["id"], - lease_owner=self._lease_owner, - error=str(exc), - now=now, + **fail_kwargs, ) if not finalized: logger.warning( diff --git a/backend/packages/harness/deerflow/persistence/AGENTS.md b/backend/packages/harness/deerflow/persistence/AGENTS.md index adae208cda..5b95c740a0 100644 --- a/backend/packages/harness/deerflow/persistence/AGENTS.md +++ b/backend/packages/harness/deerflow/persistence/AGENTS.md @@ -66,6 +66,7 @@ with best-effort cleanup, preserving the original publication outcome. - **A scan selects candidates; it grants nothing.** After you take the row locks, check status and lease expiry again. The row can have been requeued and reclaimed while recovery waited, also by the same process. + Late launch writes require the `attempt_count` generation from `claim_queued_run`. - **Check expiry against the database clock after the lock wait.** The transaction-start time is already stale. A lease that expires exactly at scan time is still live. diff --git a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py b/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py index 7ca055ceb6..5b7b81fa38 100644 --- a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py +++ b/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py @@ -680,6 +680,9 @@ class ScheduledTaskRunRepository: status="launching", lease_owner=lease_owner, lease_expires_at=now + timedelta(seconds=lease_seconds), + # attempt_count doubles as the launch-claim generation: + # late bookkeeping passes it back so a reclaim cannot be + # overwritten by an older attempt with the same owner. attempt_count=ScheduledTaskRunRow.attempt_count + 1, ) ) @@ -696,15 +699,19 @@ class ScheduledTaskRunRepository: *, lease_owner: str, error: str | None = None, + expected_attempt_count: int | None = None, ) -> bool: async with self._sf() as session: + conditions = [ + ScheduledTaskRunRow.id == run_record_id, + ScheduledTaskRunRow.status == "launching", + ScheduledTaskRunRow.lease_owner == lease_owner, + ] + if expected_attempt_count is not None: + conditions.append(ScheduledTaskRunRow.attempt_count == expected_attempt_count) result = await session.execute( update(ScheduledTaskRunRow) - .where( - ScheduledTaskRunRow.id == run_record_id, - ScheduledTaskRunRow.status == "launching", - ScheduledTaskRunRow.lease_owner == lease_owner, - ) + .where(*conditions) .values( status="queued", lease_owner=None, @@ -812,6 +819,7 @@ class ScheduledTaskRunRepository: lease_owner: str, error: str, now: datetime, + expected_attempt_count: int | None = None, ) -> bool: """Fail a claimed launch and update its parent without a release gap.""" async with self._sf() as session: @@ -820,6 +828,12 @@ class ScheduledTaskRunRepository: if row is None or row.task_id != task_id or row.status != "launching" or row.lease_owner != lease_owner: await session.rollback() return False + if expected_attempt_count is not None and int(row.attempt_count or 0) != expected_attempt_count: + # A newer claim generation owns this row. The caller's launch + # already lost the race; do not terminalize or release the + # fresh claim's lease. + await session.rollback() + return False row.status = "failed" row.error = error diff --git a/backend/tests/test_scheduled_launch_recovery_fence.py b/backend/tests/test_scheduled_launch_recovery_fence.py index fbb1359715..bed5529be4 100644 --- a/backend/tests/test_scheduled_launch_recovery_fence.py +++ b/backend/tests/test_scheduled_launch_recovery_fence.py @@ -105,6 +105,89 @@ async def test_expired_scan_cannot_requeue_a_fresh_claim(repositories, monkeypat await asyncio.gather(recovering, return_exceptions=True) +async def test_late_launch_bookkeeping_cannot_release_a_reclaimed_claim(repositories): + tasks, recovery, launcher = repositories + now = datetime.now(UTC) + await seed_occurrence(tasks, launcher, now=now) + first = await launcher.claim_queued_run( + "occurrence-1", + lease_owner="same-worker", + now=now, + lease_seconds=5, + global_max_concurrent_runs=1, + ) + assert first is not None + assert first["attempt_count"] == 1 + + assert await recovery.recover_expired_launch_claims(error="expired", now=now + timedelta(seconds=6)) == 1 + fresh = await launcher.claim_queued_run( + "occurrence-1", + lease_owner="same-worker", + now=now + timedelta(seconds=6), + lease_seconds=120, + global_max_concurrent_runs=1, + ) + assert fresh is not None + assert fresh["attempt_count"] == 2 + before = (await launcher.list_by_task("task-1"))[0] + + # The original attempt's late bookkeeping carries attempt 1. Neither the + # busy-thread requeue nor the launch-failure path may touch the fresh + # attempt-2 claim, even though both use the same process lease owner. + assert ( + await launcher.requeue_claimed_run( + "occurrence-1", + lease_owner="same-worker", + error="thread busy", + expected_attempt_count=first["attempt_count"], + ) + is False + ) + assert ( + await launcher.fail_launching_run( + "occurrence-1", + task_id="task-1", + lease_owner="same-worker", + error="launch failed", + now=now + timedelta(seconds=7), + expected_attempt_count=first["attempt_count"], + ) + is False + ) + assert (await launcher.list_by_task("task-1"))[0] == before + + # The fresh claim still holds the only global execution slot. + await seed_occurrence(tasks, launcher, now=now, suffix="2") + assert ( + await launcher.claim_queued_run( + "occurrence-2", + lease_owner="third-worker", + now=now + timedelta(seconds=7), + lease_seconds=120, + global_max_concurrent_runs=1, + ) + is None + ) + + # The current generation can still release its own claim. + assert await launcher.requeue_claimed_run( + "occurrence-1", + lease_owner="same-worker", + error="thread busy", + expected_attempt_count=fresh["attempt_count"], + ) + assert ( + await launcher.claim_queued_run( + "occurrence-2", + lease_owner="third-worker", + now=now + timedelta(seconds=8), + lease_seconds=120, + global_max_concurrent_runs=1, + ) + is not None + ) + + @pytest.mark.parametrize("lease_age", [5, 6]) async def test_recovery_keeps_boundary_lease_and_requeues_expired_claim(repositories, lease_age): tasks, recovery, launcher = repositories diff --git a/backend/tests/test_scheduled_task_service.py b/backend/tests/test_scheduled_task_service.py index db54e39520..f2b59dc8cb 100644 --- a/backend/tests/test_scheduled_task_service.py +++ b/backend/tests/test_scheduled_task_service.py @@ -1,4 +1,5 @@ import asyncio +import logging from datetime import UTC, datetime, timedelta import pytest @@ -141,6 +142,36 @@ class DummyRunRepo: return 0 +class GenerationRunRepo(DummyRunRepo): + """Return a claim generation so service tests can pin the fence plumbing.""" + + async def claim_queued_run(self, run_record_id, **kwargs): + claimed = await super().claim_queued_run(run_record_id, **kwargs) + if claimed is not None: + claimed["attempt_count"] = 7 + return claimed + + +class LostClaimRunRepo(GenerationRunRepo): + """Simulate a newer claim generation winning during requeue.""" + + async def requeue_claimed_run(self, run_record_id, **kwargs): + self.updated.append((run_record_id, {"status": "queued", **kwargs})) + return False + + +class LegacyRunRepo(DummyRunRepo): + """A custom repository whose writes predate the optional generation fence.""" + + async def requeue_claimed_run(self, run_record_id, *, lease_owner, error=None): + self.updated.append((run_record_id, {"status": "queued", "lease_owner": lease_owner, "error": error})) + return True + + async def fail_launching_run(self, run_record_id, *, task_id, lease_owner, error, now): + self.updated.append((run_record_id, {"status": "failed", "task_id": task_id, "lease_owner": lease_owner, "error": error, "now": now})) + return True + + @pytest.mark.asyncio async def test_service_claims_and_dispatches_due_task(): async def fake_launch(**kwargs): @@ -434,6 +465,123 @@ async def test_busy_thread_http_conflict_is_kept_in_queue(): assert run_repo.updated[-1][1]["status"] == "queued" +@pytest.mark.asyncio +async def test_overlap_requeue_is_fenced_by_the_claim_generation(): + async def fake_launch(**_kwargs): + raise ConflictError("Thread thread-1 already has an active run") + + task_repo = DummyTaskRepo( + [ + { + "id": "task-generation-requeue", + "user_id": "user-1", + "thread_id": "thread-1", + "context_mode": "reuse_thread", + "assistant_id": "lead_agent", + "prompt": "Summarize thread", + "schedule_type": "cron", + "schedule_spec": {"cron": "0 9 * * *"}, + "timezone": "UTC", + "status": "enabled", + "overlap_policy": "enqueue", + } + ] + ) + run_repo = GenerationRunRepo() + service = ScheduledTaskService( + task_repo=task_repo, + task_run_repo=run_repo, + launch_run=fake_launch, + poll_interval_seconds=5, + lease_seconds=120, + max_concurrent_runs=3, + ) + + result = await service.dispatch_task(task_repo.rows[0], now=datetime.now(UTC), trigger="scheduled") + + assert result["outcome"] == "queued" + assert run_repo.updated[-1][1]["status"] == "queued" + assert run_repo.updated[-1][1]["expected_attempt_count"] == 7 + + +@pytest.mark.asyncio +async def test_overlap_requeue_logs_when_a_newer_claim_owns_the_row(caplog): + async def fake_launch(**_kwargs): + raise ConflictError("Thread thread-1 already has an active run") + + task_repo = DummyTaskRepo( + [ + { + "id": "task-generation-lost-requeue", + "user_id": "user-1", + "thread_id": "thread-1", + "context_mode": "reuse_thread", + "assistant_id": "lead_agent", + "prompt": "Summarize thread", + "schedule_type": "cron", + "schedule_spec": {"cron": "0 9 * * *"}, + "timezone": "UTC", + "status": "enabled", + "overlap_policy": "enqueue", + } + ] + ) + run_repo = LostClaimRunRepo() + service = ScheduledTaskService( + task_repo=task_repo, + task_run_repo=run_repo, + launch_run=fake_launch, + poll_interval_seconds=5, + lease_seconds=120, + max_concurrent_runs=3, + ) + + with caplog.at_level(logging.WARNING): + result = await service.dispatch_task(task_repo.rows[0], now=datetime.now(UTC), trigger="manual") + + assert result["outcome"] == "queued" + assert "lost its launch claim before requeue bookkeeping" in caplog.text + + +@pytest.mark.asyncio +async def test_pre_launch_failure_is_fenced_by_the_claim_generation(): + async def fake_launch(**_kwargs): + raise RuntimeError("runtime refused to start the run") + + task_repo = DummyTaskRepo( + [ + { + "id": "task-generation-failure", + "user_id": "user-1", + "thread_id": "thread-1", + "context_mode": "reuse_thread", + "assistant_id": "lead_agent", + "prompt": "Summarize thread", + "schedule_type": "cron", + "schedule_spec": {"cron": "0 9 * * *"}, + "timezone": "UTC", + "status": "enabled", + "overlap_policy": "enqueue", + } + ] + ) + run_repo = GenerationRunRepo() + service = ScheduledTaskService( + task_repo=task_repo, + task_run_repo=run_repo, + launch_run=fake_launch, + poll_interval_seconds=5, + lease_seconds=120, + max_concurrent_runs=3, + ) + + result = await service.dispatch_task(task_repo.rows[0], now=datetime.now(UTC), trigger="scheduled") + + assert result["outcome"] == "failed" + assert run_repo.updated[-1][1]["status"] == "failed" + assert run_repo.updated[-1][1]["expected_attempt_count"] == 7 + + @pytest.mark.asyncio async def test_dispatch_task_records_failure_for_legacy_invalid_thread_id(): """Rows persisted before the thread-id contract was centralized may store @@ -1999,6 +2147,40 @@ async def test_launch_passes_user_language_origin_and_run_thread_title(trigger): assert launch["title"] == "检查发布清单 · 10-07 09:00" +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("launch_error", "expected_outcome"), + [ + (ConflictError("Thread busy"), "queued"), + (RuntimeError("launch failed"), "failed"), + ], +) +async def test_legacy_repo_does_not_receive_a_generation_kwarg(launch_error, expected_outcome): + async def fake_launch(**_kwargs): + raise launch_error + + task_repo = DummyTaskRepo([_provenance_task()]) + run_repo = LegacyRunRepo() + service = ScheduledTaskService( + task_repo=task_repo, + task_run_repo=run_repo, + launch_run=fake_launch, + poll_interval_seconds=5, + lease_seconds=120, + max_concurrent_runs=3, + ) + + result = await service.dispatch_task( + task_repo.rows[0], + now=datetime(2026, 10, 7, 1, 0, tzinfo=UTC), + trigger="manual", + ) + + assert result["outcome"] == expected_outcome + assert run_repo.updated + assert "expected_attempt_count" not in run_repo.updated[0][1] + + @pytest.mark.asyncio async def test_manual_launch_origin_is_stable_across_retries(): task_repo = DummyTaskRepo([_provenance_task()]) diff --git a/docs/guide/scheduled-tasks.md b/docs/guide/scheduled-tasks.md index e70c3cf951..6612e4d601 100644 --- a/docs/guide/scheduled-tasks.md +++ b/docs/guide/scheduled-tasks.md @@ -23,6 +23,7 @@ Current MVP capabilities: - A scheduled run whose completion hook fails is finalized from its durable terminal run on the next scheduler poll instead of staying `running` until a restart. - A storage or bookkeeping error on one occurrence during queue cleanup or dispatch is logged without stopping the remaining occurrences and healthy due tasks in that poll. The affected occurrence remains subject to normal queue and lease recovery; persistent storage failures can delay its recovery. A run that has exceeded `scheduler.queue_timeout_seconds` never starts while cleanup is retried, and expired rows do not consume capacity or fair-queue slots from healthy waiting work; they stay queued until a later timeout sweep records the skip. - A recurring task whose saved schedule can no longer be computed (for example, its timezone is missing from the time zone database after an upgrade) is paused with the reason as its last error, and every other task keeps running. Resuming it reports the problem until you fix the schedule. +- A late failure or busy-thread retry from a recovered scheduled launch no longer releases the execution slot of the newer claim that replaced it. - Share execution slots fairly between task owners: one owner may have at most `scheduler.max_concurrent_runs_per_user` scheduled runs starting or running at a time (default 2, never more than `max_concurrent_runs`; `0` turns the per-owner cap off), and the waiting queue is drained owner by owner, so one owner's backlog never holds back another owner's run. A run that waits longer than `scheduler.queue_timeout_seconds` is skipped, and its history reads "Skipped: it waited too long for a free slot" - Freeze a task's definition while an occurrence is `queued`, `launching`, or `running`, so a durable occurrence cannot silently pick up a different prompt, thread, or schedule; transitioning a task to paused or deleting it cancels an existing waiting occurrence, while `launching`/`running` work must finish before those mutations are retried and an explicit manual trigger may still wait and run without resuming a paused schedule - Pause, resume, trigger, inspect history, and delete tasks