mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-17 09:01:04 +00:00
* feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks
- Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal
streak boundary) and stop_condition (the user's stop rule, own column).
- Stop rule is normalized (stop_rule.py) and appended only at launch; the
stored prompt never contains it. Wording adapts when scheduler.tool_enabled
is off.
- Willem 1: page-created tasks get goal evaluation, <standing_notes>, the
previous-run reference and own-schedule stop; request_stop and the
scheduled capability no longer require an origin conversation.
- Repository: single reactivation choke point raising
ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop
condition/note changes (not on resume); goal-check failures neither count
nor reset the streak; pause of a finished task returns "finished"; trials
on a paused task keep the host pause marker.
- New repository reads: automatic_runs_used_for, active_run_status_for,
list_manageable_from_thread, thread_relation/thread_run on thread tasks,
task_ids_for_thread, run_number, and run_number/total_tokens/summary on
run history rows.
- REST: goal/max_runs/end_at/stop_condition on create and PATCH (null
clears, naive end_at = task timezone), resume recomputes from now with an
optional renewal body, once_time_passed, task_finished, limits_exhausted,
scheduler_not_running on create, trigger returns outcome/existing/thread_id,
task responses add automatic_runs_used and active_run_status.
- Coded errors {code, message, params} for routes, shared validation and the
chat capability (scheduled_task_errors.py), pinned by
contracts/scheduled_task_errors_contract.json; goal notes contract v2 with
check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py).
- /api/features adds scheduled_tasks {available, running, tool_enabled,
min_interval_seconds}; ScheduledTaskService.is_running.
- Docs: six README scheduler sections, API.md scheduled tasks section
(breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs.
- Tests: migration 0031, stop rule, error contract, run listing, lifecycle
over HTTP, repository, service and capability cases; updated tests for the
new error shape, finished-task pause and head revision.
* feat(scheduler): conversation update/resume, run-conversation authority and readable launches
- schedule_task: update and resume actions, stop_condition and clear_fields
parameters, plain parameter types (value rules come back as coded results),
rewritten docstrings for schedule_task and stop_scheduled_task.
- Capability: update/resume share the REST validation and error codes;
interactive turns in a task's run conversation may manage that task;
scheduled runs keep only own-stop; grants require a running scheduler.
- Results: compact JSON-safe task views with local times, display card/text,
last run in list, timezone_source/browser_timezone, now_local; no ISO, user,
lease or origin fields; stop_instructions removed.
- Timezone: explicit > browser (context.client_timezone, validated in
start_run and never forwarded to the run config) > not needed for intervals
and offset run_at > timezone_required.
- Trial gate accepts natural direct requests after acknowledgements and
particles; trial results report thread_id and existing.
- Scheduled launches: stable message id scheduled-<task_run_id>, server-owned
deerflow_scheduled_origin metadata with the user-language parts, run thread
title for fresh-thread tasks.
- Worker records goal continuations in the verdict; lead prompt language rule
covers stored task text and scheduled runs.
- Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs.
* feat(frontend): scheduled-task core types, formatting, errors and glossary
- throwGatewayApiError now throws GatewayApiError (status, code, params,
rawMessage); message stays the string detail or detail.message
- scheduled-task types gain stop_condition, automatic_runs_used,
active_run_status, origin/notes, run_number/total_tokens/summary, thread
relation, trigger result, renewal and scheduled origin; isTaskBusy and
hasActiveRun helpers
- api: fetchScheduledTask, resume with optional renewal body, typed trigger
result and update payload with null clears
- hooks: polling at 3 s while a run is active (list, thread list, run
history), useScheduledTask with live gating and final 404, useInView,
localized error toasts; run history refreshes tasks when a run finishes
- new pure modules: errors.ts (contract code to localized copy), format.ts
(relative task times, viewer time, display zone), status.ts, actions.ts,
run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked
goal checks, describeTaskOutcome and parseStopRunId
- features: scheduled_tasks block with defaults when missing
- buildRunContext sends client_timezone
- titleOfThread takes a localized untitled label; breadcrumb names scheduled
tasks and agents; reference chips use the localized untitled label
- schedule input: shared browserTimeZone, labelled selects, aria-pressed
type toggles, interval minimum from the hint template
- i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes,
recipe instructions moved into i18n
- e2e mocks follow the new task, trigger, resume, thread-task and features
shapes; expectNoRawIdentifiers helpers for e2e and DOM tests
- tests for errors, format, cron, status, actions, goal outcome, run errors,
hooks, run context, thread titles and features; scheduled e2e copy updated
* feat(frontend): list-first scheduled tasks page with outcomes and editable goals
- page.tsx is now a thin composition of components/workspace/scheduled-tasks:
status tabs (All/Active/Paused/Finished) and search over the task list,
?task_id= deep link with router.replace on selection, ?thread_id= chip
with Show all, empty state, scheduler-off notice and unavailable state
- task list rows show the schedule in words and one status line (next run,
running now, paused by agent, auto-paused, finished by its cap)
- task detail: status badge, actions from availableActions with reasons,
More actions menu (Duplicate, Copy task ID, Delete), outcome notice,
Runs / Stops when / Each run's goal / Does / Notes from chat / History,
origin chat link; success toasts for every action
- run rows: run number or Trial run, summary, goal and assumption badges,
continuations, tokens, Open chat link; raw errors and the evaluator
reason only behind Details
- create/edit/duplicate dialog with goal, stop condition, safety caps and
advanced agent/context options; PATCH sends only changed fields, null
clears a cap
- renew dialog raises or removes caps and resumes in one request; a
limits_exhausted Resume opens it
- removed i18n keys the old page used; added feedback.resumedNoTime
- recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for
the new page; DOM tests for the detail and the form dialog
- CHANGELOG entries and the frontend AGENTS.md scheduled-task note
* feat(frontend): live schedule card, run conversation provenance and header entry
- schedule_task results (create, update, pause, resume, trial, delete) become
an assistant:scheduled-task message group rendered as a live card: schedule,
next run, stop condition with the safety cap, goal, where results go, Run
once now, Pause/Resume and Open task; the card polls the task only while it
is on screen and shows deleted, scheduler-off and limits-exhausted states
- one card per task per assistant turn; the card group is not a turn boundary
for usage, branching, editing, outline or skill attribution
- a human message with deerflow_scheduled_origin stays a human group with
scheduledOrigin and renders as a "Scheduled run / Trial run" block with the
task instructions, stop condition and notes collapsed; the launched text is
never shown and the message is not editable
- readable step labels for schedule_task actions and stop_scheduled_task
- ThreadScheduledTasksButton replaces the header link: hidden without tasks,
shows the count, links to the task (or the chat filter), and a run's chat
links to its task
- recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread
e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card,
run block, header button, step labels and reconnect hydration
- i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and
frontend AGENTS.md notes
* fix(scheduler): address verification findings
Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check.
* fix(scheduler): address review findings
* fix(scheduled-tasks): polish from live acceptance
- zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒,
and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2
days) instead of 每 1 分钟 or 每 24 小时; several units keep the number
- zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between
the interpolated time and the Chinese after it; a copy test guards every
scheduled-task zh string against a placeholder glued to Chinese
- run summary: when the first line of the final reply only leads in to a list
(ends with ":" or ":"), the list items follow it as plain text (bullets,
numbering, checkboxes, bold, code and links stripped), joined with ";" for
CJK text and "; " otherwise, within the 160-character cap
- a schedule_task result (create, update, pause, resume, trial, delete) that
arrives in a chat invalidates that chat's task list, the tasks list and the
named task, so the header button and cards update without waiting for the
idle poll; results already in the loaded history do not refetch
- e2e and unit fixtures are now the live acceptance recordings (DeepSeek
Flash, demo data, fictional names): live-weekday.json (create, trial, edit,
pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1
and run 2, paused by the agent), fetched with GET from the demo Gateway and
trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card,
run-thread and tasks-page specs run in both locales over the recorded data
* docs(changelog): reference #6378 for the scheduled-tasks entries
* docs(agents): shorten the scheduled-task pointer to stay under the guidance limit
* test(scheduled-tasks): pin the live takes to their recorded time zone
The live fixtures were recorded in Asia/Shanghai, and an interval task
reads in the viewer's zone, so the 16:48 assertions failed on UTC CI
runners. Pin the browser zone in the two live-take e2e specs and the
zh live-take unit test.
* fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in
- The edit, duplicate and renew dialogs keep a stored end time's instant
while its field still shows it. A minute-precision wall value cannot
keep seconds or name the later 01:30 of a DST fall-back, so a
title-only edit moved the cap an hour earlier, and raising the run
limit re-sent the end time without its seconds.
- Creating or duplicating a task from a chat's task view drops the chat
filter when the new task does not belong to that chat, instead of
showing "This task isn't one of this chat's tasks".
* fix(scheduled-tasks): tie pause notices to the pause they describe
- Paused by agent links and times only the run named in last_error. When
that run is not loaded, the notice shows no link and the task's own time
instead of an earlier run that also asked to stop.
- The outcome notice reads the latest history page whichever page is
shown, so browsing older runs no longer swaps the auto-pause reason and
"Open latest run" to an earlier miss. It shares that page's request.
* fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes
- The edit form compares a cron in the form its schedule input emits, so an
untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only
edit no longer re-arms or refuses a finished task.
- Chat resume asks for a timezone, as update does, when a naive end_at would
land in the UTC placeholder of a zone-free task.
- Re-arming a task clears a host pause marker, as resume does, so a later
manual pause no longer reads as "Paused by agent".
* fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages
useLatestScheduledTaskRuns kept React Query's default refetch on mount,
focus and reconnect, so going offline and back while an older history page
was shown fetched the latest page again. Refreshing stays with the history,
which refreshes the latest page only while it is shown.
* test(scheduled-tasks): let the viewer's time carry its own day label
Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days,
so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the
fixed "01:00 · 09:00 your time" assertion failed depending on the hour.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
1674 lines
59 KiB
Python
1674 lines
59 KiB
Python
import asyncio
|
||
from datetime import UTC, datetime, timedelta
|
||
|
||
import pytest
|
||
|
||
from app.scheduler.service import ScheduledTaskService
|
||
from deerflow.runtime import ConflictError, RunStatus
|
||
from deerflow.runtime.runs.manager import RunRecord
|
||
from deerflow.runtime.runs.schemas import DisconnectMode
|
||
|
||
|
||
class DummyTaskRepo:
|
||
def __init__(self, rows):
|
||
self.rows = rows
|
||
self.claimed = False
|
||
self.updated = None
|
||
self.completions = []
|
||
self.release_calls = []
|
||
self.cancelled_stuck_once = None
|
||
self.reconciled_stuck_once = None
|
||
|
||
async def cancel_stuck_once_tasks(self, *, error):
|
||
self.cancelled_stuck_once = error
|
||
return 0
|
||
|
||
async def reconcile_stuck_once_tasks(self, **kwargs):
|
||
self.reconciled_stuck_once = kwargs
|
||
return 0
|
||
|
||
async def claim_dispatch_lease(self, task_id, **_kwargs):
|
||
return next((dict(row) for row in self.rows if row["id"] == task_id), None)
|
||
|
||
async def release_queued_admission_lease(self, task_id):
|
||
return False
|
||
|
||
async def release_dispatch_lease(self, task_id, **kwargs):
|
||
self.release_calls.append((task_id, kwargs))
|
||
return True
|
||
|
||
async def claim_due_tasks(self, **_kwargs):
|
||
if self.claimed:
|
||
return []
|
||
self.claimed = True
|
||
return self.rows
|
||
|
||
async def update_after_launch(self, *args, **kwargs):
|
||
self.updated = (args, kwargs)
|
||
|
||
async def complete_run(self, task_id, **kwargs):
|
||
self.completions.append((task_id, kwargs))
|
||
return True
|
||
|
||
async def get(self, task_id: str, *, user_id: str):
|
||
row = next((item for item in self.rows if item["id"] == task_id and item["user_id"] == user_id), None)
|
||
return dict(row) if row is not None else None
|
||
|
||
async def get_internal(self, task_id: str):
|
||
row = next((item for item in self.rows if item["id"] == task_id), None)
|
||
return dict(row) if row is not None else None
|
||
|
||
async def update(self, task_id: str, *, user_id: str, updates):
|
||
row = next((item for item in self.rows if item["id"] == task_id and item["user_id"] == user_id), None)
|
||
if row is None:
|
||
return None
|
||
row.update(updates)
|
||
return dict(row)
|
||
|
||
|
||
class DummyRunRepo:
|
||
def __init__(self, *, active=False, active_count=0):
|
||
self.created = None
|
||
self.updated = []
|
||
self.active = active
|
||
self.active_count = active_count
|
||
self.stale_marked = None
|
||
self.reconciled = None
|
||
self.reconcile_count = 0
|
||
|
||
async def count_active_runs(self):
|
||
return self.active_count
|
||
|
||
async def list_queued_runs(self, *, limit):
|
||
return []
|
||
|
||
async def expire_queued_runs(self, **_kwargs):
|
||
return []
|
||
|
||
async def recover_expired_launch_claims(self, **_kwargs):
|
||
return 0
|
||
|
||
async def get_active_run(self, task_id):
|
||
if not self.active:
|
||
return None
|
||
return {
|
||
"id": "task-run-active",
|
||
"task_id": task_id,
|
||
"thread_id": "thread-active",
|
||
"status": "running",
|
||
}
|
||
|
||
async def claim_queued_run(self, run_record_id, *, global_max_concurrent_runs, **_kwargs):
|
||
if self.active_count >= global_max_concurrent_runs:
|
||
return None
|
||
return {"id": run_record_id, "status": "launching"}
|
||
|
||
async def requeue_claimed_run(self, run_record_id, **kwargs):
|
||
self.updated.append((run_record_id, {"status": "queued", **kwargs}))
|
||
return True
|
||
|
||
async def create(self, **kwargs):
|
||
self.created = kwargs
|
||
return {"id": kwargs["run_record_id"]}
|
||
|
||
async def update_status(self, run_record_id, **kwargs):
|
||
self.updated.append((run_record_id, kwargs))
|
||
return True
|
||
|
||
async def reconcile_launched_run(self, run_record_id, **kwargs):
|
||
self.updated.append((run_record_id, {"reconciled": True, **kwargs}))
|
||
return True
|
||
|
||
async def fail_launching_run(self, run_record_id, **kwargs):
|
||
self.updated.append((run_record_id, {"status": "failed", **kwargs}))
|
||
return True
|
||
|
||
async def has_active_runs(self, task_id):
|
||
return self.active
|
||
|
||
async def mark_stale_active_runs(self, *, error):
|
||
self.stale_marked = error
|
||
return 0
|
||
|
||
async def reconcile_active_runs(self, **kwargs):
|
||
self.reconcile_count += 1
|
||
self.reconciled = kwargs
|
||
return 0
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_service_claims_and_dispatches_due_task():
|
||
async def fake_launch(**kwargs):
|
||
assert kwargs["owner_user_id"] == "user-1"
|
||
assert kwargs["metadata"]["scheduled_task_id"] == "task-1"
|
||
assert kwargs["metadata"]["scheduled_trigger"] == "scheduled"
|
||
return {"run_id": "run-1", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-1",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-1",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "once",
|
||
"schedule_spec": {"run_at": "2026-07-02T01:00:00+00:00"},
|
||
"timezone": "UTC",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
await service.run_once(now=datetime.now(UTC) + timedelta(days=1))
|
||
|
||
assert run_repo.created["task_id"] == "task-1"
|
||
assert run_repo.updated[0][1]["status"] == "running"
|
||
assert run_repo.updated[0][1]["protect_terminal"] is True
|
||
# `once` terminal status is owned by handle_run_completion, not the launch.
|
||
assert task_repo.updated[1]["status"] == "running"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_manual_trigger_keeps_paused_cron_task_paused():
|
||
async def fake_launch(**kwargs):
|
||
return {"run_id": "run-2", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-2",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-1",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "paused",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
await service.dispatch_task(
|
||
task_repo.rows[0],
|
||
now=datetime.now(UTC),
|
||
trigger="manual",
|
||
)
|
||
|
||
assert task_repo.updated[1]["status"] == "paused"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fresh_thread_per_run_creates_new_execution_thread():
|
||
async def fake_launch(**kwargs):
|
||
assert kwargs["thread_id"] != "thread-template"
|
||
return {"run_id": "run-3", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-3",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-template",
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
await service.dispatch_task(
|
||
task_repo.rows[0],
|
||
now=datetime.now(UTC),
|
||
trigger="scheduled",
|
||
)
|
||
|
||
assert run_repo.created["thread_id"] != "thread-template"
|
||
assert task_repo.updated[1]["last_thread_id"] == run_repo.created["thread_id"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_scheduled_overlap_conflict_is_kept_in_queue():
|
||
async def fake_launch(**_kwargs):
|
||
raise ConflictError("Thread thread-1 already has an active run")
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-4",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-1",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "running",
|
||
"overlap_policy": "enqueue",
|
||
"last_run_id": "run-old",
|
||
"last_thread_id": "thread-1",
|
||
"last_run_at": "2026-07-01T00:00:00+00:00",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(
|
||
task_repo.rows[0],
|
||
now=datetime.now(UTC),
|
||
trigger="scheduled",
|
||
)
|
||
|
||
assert result["outcome"] == "queued"
|
||
assert run_repo.created["status"] == "queued"
|
||
assert run_repo.updated[-1][1]["status"] == "queued"
|
||
assert task_repo.updated is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_manual_overlap_conflict_is_kept_in_queue():
|
||
async def fake_launch(**_kwargs):
|
||
raise ConflictError("Thread thread-1 already has an active run")
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-5",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-1",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
"overlap_policy": "enqueue",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(
|
||
task_repo.rows[0],
|
||
now=datetime.now(UTC),
|
||
trigger="manual",
|
||
)
|
||
|
||
assert result["outcome"] == "queued"
|
||
assert run_repo.updated[-1][1]["status"] == "queued"
|
||
assert task_repo.release_calls == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_dispatch_task_records_failure_for_legacy_invalid_thread_id():
|
||
"""Rows persisted before the thread-id contract was centralized may store
|
||
IDs that fail the canonical pattern (dots, >64 chars). Dispatch must record
|
||
the failure through normal bookkeeping instead of raising — an uncaught
|
||
ValueError surfaces as HTTP 500 on manual trigger and, in the poller,
|
||
aborts the rest of the claimed batch every cycle."""
|
||
|
||
async def fake_launch(**_kwargs):
|
||
raise AssertionError("launch_run must not be called for an invalid thread_id")
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-legacy",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread.with.dot",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(
|
||
task_repo.rows[0],
|
||
now=datetime.now(UTC),
|
||
trigger="scheduled",
|
||
)
|
||
|
||
assert result["outcome"] == "failed"
|
||
assert result["task_run_id"] is None
|
||
assert result["run_id"] is None
|
||
assert "Invalid thread_id" in result["error"]
|
||
assert run_repo.created is None
|
||
assert task_repo.updated[1]["last_error"] == result["error"]
|
||
assert task_repo.updated[1]["last_thread_id"] == "thread.with.dot"
|
||
assert task_repo.updated[1]["increment_run_count"] is False
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_run_once_continues_batch_after_invalid_thread_id():
|
||
"""A poison legacy row must not prevent later claimed tasks from dispatching."""
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-ok", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-legacy",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread.with.dot",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
},
|
||
{
|
||
"id": "task-valid",
|
||
"user_id": "user-1",
|
||
"thread_id": "thread-ok",
|
||
"context_mode": "reuse_thread",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
},
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
await service.run_once(now=datetime.now(UTC))
|
||
|
||
assert len(launched) == 1
|
||
assert launched[0]["thread_id"] == "thread-ok"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_handle_run_completion_uses_atomic_repository_boundary():
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-6",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
}
|
||
]
|
||
)
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=lambda **_kwargs: None,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
record = RunRecord(
|
||
run_id="run-6",
|
||
thread_id="thread-6",
|
||
assistant_id="lead_agent",
|
||
status=RunStatus.success,
|
||
on_disconnect=DisconnectMode.continue_,
|
||
metadata={
|
||
"scheduled_task_id": "task-6",
|
||
"scheduled_task_run_id": "task-run-6",
|
||
},
|
||
user_id="user-1",
|
||
)
|
||
|
||
await service.handle_run_completion(record)
|
||
|
||
assert len(task_repo.completions) == 1
|
||
task_id, completion = task_repo.completions[0]
|
||
assert task_id == "task-6"
|
||
assert completion["user_id"] == "user-1"
|
||
assert completion["task_run_id"] == "task-run-6"
|
||
assert completion["run_id"] == "run-6"
|
||
assert completion["status"] == "success"
|
||
assert completion["error"] is None
|
||
assert completion["finished_at"].tzinfo == UTC
|
||
assert run_repo.updated == []
|
||
|
||
|
||
def _make_service(task_repo, run_repo):
|
||
return ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=lambda **_kwargs: None,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
|
||
def _once_task_row(task_id="task-once", status="running"):
|
||
return {
|
||
"id": task_id,
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"title": "Once summary",
|
||
"prompt": "Summarize thread",
|
||
"schedule_type": "once",
|
||
"schedule_spec": {"run_at": "2026-07-02T01:00:00+00:00"},
|
||
"timezone": "UTC",
|
||
"status": status,
|
||
}
|
||
|
||
|
||
def _completion_record(status, *, task_id="task-once", error=None, trigger="scheduled"):
|
||
return RunRecord(
|
||
run_id="run-x",
|
||
thread_id="thread-x",
|
||
assistant_id="lead_agent",
|
||
status=status,
|
||
on_disconnect=DisconnectMode.continue_,
|
||
metadata={
|
||
"scheduled_task_id": task_id,
|
||
"scheduled_task_run_id": "task-run-x",
|
||
"scheduled_trigger": trigger,
|
||
},
|
||
user_id="user-1",
|
||
error=error,
|
||
)
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
@pytest.mark.parametrize(
|
||
("run_status", "error", "occurrence_status", "expected_error"),
|
||
[
|
||
(RunStatus.success, None, "success", None),
|
||
(RunStatus.error, "boom", "failed", "boom"),
|
||
(RunStatus.timeout, "time limit", "failed", "time limit"),
|
||
(RunStatus.interrupted, None, "interrupted", "run was interrupted before completion"),
|
||
(RunStatus.interrupted, "cancelled by user", "interrupted", "cancelled by user"),
|
||
],
|
||
)
|
||
async def test_handle_run_completion_forwards_terminal_outcome(run_status, error, occurrence_status, expected_error):
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
service = _make_service(task_repo, run_repo)
|
||
|
||
await service.handle_run_completion(_completion_record(run_status, error=error))
|
||
|
||
assert len(task_repo.completions) == 1
|
||
task_id, completion = task_repo.completions[0]
|
||
assert task_id == "task-once"
|
||
assert completion["status"] == occurrence_status
|
||
assert completion["error"] == expected_error
|
||
assert run_repo.updated == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_existing_running_occurrence_blocks_duplicate_fresh_thread_run():
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-9", "thread_id": kwargs["thread_id"]}
|
||
|
||
row = _once_task_row(task_id="task-9")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "status": "running", "overlap_policy": "enqueue"})
|
||
task_repo = DummyTaskRepo([row])
|
||
run_repo = DummyRunRepo(active=True)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="scheduled")
|
||
|
||
assert result["outcome"] == "conflict"
|
||
assert launched == []
|
||
assert run_repo.created is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_startup_sweep_reconciles_stale_runs_and_stuck_once_tasks():
|
||
task_repo = DummyTaskRepo([])
|
||
run_repo = DummyRunRepo()
|
||
service = _make_service(task_repo, run_repo)
|
||
|
||
await service.start()
|
||
await service.stop()
|
||
|
||
assert run_repo.stale_marked is not None
|
||
assert task_repo.cancelled_stuck_once == run_repo.stale_marked
|
||
|
||
|
||
@pytest.mark.parametrize("failure_stage", ["occurrence", "parent"])
|
||
@pytest.mark.asyncio
|
||
async def test_single_instance_start_fails_closed_before_polling(failure_stage):
|
||
order = []
|
||
|
||
class StartupTaskRepo(DummyTaskRepo):
|
||
def __init__(self):
|
||
super().__init__([])
|
||
self.recovery_attempts = 0
|
||
|
||
async def cancel_stuck_once_tasks(self, *, error):
|
||
self.recovery_attempts += 1
|
||
order.append("parent")
|
||
assert service._task is None
|
||
if failure_stage == "parent" and self.recovery_attempts == 1:
|
||
raise RuntimeError("simulated parent recovery failure")
|
||
return await super().cancel_stuck_once_tasks(error=error)
|
||
|
||
class StartupRunRepo(DummyRunRepo):
|
||
def __init__(self):
|
||
super().__init__()
|
||
self.recovery_attempts = 0
|
||
|
||
async def mark_stale_active_runs(self, *, error):
|
||
self.recovery_attempts += 1
|
||
order.append("occurrence")
|
||
assert service._task is None
|
||
if failure_stage == "occurrence" and self.recovery_attempts == 1:
|
||
raise RuntimeError("simulated occurrence recovery failure")
|
||
return await super().mark_stale_active_runs(error=error)
|
||
|
||
task_repo = StartupTaskRepo()
|
||
run_repo = StartupRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=lambda **_kwargs: None,
|
||
poll_interval_seconds=0,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
async def parked_run_loop():
|
||
await service._stop.wait()
|
||
|
||
service._run_loop = parked_run_loop
|
||
try:
|
||
with pytest.raises(RuntimeError, match=f"simulated {failure_stage} recovery failure"):
|
||
await service.start()
|
||
assert run_repo.recovery_attempts == 1
|
||
assert task_repo.recovery_attempts == (0 if failure_stage == "occurrence" else 1)
|
||
assert order == (["occurrence"] if failure_stage == "occurrence" else ["occurrence", "parent"])
|
||
assert service._task is None
|
||
assert task_repo.claimed is False
|
||
finally:
|
||
await service.stop()
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_multi_instance_start_uses_lease_aware_reconciliation():
|
||
task_repo = DummyTaskRepo([])
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=lambda **_kwargs: None,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
multi_instance=True,
|
||
run_lease_grace_seconds=17,
|
||
)
|
||
|
||
await service.start()
|
||
await asyncio.sleep(0)
|
||
await service.stop()
|
||
|
||
assert run_repo.reconcile_count == 1
|
||
assert run_repo.reconciled is not None
|
||
assert run_repo.reconciled["lease_grace_seconds"] == 17
|
||
assert task_repo.reconciled_stuck_once is not None
|
||
assert task_repo.reconciled_stuck_once["lease_grace_seconds"] == 17
|
||
assert task_repo.cancelled_stuck_once is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_manual_trigger_with_active_run_returns_conflict_without_launching():
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-x", "thread_id": kwargs["thread_id"]}
|
||
|
||
row = _once_task_row(task_id="task-manual-busy")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "status": "enabled", "overlap_policy": "enqueue"})
|
||
task_repo = DummyTaskRepo([row])
|
||
run_repo = DummyRunRepo(active=True)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual")
|
||
|
||
assert result["outcome"] == "conflict"
|
||
assert launched == []
|
||
# Nothing was scheduled to happen, so no run-history row is recorded.
|
||
assert run_repo.created is None
|
||
assert result["task_run_id"] is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_run_once_admits_due_occurrences_independently_of_execution_budget():
|
||
claim_limits = []
|
||
|
||
class BudgetTaskRepo(DummyTaskRepo):
|
||
async def claim_due_tasks(self, **kwargs):
|
||
claim_limits.append(kwargs["limit"])
|
||
return []
|
||
|
||
task_repo = BudgetTaskRepo([])
|
||
run_repo = DummyRunRepo(active_count=2)
|
||
service = _make_service(task_repo, run_repo)
|
||
|
||
await service.run_once(now=datetime.now(UTC))
|
||
assert claim_limits == [3]
|
||
|
||
run_repo.active_count = 3
|
||
await service.run_once(now=datetime.now(UTC))
|
||
assert claim_limits == [3, 3]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_launch_bookkeeping_passes_protect_terminal():
|
||
async def fake_launch(**kwargs):
|
||
return {"run_id": "run-pt", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo([_once_task_row(task_id="task-pt", status="enabled")])
|
||
run_repo = DummyRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
await service.dispatch_task(task_repo.rows[0], now=datetime.now(UTC), trigger="scheduled")
|
||
|
||
assert task_repo.updated[1]["protect_terminal"] is True
|
||
|
||
|
||
class _StatefulRunRepo:
|
||
"""Stateful fake ``ScheduledTaskRunRepository`` for the #4452 tests.
|
||
|
||
Mirrors just enough of the real repository to let a second dispatch
|
||
observe the active slot held by the first:
|
||
|
||
* ``create()`` tracks each row by id, carrying its ``status`` and
|
||
``run_id``;
|
||
* with ``fail_first_update=True`` the very FIRST ``update_status()``
|
||
call raises, simulating a transient DB failure on the
|
||
``queued -> running`` write that fires right after ``_launch_run``
|
||
returns a live ``run_id``; every later ``update_status()`` applies;
|
||
* ``has_active_runs()`` reflects whether any tracked row for the task
|
||
is still in an active status (``queued``/``running``), exactly like
|
||
the partial unique index ``uq_scheduled_task_run_active``.
|
||
"""
|
||
|
||
_ACTIVE = {"queued", "launching", "running"}
|
||
|
||
def __init__(self, *, fail_first_update: bool = False, fail_updates: int = 0) -> None:
|
||
self.created: list[dict] = []
|
||
self.updates: list[tuple[str, dict]] = []
|
||
self.rows: dict[str, dict] = {}
|
||
self._fail_updates = max(fail_updates, 1 if fail_first_update else 0)
|
||
self._updates_raised = 0
|
||
|
||
async def count_active_runs(self) -> int:
|
||
return sum(1 for row in self.rows.values() if row["status"] in {"launching", "running"})
|
||
|
||
async def list_queued_runs(self, *, limit: int) -> list[dict]:
|
||
return []
|
||
|
||
async def expire_queued_runs(self, **_kwargs) -> list[dict]:
|
||
return []
|
||
|
||
async def create(self, **kwargs) -> dict:
|
||
self.created.append(kwargs)
|
||
self.rows[kwargs["run_record_id"]] = {
|
||
"id": kwargs["run_record_id"],
|
||
"task_id": kwargs["task_id"],
|
||
"thread_id": kwargs["thread_id"],
|
||
"trigger": kwargs["trigger"],
|
||
"status": kwargs["status"],
|
||
"run_id": None,
|
||
}
|
||
return {"id": kwargs["run_record_id"]}
|
||
|
||
async def get_active_run(self, task_id: str) -> dict | None:
|
||
return next(
|
||
(dict(row) for row in self.rows.values() if row["task_id"] == task_id and row["status"] in self._ACTIVE),
|
||
None,
|
||
)
|
||
|
||
async def claim_queued_run(self, run_record_id: str, **_kwargs) -> dict | None:
|
||
row = self.rows.get(run_record_id)
|
||
if row is None or row["status"] != "queued":
|
||
return None
|
||
row["status"] = "launching"
|
||
return dict(row)
|
||
|
||
async def requeue_claimed_run(self, run_record_id: str, **_kwargs) -> bool:
|
||
row = self.rows.get(run_record_id)
|
||
if row is None or row["status"] != "launching":
|
||
return False
|
||
row["status"] = "queued"
|
||
return True
|
||
|
||
async def update_status(self, run_record_id: str, **kwargs) -> bool:
|
||
self.updates.append((run_record_id, kwargs))
|
||
if self._updates_raised < self._fail_updates:
|
||
# The launch-path queued->running write fails AFTER _launch_run has
|
||
# already returned a live run_id. Some tests fail both attempts to
|
||
# pin the last-resort active-slot behavior.
|
||
self._updates_raised += 1
|
||
raise RuntimeError("simulated transient DB error on queued->running write")
|
||
row = self.rows.get(run_record_id)
|
||
if row is None:
|
||
return False
|
||
if "status" in kwargs:
|
||
row["status"] = kwargs["status"]
|
||
if kwargs.get("run_id") is not None:
|
||
row["run_id"] = kwargs["run_id"]
|
||
return True
|
||
|
||
async def reconcile_launched_run(self, run_record_id: str, **kwargs) -> bool:
|
||
row = self.rows.get(run_record_id)
|
||
if row is None:
|
||
return False
|
||
row["status"] = "running"
|
||
row["run_id"] = kwargs["run_id"]
|
||
return True
|
||
|
||
async def fail_launching_run(self, run_record_id: str, **kwargs) -> bool:
|
||
row = self.rows.get(run_record_id)
|
||
if row is None or row["status"] != "launching":
|
||
return False
|
||
row["status"] = "failed"
|
||
return True
|
||
|
||
async def has_active_runs(self, task_id: str) -> bool:
|
||
return any(row["task_id"] == task_id and row["status"] in self._ACTIVE for row in self.rows.values())
|
||
|
||
async def mark_stale_active_runs(self, *, error: str) -> int:
|
||
return 0
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_post_launch_bookkeeping_failure_does_not_release_active_slot():
|
||
"""Regression for issue #4452.
|
||
|
||
A transient failure in the ``queued -> running`` bookkeeping write
|
||
(after ``_launch_run`` has already returned a live ``run_id``) must NOT
|
||
flip the task-run row to ``failed``: ``failed`` is outside the partial
|
||
unique index ``uq_scheduled_task_run_active``, so releasing the slot
|
||
would let the next dispatch launch a DUPLICATE run. The fix keeps the
|
||
row ``running`` with the launched ``run_id`` retained for recovery,
|
||
reconciliation, and cancellation.
|
||
"""
|
||
launched: list[dict] = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": f"run-{len(launched)}", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-4452",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "do the thing",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "*/5 * * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
"overlap_policy": "enqueue",
|
||
}
|
||
]
|
||
)
|
||
run_repo = _StatefulRunRepo(fail_first_update=True)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
now = datetime.now(UTC)
|
||
task = dict(task_repo.rows[0])
|
||
|
||
first = await service.dispatch_task(task, now=now, trigger="scheduled")
|
||
# The run launched despite the post-launch bookkeeping error; the
|
||
# outcome and run_id reflect that a live run is in flight.
|
||
assert first["outcome"] == "launched"
|
||
assert first["run_id"] == "run-1"
|
||
assert first["error"] is not None # the bookkeeping error is surfaced, not hidden
|
||
|
||
# Second dispatch must observe the active slot held by run-1 and NOT
|
||
# launch a duplicate. On main (bug) this would launch run-2 here.
|
||
second = await service.dispatch_task(task, now=now, trigger="scheduled")
|
||
assert len(launched) == 1, launched
|
||
assert second["outcome"] == "conflict", second
|
||
|
||
# The launched run_id is retained on the task-run row (status "running",
|
||
# not "failed") so reconciliation / cancellation can still reach it.
|
||
first_row_id = run_repo.created[0]["run_record_id"]
|
||
assert run_repo.rows[first_row_id]["status"] == "running"
|
||
assert run_repo.rows[first_row_id]["run_id"] == "run-1"
|
||
|
||
# The bookkeeping transient is NOT surfaced as the parent task's
|
||
# last_error: the run launched and is still in flight, so the task list
|
||
# must not show an error on an actively running task (matching the
|
||
# success path's clear-on-launch model). The real terminal outcome is
|
||
# written by handle_run_completion.
|
||
assert task_repo.updated[1]["last_error"] is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_both_post_launch_association_writes_can_fail_without_releasing_slot():
|
||
launched: list[dict] = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-live", "thread_id": kwargs["thread_id"]}
|
||
|
||
class FailingTaskRepo(DummyTaskRepo):
|
||
def __init__(self, rows):
|
||
super().__init__(rows)
|
||
self.failures_remaining = 1
|
||
|
||
async def update_after_launch(self, *args, **kwargs):
|
||
if self.failures_remaining:
|
||
self.failures_remaining -= 1
|
||
raise RuntimeError("simulated parent bookkeeping failure")
|
||
await super().update_after_launch(*args, **kwargs)
|
||
|
||
task = {
|
||
"id": "task-double-failure",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "do the thing",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "*/5 * * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
"overlap_policy": "enqueue",
|
||
}
|
||
task_repo = FailingTaskRepo([task])
|
||
run_repo = _StatefulRunRepo(fail_updates=2)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
now = datetime.now(UTC)
|
||
|
||
first = await service.dispatch_task(dict(task), now=now, trigger="scheduled")
|
||
assert first["outcome"] == "launched"
|
||
first_row_id = run_repo.created[0]["run_record_id"]
|
||
assert run_repo.rows[first_row_id]["task_id"] == "task-double-failure"
|
||
assert run_repo.rows[first_row_id]["status"] == "launching"
|
||
assert run_repo.rows[first_row_id]["run_id"] is None
|
||
|
||
second = await service.dispatch_task(dict(task), now=now, trigger="scheduled")
|
||
assert len(launched) == 1
|
||
assert second["outcome"] == "conflict"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_pre_launch_failure_still_releases_active_slot():
|
||
"""Complement to the #4452 fix: when ``_launch_run`` itself fails (no run
|
||
was ever started), the task-run row is marked ``failed`` and the active
|
||
slot is released as before -- the post-launch retention path does not
|
||
apply because there is no live run to protect.
|
||
"""
|
||
launched: list[dict] = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
raise RuntimeError("runtime refused to start the run")
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-4452-pre",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "do the thing",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "*/5 * * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
"overlap_policy": "enqueue",
|
||
}
|
||
]
|
||
)
|
||
run_repo = _StatefulRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(dict(task_repo.rows[0]), now=datetime.now(UTC), trigger="scheduled")
|
||
|
||
assert result["outcome"] == "failed"
|
||
assert result["run_id"] is None
|
||
# launch was attempted (and raised), so exactly one launch attempt, and
|
||
# the row is terminal -> the slot is released for the next dispatch.
|
||
assert len(launched) == 1
|
||
first_row_id = run_repo.created[0]["run_record_id"]
|
||
assert run_repo.rows[first_row_id]["status"] == "failed"
|
||
assert run_repo.rows[first_row_id]["run_id"] is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_malformed_launch_result_still_retains_active_slot():
|
||
"""Defense-in-depth for the #4452 invariant.
|
||
|
||
If ``_launch_run`` returns a malformed result (e.g. missing ``run_id``),
|
||
the unpacking line raises AFTER a live run was already created. The
|
||
dispatch must still take the retention path (keep the row active so the
|
||
slot stays held and no duplicate launches) rather than the pre-launch
|
||
generic-failure path, which would mark the row ``failed`` and release
|
||
the slot while a run is in flight. Keyed off ``launch_succeeded``, not
|
||
``launched_run_id is not None``.
|
||
"""
|
||
launched: list[dict] = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
# Live run started, but the result payload is malformed.
|
||
return {"thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo(
|
||
[
|
||
{
|
||
"id": "task-4452-malformed",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"prompt": "do the thing",
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "*/5 * * * *"},
|
||
"timezone": "UTC",
|
||
"status": "enabled",
|
||
"overlap_policy": "enqueue",
|
||
}
|
||
]
|
||
)
|
||
run_repo = _StatefulRunRepo()
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
now = datetime.now(UTC)
|
||
task = dict(task_repo.rows[0])
|
||
|
||
first = await service.dispatch_task(task, now=now, trigger="scheduled")
|
||
# Launch succeeded, so the outcome is "launched" (a run is in flight)
|
||
# even though the result unpacking raised; run_id is unknown.
|
||
assert first["outcome"] == "launched"
|
||
assert first["run_id"] is None
|
||
|
||
# Second dispatch must observe the active slot still held (row stays in
|
||
# an active status, NOT "failed") and NOT launch a duplicate.
|
||
second = await service.dispatch_task(task, now=now, trigger="scheduled")
|
||
assert len(launched) == 1, launched
|
||
assert second["outcome"] == "conflict", second
|
||
|
||
first_row_id = run_repo.created[0]["run_record_id"]
|
||
assert run_repo.rows[first_row_id]["status"] == "running"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_manual_trigger_is_queued_when_global_budget_exhausted():
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-budget", "thread_id": kwargs["thread_id"]}
|
||
|
||
row = _once_task_row(task_id="task-budget", status="enabled")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "overlap_policy": "enqueue"})
|
||
task_repo = DummyTaskRepo([row])
|
||
# active_count equals max_concurrent_runs → budget is exhausted
|
||
run_repo = DummyRunRepo(active_count=3)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual")
|
||
|
||
assert result["outcome"] == "queued"
|
||
assert launched == []
|
||
assert run_repo.created["status"] == "queued"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_manual_trigger_proceeds_when_global_budget_available():
|
||
"""Manual trigger must launch when active count is below max_concurrent_runs."""
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-ok", "thread_id": kwargs["thread_id"]}
|
||
|
||
row = _once_task_row(task_id="task-ok", status="enabled")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "overlap_policy": "enqueue"})
|
||
task_repo = DummyTaskRepo([row])
|
||
# active_count is 2, max_concurrent_runs is 3 → one slot left
|
||
run_repo = DummyRunRepo(active_count=2)
|
||
service = ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=fake_launch,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual")
|
||
|
||
assert result["outcome"] == "launched"
|
||
assert len(launched) == 1
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Notification enqueue on completion (issue #4254): the hook writes durable
|
||
# outbox rows; a separate delivery worker does the actual IM send.
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class DummyConnectionRepo:
|
||
def __init__(self, connections):
|
||
self.connections = connections
|
||
|
||
async def list_connections(self, owner_user_id):
|
||
return [dict(connection, owner_user_id=owner_user_id) for connection in self.connections]
|
||
|
||
|
||
class DummyNotificationRepo:
|
||
def __init__(self, *, fail=False):
|
||
self.enqueued = []
|
||
self.fail = fail
|
||
|
||
async def enqueue(self, **kwargs):
|
||
if self.fail:
|
||
raise RuntimeError("delivery backend unavailable")
|
||
self.enqueued.append(kwargs)
|
||
return {"id": f"delivery-{len(self.enqueued)}", **kwargs}
|
||
|
||
|
||
def _make_notifying_service(task_repo, run_repo, *, connection_repo, notification_repo):
|
||
return ScheduledTaskService(
|
||
task_repo=task_repo,
|
||
task_run_repo=run_repo,
|
||
launch_run=lambda **_kwargs: None,
|
||
poll_interval_seconds=5,
|
||
lease_seconds=120,
|
||
max_concurrent_runs=3,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_enqueues_run_completed_for_connected_channels():
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert len(notification_repo.enqueued) == 1
|
||
enqueued = notification_repo.enqueued[0]
|
||
assert enqueued["event"] == "run_completed"
|
||
assert enqueued["provider"] == "wecom"
|
||
assert enqueued["target"] == "GaoZhiChao"
|
||
assert enqueued["task_id"] == "task-once"
|
||
assert enqueued["task_run_id"] == "task-run-x"
|
||
assert enqueued["run_id"] == "run-x"
|
||
assert enqueued["owner_user_id"] == "user-1"
|
||
assert enqueued["payload"]["run_status"] == "success"
|
||
assert enqueued["payload"]["error"] is None
|
||
assert enqueued["payload"]["task_title"] == "Once summary"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_enqueues_run_failed_event_on_error():
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.error, error="boom"))
|
||
|
||
assert len(notification_repo.enqueued) == 1
|
||
enqueued = notification_repo.enqueued[0]
|
||
assert enqueued["event"] == "run_failed"
|
||
assert enqueued["payload"]["run_status"] == "failed"
|
||
assert enqueued["payload"]["error"] == "boom"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_does_not_notify_for_interrupted_runs():
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.interrupted))
|
||
|
||
# An interrupt is a user-initiated cancel; the user is already aware.
|
||
assert notification_repo.enqueued == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_does_not_notify_for_manual_triggers():
|
||
"""A manual "run now" happens with the user watching the UI; pushing the
|
||
same outcome to IM is noise, so only scheduled firings notify."""
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success, trigger="manual"))
|
||
|
||
# Execution status is still recorded; only the IM push is suppressed.
|
||
assert task_repo.completions[-1][1]["status"] == "success"
|
||
assert notification_repo.enqueued == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_still_notifies_when_trigger_metadata_is_missing():
|
||
"""Fail safe: an untagged completion (older rows, unexpected paths) keeps
|
||
notifying -- dropping a push silently is worse than a rare extra one."""
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
record = _completion_record(RunStatus.success)
|
||
record.metadata.pop("scheduled_trigger")
|
||
await service.handle_run_completion(record)
|
||
|
||
assert len(notification_repo.enqueued) == 1
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_skips_unusable_connections():
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "revoked-user", "status": "revoked"},
|
||
{"provider": "wecom", "external_account_id": None, "status": "connected"},
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert [item["target"] for item in notification_repo.enqueued] == ["GaoZhiChao"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_notification_failure_does_not_break_status_writes():
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
run_repo = DummyRunRepo()
|
||
connection_repo = DummyConnectionRepo(
|
||
[
|
||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||
]
|
||
)
|
||
notification_repo = DummyNotificationRepo(fail=True)
|
||
service = _make_notifying_service(
|
||
task_repo,
|
||
run_repo,
|
||
connection_repo=connection_repo,
|
||
notification_repo=notification_repo,
|
||
)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
# Execution status is written regardless: delivery is best-effort and must
|
||
# never shadow the run outcome (execution/delivery status separation).
|
||
assert task_repo.completions[-1][1]["status"] == "success"
|
||
|
||
|
||
def _notifying_service_with_wecom(task_repo, notification_repo):
|
||
connection_repo = DummyConnectionRepo([{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"}])
|
||
return _make_notifying_service(task_repo, DummyRunRepo(), connection_repo=connection_repo, notification_repo=notification_repo)
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_detached_outbox_stops_the_enqueue():
|
||
# The Gateway detaches the outbox when the delivery side cannot start.
|
||
task_repo = DummyTaskRepo([_once_task_row()])
|
||
notification_repo = DummyNotificationRepo()
|
||
service = _notifying_service_with_wecom(task_repo, notification_repo)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
assert [row["event"] for row in notification_repo.enqueued] == ["run_completed"]
|
||
|
||
service.detach_notification_outbox()
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert len(notification_repo.enqueued) == 1
|
||
assert len(task_repo.completions) == 2
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_that_was_not_recorded_does_not_notify():
|
||
# complete_run returns False for a missing occurrence, another run's occurrence or another owner.
|
||
class RejectingTaskRepo(DummyTaskRepo):
|
||
async def complete_run(self, task_id, **kwargs):
|
||
await super().complete_run(task_id, **kwargs)
|
||
return False
|
||
|
||
task_repo = RejectingTaskRepo([_once_task_row()])
|
||
notification_repo = DummyNotificationRepo()
|
||
|
||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert len(task_repo.completions) == 1
|
||
assert notification_repo.enqueued == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_for_a_task_deleted_meanwhile_does_not_notify():
|
||
task_repo = DummyTaskRepo([])
|
||
notification_repo = DummyNotificationRepo()
|
||
|
||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert task_repo.completions[-1][1]["status"] == "success"
|
||
assert notification_repo.enqueued == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_still_notifies_when_the_task_read_fails():
|
||
class UnreadableTaskRepo(DummyTaskRepo):
|
||
async def get(self, task_id, *, user_id):
|
||
raise RuntimeError("database unavailable")
|
||
|
||
task_repo = UnreadableTaskRepo([_once_task_row()])
|
||
notification_repo = DummyNotificationRepo()
|
||
|
||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert task_repo.completions[-1][1]["status"] == "success"
|
||
# The title is optional: the push still goes out and falls back to the task id.
|
||
assert [row["payload"]["task_title"] for row in notification_repo.enqueued] == [None]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_completion_without_the_outbox_does_not_read_the_task():
|
||
class CountingTaskRepo(DummyTaskRepo):
|
||
reads = 0
|
||
|
||
async def get(self, task_id, *, user_id):
|
||
self.reads += 1
|
||
return await super().get(task_id, user_id=user_id)
|
||
|
||
task_repo = CountingTaskRepo([_once_task_row()])
|
||
service = _make_notifying_service(task_repo, DummyRunRepo(), connection_repo=None, notification_repo=None)
|
||
|
||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||
|
||
assert task_repo.completions[-1][1]["status"] == "success"
|
||
assert task_repo.reads == 0
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_trigger_while_a_scheduled_occurrence_is_queued_reports_the_existing_row():
|
||
launched = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launched.append(kwargs)
|
||
return {"run_id": "run-x", "thread_id": kwargs["thread_id"]}
|
||
|
||
class QueuedRunRepo(DummyRunRepo):
|
||
async def get_active_run(self, task_id):
|
||
return {"id": "task-run-waiting", "task_id": task_id, "thread_id": "thread-waiting", "status": "queued", "trigger": "scheduled"}
|
||
|
||
row = _once_task_row(task_id="task-waiting")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "0 9 * * *"}, "status": "enabled"})
|
||
run_repo = QueuedRunRepo()
|
||
service = ScheduledTaskService(task_repo=DummyTaskRepo([row]), task_run_repo=run_repo, launch_run=fake_launch, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual")
|
||
|
||
assert result["outcome"] == "queued"
|
||
assert result["existing"] is True
|
||
assert result["task_run_id"] == "task-run-waiting"
|
||
assert result["thread_id"] == "thread-waiting"
|
||
assert run_repo.created is None
|
||
assert launched == []
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_a_new_queued_trial_is_not_reported_as_existing():
|
||
async def fake_launch(**kwargs):
|
||
raise AssertionError("budget is exhausted; nothing launches")
|
||
|
||
row = _once_task_row(task_id="task-budget")
|
||
row.update({"schedule_type": "cron", "schedule_spec": {"cron": "0 9 * * *"}, "status": "enabled"})
|
||
run_repo = DummyRunRepo(active_count=3)
|
||
service = ScheduledTaskService(task_repo=DummyTaskRepo([row]), task_run_repo=run_repo, launch_run=fake_launch, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3)
|
||
|
||
result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual")
|
||
|
||
assert result["outcome"] == "queued"
|
||
assert result["existing"] is False
|
||
assert run_repo.created is not None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_is_running_reflects_the_poller_task():
|
||
service = ScheduledTaskService(task_repo=DummyTaskRepo([]), task_run_repo=DummyRunRepo(), launch_run=None, poll_interval_seconds=60, lease_seconds=120, max_concurrent_runs=1)
|
||
assert service.is_running is False
|
||
await service.start()
|
||
try:
|
||
assert service.is_running is True
|
||
finally:
|
||
await service.stop()
|
||
assert service.is_running is False
|
||
|
||
|
||
class NumberedRunRepo(DummyRunRepo):
|
||
def __init__(self, number):
|
||
super().__init__()
|
||
self.number = number
|
||
self.number_calls = []
|
||
|
||
async def run_number(self, task_run_id):
|
||
self.number_calls.append(task_run_id)
|
||
return self.number
|
||
|
||
|
||
def _provenance_task(**updates):
|
||
return {
|
||
"id": "task-prov",
|
||
"user_id": "user-1",
|
||
"thread_id": None,
|
||
"context_mode": "fresh_thread_per_run",
|
||
"assistant_id": "lead_agent",
|
||
"title": "检查发布清单",
|
||
"prompt": "检查 release-checklist.md,列出没勾的项",
|
||
"stop_condition": "清单全部勾完",
|
||
"standing_notes": ["用 develop 分支"],
|
||
"schedule_type": "cron",
|
||
"schedule_spec": {"cron": "0 9 * * 1-5"},
|
||
"timezone": "Asia/Shanghai",
|
||
"status": "enabled",
|
||
**updates,
|
||
}
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
@pytest.mark.parametrize("trigger", ["scheduled", "manual"])
|
||
async def test_launch_passes_user_language_origin_and_run_thread_title(trigger):
|
||
launches = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launches.append(kwargs)
|
||
return {"run_id": "run-prov", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo([_provenance_task()])
|
||
run_repo = NumberedRunRepo(4)
|
||
service = ScheduledTaskService(task_repo=task_repo, task_run_repo=run_repo, launch_run=fake_launch, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3, own_stop_available=True)
|
||
now = datetime(2026, 10, 7, 1, 0, tzinfo=UTC)
|
||
|
||
await service.dispatch_task(task_repo.rows[0], now=now, trigger=trigger)
|
||
|
||
(launch,) = launches
|
||
origin = launch["origin"]
|
||
task_run_id = run_repo.created["run_record_id"]
|
||
assert origin == {
|
||
"task_id": "task-prov",
|
||
"task_run_id": task_run_id,
|
||
"trigger": trigger,
|
||
"run_number": 4 if trigger == "scheduled" else None,
|
||
"scheduled_for": now.isoformat(),
|
||
"timezone": "Asia/Shanghai",
|
||
"schedule_type": "cron",
|
||
"task_title": "检查发布清单",
|
||
"instructions": "检查 release-checklist.md,列出没勾的项",
|
||
"stop_condition": "清单全部勾完",
|
||
"standing_notes": ["用 develop 分支"],
|
||
}
|
||
assert run_repo.number_calls == ([task_run_id] if trigger == "scheduled" else [])
|
||
# The launched text carries the host-written stop rule and notes block; the
|
||
# origin keeps only the user's own words.
|
||
assert launch["prompt"].startswith(origin["instructions"])
|
||
assert "stop_scheduled_task" in launch["prompt"] and "<standing_notes>" in launch["prompt"]
|
||
assert launch["title"] == "检查发布清单 · 10-07 09:00"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
@pytest.mark.parametrize(
|
||
("updates", "trigger", "expected"),
|
||
[
|
||
({"schedule_type": "interval", "schedule_spec": {"every_seconds": 3600}, "timezone": "UTC"}, "scheduled", "检查发布清单 · #4"),
|
||
({"schedule_type": "interval", "schedule_spec": {"every_seconds": 3600}, "timezone": "UTC"}, "manual", "检查发布清单"),
|
||
({"schedule_type": "once", "schedule_spec": {"run_at": "2026-10-07T01:00:00+00:00"}, "timezone": "UTC"}, "scheduled", "检查发布清单 · #4"),
|
||
({"schedule_type": "interval", "schedule_spec": {"every_seconds": 3600}, "timezone": "Asia/Shanghai"}, "scheduled", "检查发布清单 · 10-07 09:00"),
|
||
({"schedule_type": "cron", "schedule_spec": {"cron": "0 1 * * *"}, "timezone": "UTC"}, "scheduled", "检查发布清单 · 10-07 01:00"),
|
||
],
|
||
ids=["interval-placeholder", "interval-placeholder-trial", "once-offset-placeholder", "interval-real-zone", "cron-real-utc"],
|
||
)
|
||
async def test_run_thread_title_never_shows_a_placeholder_utc_time(updates, trigger, expected):
|
||
launches = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launches.append(kwargs)
|
||
return {"run_id": "run-prov", "thread_id": kwargs["thread_id"]}
|
||
|
||
task_repo = DummyTaskRepo([_provenance_task(**updates)])
|
||
service = ScheduledTaskService(task_repo=task_repo, task_run_repo=NumberedRunRepo(4), launch_run=fake_launch, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3, own_stop_available=True)
|
||
await service.dispatch_task(task_repo.rows[0], now=datetime(2026, 10, 7, 1, 0, tzinfo=UTC), trigger=trigger)
|
||
(launch,) = launches
|
||
assert launch["title"] == expected
|
||
assert launch["origin"]["schedule_type"] == updates["schedule_type"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_reuse_thread_launch_has_no_title_and_survives_a_missing_run_number():
|
||
launches = []
|
||
|
||
async def fake_launch(**kwargs):
|
||
launches.append(kwargs)
|
||
return {"run_id": "run-prov", "thread_id": kwargs["thread_id"]}
|
||
|
||
class BrokenNumberRepo(DummyRunRepo):
|
||
async def run_number(self, task_run_id):
|
||
raise RuntimeError("database unavailable")
|
||
|
||
task_repo = DummyTaskRepo([_provenance_task(context_mode="reuse_thread", thread_id="thread-1", stop_condition=None, standing_notes=None)])
|
||
service = ScheduledTaskService(task_repo=task_repo, task_run_repo=BrokenNumberRepo(), launch_run=fake_launch, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3)
|
||
|
||
result = await service.dispatch_task(task_repo.rows[0], now=datetime(2026, 10, 7, 1, 0, tzinfo=UTC), trigger="scheduled")
|
||
|
||
assert result["outcome"] == "launched"
|
||
(launch,) = launches
|
||
assert launch["title"] is None
|
||
assert launch["origin"]["run_number"] is None
|
||
assert (launch["origin"]["stop_condition"], launch["origin"]["standing_notes"]) == (None, [])
|