rayhpeng
0d86a0d8fe
fix(schedule): stop the completion hook rolling back the launch write
...
A cron task could become permanently unschedulable after a run that failed
fast. Reproduced against the real SQL adapters, and against the legacy path
for contrast -- the legacy code does not have this bug, so the hexagonal
slice introduced it.
The window
----------
`dispatch_task` already documents it: "a fast-failing run can reach
handle_run_completion before this write lands". The hook therefore reads the
task while the dispatch path has not yet written its bookkeeping, and the
snapshot it holds still carries the elapsed `next_run_at` the claim was made
on. `handle_run_completion` then wrote that whole snapshot back through
`save()`, so whichever landed second undid the other.
For a cron task the result is terminal in the worst way. `record_launch`
writes the fresh fire time, bumps `run_count` and clears the claim; the
completion's whole-aggregate write restores the elapsed fire time, rolls
`run_count` back and leaves `status='running'` with `lease_expires_at IS
NULL`. Neither `claim_due` branch matches that shape (one needs `enabled`,
the other needs an expired claim), and `cancel_stuck_once_tasks` only sweeps
`once` rows -- so nothing can ever reach the task again.
The legacy `app/scheduler/service.py` wrote the same outcome field by field
(`update(..., updates={"last_error": ...})`) and never touched scheduling
state, which is why it survives the same interleaving.
The fix
-------
`record_completion` joins `record_launch` as a second deliberately narrow
port method, for the same stated reason `record_launch` is not expressed as
`save(task)`: the two race, so neither may write through the aggregate. They
now own disjoint fields -- the launch owns the schedule, the completion owns
the verdict (terminal status plus `last_error`, with `None` meaning "do not
move the status", i.e. every cron task).
`save()` keeps its documented purpose, the user-initiated whole-aggregate
updates (`update_task` / `pause` / `resume`). `handle_run_completion` still
reads the task first, but only to ask `status_after_completion`, which reads
nothing but the schedule type -- immutable, so that read carries no
time-of-check risk.
Tests
-----
The contract suite missed this because it groups cases by port method:
`record_launch` appears only among its own, never interleaved with `save`,
and `test_protect_terminal_keeps_a_concurrently_finalized_verdict` pins the
mirror-image direction only. The in-memory double replaces the whole row too,
so both implementations were faithfully wrong -- the defect was in the
contract, not either adapter.
Added: `TestRecordCompletion` (5 cases, both implementations), including the
interleaving itself; and `test_a_cron_task_survives_a_launch_write_landing_mid_completion`,
which drives a real `ScheduleService` over a repo double that commits
`record_launch` between the hook's read and its write. Both were watched
failing first -- the service case failing with the actual symptom, a rolled
back `next_run_at`, not a missing method.
Verified end to end against file-backed sqlite with the real service: the
task keeps `enabled`, the concurrent fire time and run count survive, the
verdict is recorded, and the next poll still claims it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:46:16 +08:00
..
2026-07-27 14:19:39 +08:00
2026-07-22 14:02:56 +08:00
2026-07-19 18:26:26 +08:00
2026-07-22 13:55:40 +08:00
2026-05-12 23:18:54 +08:00
2026-07-26 14:43:08 +08:00
2026-04-26 15:09:25 +08:00
2026-06-01 15:50:39 +08:00
2026-07-28 07:13:25 +08:00
2026-07-27 10:58:45 +08:00
2026-07-21 11:46:33 +08:00
2026-06-21 20:56:42 +08:00
2026-07-28 19:46:16 +08:00
2026-06-08 12:35:03 +08:00
2026-07-22 14:47:08 +08:00
2026-07-17 15:11:15 +08:00
2026-07-22 13:44:55 +08:00
2026-07-23 08:03:21 +08:00
2026-07-23 08:03:21 +08:00
2026-07-22 13:44:55 +08:00
2026-07-21 09:09:40 +08:00
2026-07-26 09:47:57 +08:00
2026-07-27 07:47:39 +08:00
2026-07-27 07:47:39 +08:00
2026-06-23 10:39:40 +08:00
2026-07-22 08:33:29 +08:00
2026-07-19 07:31:29 +08:00
2026-05-29 23:05:59 +08:00
2026-05-16 09:24:40 +08:00
2026-04-26 11:08:11 +08:00
2026-07-27 14:19:04 +08:00
2026-07-18 22:17:15 +08:00
2026-07-26 09:52:16 +08:00
2026-07-23 22:51:35 +08:00
2026-07-17 14:49:51 +08:00
2026-07-17 14:49:51 +08:00
2026-07-27 14:19:04 +08:00
2026-07-23 22:51:35 +08:00
2026-07-23 22:51:35 +08:00
2026-06-24 13:57:12 +08:00
2026-07-27 11:47:49 +08:00
2026-07-27 11:47:49 +08:00
2026-07-27 11:47:49 +08:00
2026-07-28 07:49:13 +08:00
2026-07-26 08:00:48 +08:00
2026-07-02 11:00:49 +08:00
2026-07-21 11:46:33 +08:00
2026-07-21 11:46:33 +08:00
2026-07-28 07:56:04 +08:00
2026-07-14 07:37:59 +08:00
2026-06-16 23:04:39 +08:00
2026-06-18 10:15:31 +08:00
2026-07-12 08:54:03 +08:00
2026-07-25 21:55:31 +08:00
2026-07-17 14:49:51 +08:00
2026-06-18 10:45:33 +08:00
2026-07-27 23:07:40 +08:00
2026-04-14 10:29:44 +08:00
2026-07-27 11:47:49 +08:00
2026-07-26 10:44:50 +08:00
2026-07-22 08:33:29 +08:00
2026-07-12 08:15:32 +08:00
2026-07-27 11:47:49 +08:00
2026-07-27 14:05:31 +08:00
2026-03-30 07:41:18 +08:00
2026-04-25 19:40:06 +08:00
2026-04-07 18:21:22 +08:00
2026-07-22 14:02:56 +08:00
2026-07-23 22:51:35 +08:00
2026-03-29 21:03:58 +08:00
2026-05-04 09:56:16 +08:00
2026-07-27 11:47:49 +08:00
2026-05-02 15:04:11 +08:00
2026-06-10 21:36:25 +08:00
2026-07-28 19:09:22 +08:00
2026-07-22 08:33:29 +08:00
2026-07-25 23:18:34 +08:00
2026-07-26 07:39:39 +08:00
2026-04-26 11:05:47 +08:00
2026-07-05 09:54:55 +08:00
2026-03-29 15:31:18 +08:00
2026-07-22 08:33:29 +08:00
2026-05-07 16:15:15 +08:00
2026-06-12 15:24:58 +08:00
2026-07-22 07:46:03 +08:00
2026-07-23 22:56:12 +08:00
2026-07-17 19:16:42 +08:00
2026-06-08 07:59:50 +08:00
2026-07-26 21:16:36 +08:00
2026-07-13 17:57:34 +08:00
2026-06-02 22:43:22 +08:00
2026-06-05 15:21:41 +08:00
2026-07-12 00:17:11 +08:00
2026-07-04 23:09:29 +08:00
2026-06-02 22:43:22 +08:00
2026-07-01 22:49:17 +08:00
2026-07-13 19:26:07 +08:00
2026-07-22 08:33:29 +08:00
2026-07-27 11:47:49 +08:00
2026-07-04 09:21:19 +08:00
2026-07-22 13:55:40 +08:00
2026-07-04 21:31:01 +08:00
2026-07-26 09:56:17 +08:00
2026-06-12 17:16:01 +08:00
2026-06-09 15:29:40 +08:00
2026-07-27 14:19:39 +08:00
2026-06-12 15:24:58 +08:00
2026-06-13 23:27:17 +08:00
2026-03-31 22:19:27 +08:00
2026-07-02 11:00:49 +08:00
2026-07-26 07:39:39 +08:00
2026-07-16 09:39:09 +08:00
2026-07-28 07:49:13 +08:00
2026-04-26 11:09:56 +08:00
2026-04-08 17:13:39 +08:00
2026-07-05 09:54:55 +08:00
2026-07-21 11:46:33 +08:00
2026-07-23 15:30:33 +08:00
2026-07-27 10:58:45 +08:00
2026-07-28 17:34:10 +08:00
2026-07-17 11:18:20 +08:00
2026-07-02 08:24:15 +08:00
2026-07-04 21:31:01 +08:00
2026-07-19 22:12:30 +08:00
2026-07-14 11:57:48 +08:00
2026-07-25 23:18:34 +08:00
2026-07-22 08:33:29 +08:00
2026-05-11 17:38:37 +08:00
2026-07-26 08:09:17 +08:00
2026-07-27 23:17:18 +08:00
2026-07-28 19:09:22 +08:00
2026-07-28 19:09:22 +08:00
2026-07-18 17:50:20 +08:00
2026-07-27 07:13:06 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-27 23:07:40 +08:00
2026-07-04 22:56:24 +08:00
2026-07-14 22:56:50 +08:00
2026-07-04 22:56:24 +08:00
2026-07-04 22:56:24 +08:00
2026-07-16 09:12:29 +08:00
2026-07-23 14:32:49 +08:00
2026-07-15 22:30:34 +08:00
2026-07-25 23:50:21 +08:00
2026-06-21 15:55:10 +08:00
2026-07-23 21:33:57 +08:00
2026-03-14 22:55:52 +08:00
2026-07-23 15:30:33 +08:00
2026-07-06 07:21:12 +08:00
2026-07-27 22:46:51 +08:00
2026-07-27 14:05:31 +08:00
2026-07-10 11:08:30 +08:00
2026-07-27 23:07:40 +08:00
2026-03-26 14:20:18 +08:00
2026-06-28 16:10:44 +08:00
2026-07-08 17:10:27 +08:00
2026-07-22 14:02:56 +08:00
2026-07-04 13:54:04 +08:00
2026-06-17 15:29:22 +08:00
2026-07-22 14:47:08 +08:00
2026-06-08 23:25:29 +08:00
2026-07-12 11:51:31 +08:00
2026-06-10 16:11:00 +08:00
2026-07-26 08:09:17 +08:00
2026-07-26 14:43:08 +08:00
2026-07-24 21:56:11 +08:00
2026-07-27 11:47:49 +08:00
2026-07-23 08:22:15 +08:00
2026-07-23 22:56:12 +08:00
2026-04-11 16:52:10 +08:00
2026-07-20 23:33:09 +08:00
2026-07-09 17:21:25 +08:00
2026-07-03 07:51:22 +08:00
2026-07-13 00:01:22 +08:00
2026-07-26 08:09:17 +08:00
2026-07-13 00:01:22 +08:00
2026-07-04 13:54:04 +08:00
2026-07-03 08:01:46 +08:00
2026-04-30 22:27:14 +08:00
2026-05-07 16:15:15 +08:00
2026-07-19 17:13:44 +08:00
2026-07-26 14:43:08 +08:00
2026-07-19 22:12:30 +08:00
2026-07-03 15:39:12 +08:00
2026-07-09 16:26:31 +08:00
2026-04-25 09:18:13 +08:00
2026-07-25 21:43:33 +08:00
2026-07-22 19:58:43 +08:00
2026-07-10 07:54:36 +08:00
2026-07-09 16:26:31 +08:00
2026-07-09 16:26:31 +08:00
2026-07-14 09:49:43 +08:00
2026-07-04 09:40:05 +08:00
2026-06-25 22:57:00 +08:00
2026-07-14 09:49:43 +08:00
2026-07-26 21:16:36 +08:00
2026-07-22 14:40:57 +08:00
2026-07-26 21:16:36 +08:00
2026-07-22 07:46:03 +08:00
2026-07-15 11:21:04 +08:00
2026-07-26 21:16:36 +08:00
2026-07-26 21:16:36 +08:00
2026-07-27 23:17:18 +08:00
2026-07-22 14:40:57 +08:00
2026-07-15 11:21:04 +08:00
2026-07-26 21:16:36 +08:00
2026-07-22 07:46:03 +08:00
2026-07-22 07:46:03 +08:00
2026-07-22 07:46:03 +08:00
2026-04-26 11:09:55 +08:00
2026-07-23 22:51:35 +08:00
2026-07-15 11:21:04 +08:00
2026-07-22 14:40:57 +08:00
2026-07-22 14:02:56 +08:00
2026-07-26 21:16:36 +08:00
2026-07-26 21:16:36 +08:00
2026-07-28 07:13:25 +08:00
2026-07-28 07:13:25 +08:00
2026-07-27 23:07:40 +08:00
2026-07-04 13:54:04 +08:00
2026-07-17 14:23:44 +08:00
2026-03-22 20:39:26 +08:00
2026-07-22 13:44:55 +08:00
2026-07-26 14:43:08 +08:00
2026-07-26 14:43:08 +08:00
2026-07-24 22:41:07 +08:00
2026-07-27 23:07:40 +08:00
2026-07-26 21:45:47 +08:00
2026-07-27 07:25:34 +08:00
2026-06-22 18:55:20 +08:00
2026-07-18 17:48:42 +08:00
2026-06-21 15:47:53 +08:00
2026-05-28 08:20:52 +08:00
2026-07-28 17:34:10 +08:00
2026-07-04 21:44:22 +08:00
2026-05-28 18:24:32 +08:00
2026-06-08 22:04:38 +08:00
2026-03-26 15:07:05 +08:00
2026-06-09 18:01:43 +08:00
2026-07-26 08:09:17 +08:00
2026-06-24 13:57:12 +08:00
2026-07-28 07:13:25 +08:00
2026-06-24 13:57:12 +08:00
2026-07-28 07:13:25 +08:00
2026-06-24 13:57:12 +08:00
2026-06-24 13:57:12 +08:00
2026-07-28 07:13:25 +08:00
2026-07-21 08:52:31 +08:00
2026-06-24 13:57:12 +08:00
2026-07-19 17:01:28 +08:00
2026-07-28 17:34:10 +08:00
2026-04-26 11:13:01 +08:00
2026-07-20 07:24:34 +08:00
2026-04-10 20:40:30 +08:00
2026-07-26 08:09:17 +08:00
2026-07-26 08:09:17 +08:00
2026-07-14 07:51:59 +08:00
2026-07-21 09:23:14 +08:00
2026-07-06 20:57:41 +08:00
2026-07-11 23:20:53 +08:00
2026-06-17 21:11:44 +08:00
2026-03-14 22:55:52 +08:00
2026-03-14 22:55:52 +08:00
2026-07-03 08:01:46 +08:00
2026-07-26 08:09:17 +08:00
2026-06-08 17:32:41 +08:00
2026-06-09 21:58:31 +08:00
2026-07-14 23:08:33 +08:00
2026-07-11 15:58:07 +08:00
2026-07-25 23:50:21 +08:00
2026-06-24 17:47:04 +08:00
2026-07-02 07:43:09 +08:00
2026-04-26 15:09:25 +08:00
2026-07-26 21:45:47 +08:00
2026-07-23 21:33:57 +08:00
2026-07-16 09:39:09 +08:00
2026-07-26 21:45:47 +08:00
2026-07-26 21:57:39 +08:00
2026-05-21 14:48:28 +08:00
2026-07-27 07:25:34 +08:00
2026-07-27 22:27:16 +08:00
2026-07-27 22:27:16 +08:00
2026-07-27 11:32:10 +08:00
2026-07-27 22:46:51 +08:00
2026-07-23 15:30:48 +08:00
2026-06-24 10:24:36 +08:00
2026-07-22 08:33:29 +08:00
2026-07-19 22:12:30 +08:00
2026-07-23 22:56:12 +08:00
2026-07-23 22:56:12 +08:00
2026-05-22 21:20:28 +08:00
2026-06-25 22:43:28 +08:00
2026-06-03 22:02:27 +08:00
2026-07-26 09:02:40 +08:00
2026-04-09 17:21:23 +08:00
2026-07-26 09:47:57 +08:00
2026-07-21 09:09:40 +08:00
2026-07-13 18:34:41 +08:00
2026-06-25 08:06:17 +08:00
2026-07-28 07:49:13 +08:00
2026-07-26 08:09:17 +08:00
2026-07-04 13:54:04 +08:00
2026-06-12 10:20:38 +08:00
2026-07-28 12:08:17 +08:00
2026-07-28 19:46:16 +08:00
2026-07-28 18:28:46 +08:00
2026-07-28 19:09:22 +08:00
2026-07-28 19:09:22 +08:00
2026-07-28 18:28:46 +08:00
2026-07-28 18:28:46 +08:00
2026-07-28 19:46:16 +08:00
2026-07-28 19:09:22 +08:00
2026-07-28 18:28:46 +08:00
2026-07-04 21:51:57 +08:00
2026-07-24 21:41:09 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-24 21:41:09 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-04 21:51:57 +08:00
2026-07-24 21:41:09 +08:00
2026-07-28 11:20:31 +08:00
2026-06-12 09:45:26 +08:00
2026-07-21 23:41:07 +08:00
2026-06-19 11:23:07 +08:00
2026-03-26 14:20:18 +08:00
2026-07-21 08:15:33 +08:00
2026-06-27 23:30:54 +08:00
2026-07-07 23:09:06 +08:00
2026-07-23 08:03:21 +08:00
2026-05-12 23:18:54 +08:00
2026-07-23 08:03:21 +08:00
2026-07-05 19:55:39 +08:00
2026-06-19 21:57:03 +08:00
2026-07-04 23:09:29 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 11:27:19 +08:00
2026-07-04 23:09:29 +08:00
2026-07-20 08:18:23 +08:00
2026-07-13 10:40:22 +08:00
2026-05-28 15:48:32 +08:00
2026-07-17 14:39:35 +08:00
2026-07-23 21:33:57 +08:00
2026-07-24 21:25:53 +08:00
2026-07-11 15:58:07 +08:00
2026-07-04 13:54:04 +08:00
2026-07-16 14:12:02 +08:00
2026-03-25 16:28:33 +08:00
2026-07-16 14:12:02 +08:00
2026-07-07 21:44:28 +08:00
2026-07-19 22:36:15 +08:00
2026-07-16 14:12:02 +08:00
2026-07-04 23:09:29 +08:00
2026-07-17 23:22:16 +08:00
2026-07-21 09:39:44 +08:00
2026-07-24 21:25:53 +08:00
2026-07-21 00:02:16 +08:00
2026-06-18 10:09:46 +08:00
2026-07-08 21:58:33 +08:00
2026-07-23 21:33:57 +08:00
2026-07-13 19:33:40 +08:00
2026-03-30 16:02:23 +08:00
2026-06-12 15:24:58 +08:00
2026-07-22 09:20:54 +08:00
2026-07-27 07:13:06 +08:00
2026-06-14 10:30:45 +08:00
2026-06-08 23:17:22 +08:00
2026-07-14 08:14:21 +08:00
2026-07-23 22:51:35 +08:00
2026-07-24 21:56:11 +08:00
2026-07-14 22:51:21 +08:00
2026-05-02 06:37:49 +08:00
2026-07-11 15:41:57 +08:00
2026-07-11 15:41:57 +08:00
2026-07-13 19:26:07 +08:00
2026-06-19 21:42:42 +08:00
2026-07-08 17:10:27 +08:00
2026-07-26 07:39:39 +08:00
2026-07-07 19:55:33 +08:00
2026-07-27 11:47:49 +08:00
2026-07-27 11:47:49 +08:00
2026-07-22 08:33:29 +08:00
2026-06-24 07:28:17 +08:00
2026-07-23 22:56:12 +08:00
2026-05-21 21:18:10 +08:00
2026-06-18 10:15:31 +08:00
2026-07-28 07:49:13 +08:00
2026-07-11 15:53:14 +08:00
2026-07-10 22:13:50 +08:00
2026-07-28 07:13:25 +08:00
2026-07-28 07:13:25 +08:00
2026-07-27 22:18:02 +08:00
2026-07-27 22:46:51 +08:00
2026-07-04 09:21:19 +08:00
2026-07-27 22:46:51 +08:00
2026-06-02 22:43:22 +08:00
2026-07-22 08:33:29 +08:00
2026-05-22 21:42:14 +08:00
2026-07-22 08:33:29 +08:00
2026-07-26 20:47:58 +08:00
2026-07-08 14:41:16 +08:00
2026-07-15 11:21:04 +08:00
2026-03-14 22:55:52 +08:00
2026-07-11 23:15:03 +08:00
2026-06-12 22:48:47 +08:00
2026-07-08 22:26:06 +08:00
2026-07-04 23:14:46 +08:00
2026-05-10 22:00:57 +08:00
2026-06-23 23:37:41 +08:00
2026-07-27 11:47:49 +08:00
2026-07-27 22:46:51 +08:00
2026-05-19 22:11:46 +08:00
2026-07-26 07:39:39 +08:00
2026-07-04 11:27:19 +08:00
2026-07-16 16:41:04 +08:00
2026-04-06 15:09:57 +08:00
2026-07-06 20:57:41 +08:00
2026-07-06 20:57:41 +08:00
2026-07-12 11:24:24 +08:00
2026-07-14 09:49:43 +08:00
2026-07-19 09:13:51 +08:00
2026-07-19 09:13:51 +08:00
2026-07-14 08:58:06 +08:00
2026-07-14 08:58:06 +08:00
2026-07-03 08:01:46 +08:00
2026-07-22 07:53:50 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-22 07:53:50 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-06-25 20:10:49 +08:00
2026-07-15 22:27:00 +08:00
2026-06-25 20:10:49 +08:00
2026-07-20 23:56:37 +08:00
2026-05-12 23:18:54 +08:00
2026-07-23 08:03:21 +08:00
2026-07-26 21:16:36 +08:00
2026-07-26 21:16:36 +08:00
2026-07-01 16:43:56 +08:00
2026-07-22 14:02:56 +08:00
2026-07-19 17:24:31 +08:00
2026-04-26 15:09:25 +08:00
2026-07-26 08:09:17 +08:00
2026-07-11 09:18:12 +08:00
2026-07-14 10:43:13 +08:00
2026-05-02 15:16:16 +08:00
2026-06-09 15:29:40 +08:00
2026-07-22 08:41:50 +08:00
2026-07-24 13:12:43 +08:00
2026-04-06 15:18:34 +08:00
2026-07-27 07:13:06 +08:00
2026-07-06 07:21:12 +08:00
2026-07-20 23:05:47 +08:00
2026-07-18 17:53:30 +08:00
2026-07-12 07:52:45 +08:00
2026-07-25 23:50:21 +08:00
2026-07-25 23:50:21 +08:00
2026-07-12 11:51:31 +08:00
2026-07-27 22:27:16 +08:00
2026-06-07 17:47:11 +08:00