* fix(scheduler): isolate queued run drain failures * fix(scheduler): isolate queue timeout rows * fix(scheduler): enforce queue timeout at launch claim Keep expired queued occurrences ineligible to start when the timeout sweep fails. Cover per-row and whole-scan cleanup failures with SQLite service tests that also verify healthy queued and due work still launches. * fix(scheduler): exclude expired rows from queue selection Filter rows past the queue deadline before owner ranking and the fair batch limit so failed timeout cleanup cannot starve healthy waiting work. Keep the atomic claim cutoff as the final launch guard. --------- Co-authored-by: AochenShen99 <sac1063467113@hotmail.com> Co-authored-by: Nan Gao <thunderggnn@gmail.com>
1.5 KiB
Scheduler failure isolation
Scheduled work shares one polling cycle. Treat an individual occurrence as the failure boundary so one damaged record cannot repeatedly prevent unrelated owners' work from progressing. Apply this convention to maintenance passes as well as launches: failing to clean up an older occurrence is not a reason to withhold admission from a healthy task.
An exception during dispatch does not establish whether Gateway accepted the run. Leave that uncertainty to durable queue and lease recovery. Eagerly clearing a lease or starting a replacement from the exception handler can duplicate work that already started. Cleanup is appropriate only when the operation's outcome and ownership are known.
Containment preserves progress, not success. Keep failures observable and let normal recovery retry the affected occurrence. Apply queue age before building the fair drain batch: expired rows must not consume owner ranks or batch slots, or a failed timeout sweep can starve healthy work indefinitely. Queue timeout is enforced again when a waiting row is claimed for launch. If timeout cleanup fails, the row remains queued for the next recovery pass but is no longer eligible to start; cleanup failure must never turn expired work into a live run. Failure to read the queue or claim due work still prevents that stage from progressing; the next poll retries it.
For user-visible behavior and deployment requirements, see the scheduled tasks guide.