* fix(mcp): consume resolved notification snapshots instead of launching empty events
A tracking-degraded snapshot recorded by a failed poll can resolve before any
notification worker claims it: the next poll succeeds, or an answered
input_required prompt returns the task to working. _record_event_if_changed
left that pending snapshot in place when the event disappeared, so
claim_notification_work rebuilt it as None and still claimed it, and the
service launched an empty notification run (dispatch_event or {}).
Consume such snapshots at claim time instead: advance notified_version, return
the status to none, clear the claim, and skip the row. The same guard consumes
rows left without a dispatch event by pre-fix claims, which would otherwise
relaunch empty notifications on every retry until dead-lettering.
* fix(mcp): clear consumed snapshot state for the review follow-ups
Review feedback on the claim-time consume guard:
- clear event_fingerprint when consuming, so an identical later attention
event is recorded again instead of being suppressed by a snapshot that was
never delivered;
- clear stale notification_error/attempt/dispatch state so a status
projection cannot show a notification that no longer exists;
- cover the reclaimed expired-claim shape (claim, lease expiry, resolve,
reclaim) and the identical-event sequence with regressions.
* docs(persistence): document the notification consume rule
mcp/AGENTS.md's effective chain has only 85 bytes of headroom on main (98,219 / 98,304), so the rule moves to persistence/AGENTS.md, which owns the mcp_tasks storage contract. No behavior change.
* fix(mcp): fence the notification consume against newer events (review)
Two P2 findings on the claim-time consume:
- the consume acknowledged the latest event version from a stale read, so on
SQLite a completion recorded after the claim's SELECT was overwritten with
status none and lost forever. The consume is now a conditional UPDATE fenced
on the event version and phase that were read; a newer event makes it a
no-op and stays claimable;
- a legacy launching/dispatched row whose dispatch payload is empty but whose
event version moved on consumed that newer event. Such rows are now rebuilt
and claimed; only a generation with nothing newer is consumed.
Regressions: a completion recorded while a claim is paused after its SELECT,
and legacy launching/dispatched rows with and without a newer completion.
* fix(mcp): skip tasks a cancel released while the poll claim was in flight
claim_due_tasks selected due rows with FOR UPDATE SKIP LOCKED, then set
the lease through an ORM flush that updates by id alone. FOR UPDATE is a
no-op on SQLite, so a request_cancel that committed between the SELECT
and the flush had its cleared poll lease written back. The cancellation
worker only claims rows without a live lease, so the remote cancel waited
for mcp_tasks.lease_seconds (120 s by default), and the task was polled
once more for a result apply_snapshot then discarded.
The claim now selects candidate ids and leases each one with a
conditional UPDATE that re-checks the claim predicates (pollable status,
no cancel request, due next_poll_at, no live lease), returning only the
rows it actually leased, the same atomic fence release_claim already
uses. On Postgres the row lock still orders the two writers.
* docs(changelog): reference #6859 for the MCP task claim fence
* docs(mcp): note that a short poll claim batch is not topped up
* feat(mcp): re-scope to MCP task claim lifecycle only
Keep PR #4966 a small, closed MCP lease/cancellation state-machine change and
move RunJournal and Run lifecycle work into dedicated follow-ups. This branch
contains only the MCP task claim lifecycle:
- mcp task release/snapshot fencing by owner + per-claim lease token
- phase-level single-flight poll/cancel/notification owners with retained handoff
- routine cancellation no longer persisted as a task failure diagnostic
- bounded ordinary release ownership retention past the drain deadline
- 0018_mcp_task_lease_tokens migration + migration/bootstrap head assertions
- wait_for_task_until helper (MCP uses it); worker-specific capture helper moved
to the run-finalization follow-up
RunJournal (journal.py + test_run_journal.py) and run lifecycle
(manager/worker/store/run sql + run tests) are preserved on
backup/cancellation-safety-full and will be raised as separate follow-ups.
* fix(mcp): unblock claims after ambiguous handoff resolves
A phase-level single-flight owner only guards an ambiguous claim outcome. Once
the claim resolves, the phase owner is released immediately; the handoff may
continue releasing returned rows as bounded, service-owned background work
(transferred to _compensation_tasks on timeout). Per-claim token fencing rejects
a late release against a newer claim generation, so a stuck release no longer
locks the whole phase until process restart.
- README: drop the stale progress-snapshot sentence from the bounded ordinary
release description.
- service: pop the identity-checked phase owner as soon as the claim outcome is
known, then release returned rows with the bounded path; carry the release in
_compensation_tasks if it exceeds the drain deadline.
- mcp/AGENTS.md: document that only an unresolved claim outcome (not the handoff)
blocks later phase scans, and that returned-row releases may continue in the
background once the owner is released.
- tests: pin that the phase owner is released before a stuck release finishes
while the release stays service strong-owned.
* refactor(mcp): remove unused single-record claim wrappers
_poll_one, _cancel_one, and _notify_one are unreachable in production: the
worker always processes claimed records through _run_claimed_batch, so these
wrappers preserved a second, dead single-record lifecycle (state is None)
whose only observable behavior was a wrapper-specific cancellation release.
Remove the three wrappers and migrate the regressions that guarded their
cancel/release invariants to exercise the production _run_claimed_batch path
(operation=_*_one_claimed, release=_release_*_after_cancellation). The single
wrapper-only "state is None" contract (test_poll_release_hang_without_batch)
is deleted; all 11 remaining invariants (CancelledError preservation, repeated
cancellation, poll-only token-fenced lease release, notification claimed vs
dispatched phase release, hung compensation -> service ownership, and
background compensation exactly-once observation) are now covered through the
real batch lifecycle.
* fix(mcp): fence claim-owned mutations against stale generations
The per-claim token check in the ORM release/apply paths was only in the
SELECT; the final write went out by primary key. On SQLite (where
with_for_update() is a no-op) a mutation from an older claim generation
could therefore clear a claim that a newer generation had reclaimed after lease
expiry — the exact distributed lease-fencing failure the per-claim token was
meant to prevent.
Make every claim-owned mutation a single atomic conditional UPDATE with the
owner and per-claim token in the WHERE clause (rowcount 0 => stale, return
False, no mutation):
- release_claim: atomic fence; record the poll-failure event after the fence
wins (same transaction, holding the write lock).
- apply_snapshot / apply_cancel_snapshot: atomic fence; record the event after.
- finish_notification_run: atomic fence; use a CASE on event_version >>
dispatch_version to keep a newer event pending for redelivery instead of
swallowing it as delivered.
Add one regression per path: a stale generation's release/apply/finish after a
same-worker reclaim is rejected and never clears the newer claim.
* test(mcp): pin the migration chain head to the lease-token revision
0026_mcp_task_lease_tokens becomes the alembic head, so the chain-head pin in the 0025 repair test had to move on. Follow the 0023 precedent there (single head plus expected predecessor) instead of pinning a literal head, and give the new revision its own migration test, which owns the pin and covers the nullable claim-token columns on upgrade and their removal on downgrade.
* refactor(mcp): close cancellation cleanup leftovers
* fix(mcp): retain cancelled release diagnostics
* test(mcp): remove obsolete settled compensation case
* test(mcp): cover interleaved lease reclaim races
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): add durable task runtime foundation
* fix(chart): sync embedded config version
* fix(mcp): isolate task polls during shutdown
* feat(mcp): track consecutive poll errors on mcp_tasks
poll_attempt_count grows on every claim (successful polls included), so it
cannot drive a failure backoff without misjudging normal long tasks. Add
consecutive_poll_error_count: incremented when a claim is released after a
poll error, reset to zero by any applied snapshot. The backoff/terminal
policy that consumes it lands with the first concrete driver.
* fix(mcp): harden durable task lifecycle
* feat(mcp): add ordinary durable task driver
* test(mcp): address durable task review feedback
* fix(mcp): preserve submit tool descriptions
* fix(mcp): bound remote task calls
* fix(mcp): bound persisted task payloads
* fix(mcp): preserve task tool error details
* fix(mcp): enforce durable task boundaries
* test(mcp): cover task config snapshot lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): add durable task runtime foundation
* fix(chart): sync embedded config version
* fix(mcp): isolate task polls during shutdown
* feat(mcp): track consecutive poll errors on mcp_tasks
poll_attempt_count grows on every claim (successful polls included), so it
cannot drive a failure backoff without misjudging normal long tasks. Add
consecutive_poll_error_count: incremented when a claim is released after a
poll error, reset to zero by any applied snapshot. The backoff/terminal
policy that consumes it lands with the first concrete driver.
* fix(mcp): harden durable task lifecycle
* fix(mcp): preserve tracked task on dedup conflict
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
