* feat(persistence): add notification delivery outbox for scheduled task IM push
New notification_deliveries table and repository (migration 0012) backing
the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id
plus event plus provider plus target, atomic claim of due rows, and
exponential-backoff retry state moving pending to sending to sent or failed.
Execution status stays in scheduled_task_runs; delivery status lives here.
* feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels
Closes the loop for issue #4254. On run completion the scheduler writes
outbox rows for the task owner's connected channel identities (success and
failure events; interrupts stay silent), and a new NotificationDeliveryWorker
polls due rows and pushes them through the channel's proactive send path.
Channel gains send_notification; WeCom implements it via the WS client's
send_message with errcode checking, the base class raises for providers
without proactive push.
Delivery is best-effort and never shadows execution status: enqueue and send
failures are logged and recorded in the outbox, which retries with backoff.
Channel liveness is re-checked at delivery time, and the worker stops before
the channel service on shutdown. Feature activates only when channel
connections are enabled; no new dependencies.
Verified end to end against a live WeCom bot: scheduled run completion
produced an outbox row that was delivered in under one second.
* feat(scheduler): include scheduled run result summary in IM notifications
The v1 notification only carried the run outcome skeleton (status, task id,
run id). Completed runs now include the agent's final answer so the IM push
is actually useful headlessly.
- The delivery worker accepts an optional resolve_run_summary(run_id,
user_id) resolver; for run_completed events it enriches the rendered text
with a "Result:" line (bounded to 1000 chars). The lookup is best-effort:
resolver failures or missing summaries degrade to the skeleton text and
never block or fail the delivery.
- The summary is read at delivery time from runs.last_ai_message (the
convenience column written on run completion), not snapshotted into the
outbox payload; retried deliveries re-read the freshest value and the
outbox row payload stays skeleton-only. The claimed outbox row is never
mutated in place.
- Gateway wiring scopes the read to the outbox row's owner_user_id so a
stale row cannot pull another user's run content.
- Failed runs keep the error-only skeleton: a partial answer on a failed
run would be misleading.
Tests: 9 new worker cases (rendering incl. truncation, sync/async
resolvers, failure/missing fallbacks, no resolver call for failed runs).
Full scheduled + outbox + channels regression green (441 passed), ruff
check + format clean.
* fix(scheduler): address review feedback on notification delivery reliability
Review follow-ups for the scheduled-task IM notification outbox:
1. Crash recovery for "sending" rows: claim now stamps updated_at and the
worker reconciles rows stuck in "sending" past a 10-minute lease back to
"pending" before every claim. Per-row try/except isolates a crashing
delivery from the rest of the claimed batch.
2. Double-send race under multiple workers: claim returns exactly the rows
its own UPDATE...RETURNING flipped (guarded by status='pending'), so a
rival worker's concurrent claim can no longer be handed out twice.
3. Restore non-ASCII comments and drop the BOM accidentally introduced in
test_scheduled_task_service.py.
4. Manual "run now" completions no longer enqueue IM notifications; the
user is already watching the UI. Missing trigger metadata fails safe
towards notifying.
5. Channel-outage failures no longer consume the retry budget: they are
parked with count_attempt=False on a flat 15-minute backoff so an
hours-long outage cannot exhaust retries and drop the notification.
Tests: repository 16, worker 21, scheduled-task service 31; targeted
regression across notification/scheduler/channel suites: 511 passed.
* fix(scheduler): address round-2 review on IM notification delivery
Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(scheduler): complete IM notification docs and drop unused list API
Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): address round-3 review on IM notification delivery
Cap channel-down parking by row age and parked attempts, redact IM egress
content, narrow WeCom transport exceptions, add lifespan wiring tests, and
run ruff format so lint-backend passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(scheduler): redact entire PEM blocks in IM egress text
The previous BEGIN-only pattern left key material and the END marker in notifications.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): align 0014 parked_attempts with bootstrap schema tests
Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(persistence): format notification tests and shorten alembic revision id
ruff format --check failed on test_notification_delivery_worker.py.
0015_notification_delivery_parked_attempts exceeds alembic_version.version_num
(VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead.
* test: fix post-merge CI failures after migration renumber
Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config.
* fix(persistence): chain the notification outbox migrations after 0026
Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two
outbox revisions are renumbered and re-parented:
- 0017_notification_deliveries -> 0027_notification_deliveries
- 0018_parked_attempts -> 0028_parked_attempts
Only the revision ids, parents and the migration guide change.
* fix(scheduler): adapt the notification hook and its docs to main
- tests: the completion tests assert on the atomic complete_run record;
new cases cover a completion that was not recorded, a task deleted
meanwhile, a failed task read, and the outbox being off.
- AGENTS.md: drop the root note and shorten the backend note to a pointer;
the two notes pushed two effective guidance chains past the 96 KiB limit.
- CHANGELOG: move the entry from the released 2.1.0 section to Unreleased
and add its link references.
* test(persistence): move the chain-head pin to the outbox migrations
test_migration_0026 pinned 0026 as the head, so it failed once the outbox
revisions chained after it. It now checks chain membership like the older
migration tests, and a new test file owns the head pin, checks the parents
and the 32-character revision id limit, and runs the upgrade and downgrade.
* fix(gateway): wire scheduled-task notifications after the channel service starts
The outbox wiring sat in the scheduler block and asked for the running
channel service there. #5035 moved the channel service start to after the
scheduler, and the two changes merged without a conflict, so on every
startup the lookup returned None and the feature disabled itself with the
"write-only outbox" warning. The lifespan tests patched the lookup to a
constant and kept passing.
The wiring now runs in its own step after the channel service start. The
scheduler is built as on main, and ScheduledTaskService gains
attach_notification_outbox() so the enqueue side is enabled once the
delivery worker is running. The guard, the worker and the shutdown order
are unchanged.
Tests: the lifespan helper can model the real accessor (None until the
channel service has started); a new case fails without this fix.
* fix(scheduler): enqueue from scheduler start and keep the push when the title read fails
Follow-up to the startup-order fix, from review.
- The enqueue side is wired with the scheduler again (it only needs the
durable table), and only the delivery worker waits for the channel
service. Attaching it later left a window after the scheduler started in
which a finished run wrote no outbox row. If the channel service is
missing or the worker cannot start, the enqueue side is detached, so the
outbox still never becomes write-only.
- A failed task read after complete_run no longer drops the notification:
the title is optional and the message falls back to the task id. A task
deleted meanwhile still stays silent.
* fix(channels): park WeCom notifications while the SDK socket is closed
The WeCom SDK raises a plain RuntimeError when its socket is not open, and
send_notification only mapped ConnectionError, TimeoutError and OSError to
ChannelUnavailable. A disconnect therefore spent the counted retry budget
and settled the row as failed after about 15 minutes, although the docs
say transport outages park without consuming retries.
send_notification now asks the client's is_connected first. The new test
uses a real, never-connected SDK client: it raised RuntimeError after three
retries before, and parks now.
* docs(scheduler): describe which outcomes notify; mirror the changelog entry
- README (en/zh) and CONFIGURATION.md: only runs that finish as success or
failed notify; occurrences that end without a finished run (launch error,
queue timeout, restart recovery) do not. Parking lasts up to about a day.
- CHANGELOG: cite the issue as text like the rest of the file, and add the
entry and link reference to CHANGELOG_zh.md.
- AGENTS.md: drop the backend pointer so this branch adds nothing to the
guidance chains closest to their size limit; the scheduler-side rules
and the lifespan order now live in app/channels/AGENTS.md.
* docs(changelog): cite #6135 next to #4843 for the notification outbox entry
* fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections
Review round 1 on #6135.
The delivery worker now takes resolve_connections (the Gateway wires
ChannelConnectionRepository.list_connections) and, before the channel
liveness check and the summary read, requires the row's owner to still
hold a connected (provider, external_account_id). A disconnected target
is finalized with mark_failed(terminal=True); a lookup failure or an
unexpected result shape parks the row. Terminal outbox rows ignore
later status writes.
WeCom: the SDK raises RuntimeError from send_message for a socket that
is not open and for a dropped connection, but also for a platform
rejection ("Reply ack error: errcode=..."). Only that prefix keeps
counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and
websockets' ConnectionClosed park the row as ChannelUnavailable.
* fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape
---------
Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
b57fa6fb9e