8 Commits

Author	SHA1	Message	Date
Totoro	c994826d06	feat(channels): generalize group notifications and add opt-in task/goal IM notices (#6903) ... * feat(channels): generalize IM group targets and add task/goal run notifications - Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only. - Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation. - Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades. - Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI. * fix(notifications): address review feedback and group/runtime notification edge cases * test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests * fix(channels): preserve group binding names and legacy authorization Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-15 10:30:55 +08:00
Nan Gao	e8c7df9042	feat(channels): arbitrate IM transports with a per-provider leader lease (#6843) ... * feat(channels): arbitrate IM transports with a per-provider leader lease Every Gateway instance connected every enabled IM platform with the same bot credentials, so channels could not run on more than one instance (Telegram 409 storms, Discord answering twice, scattered per-process state). channels.leader_lease was reserved by #6746; this implements it. - channel_leases table (migration 0041): one row per provider. Acquire is one dialect-native upsert decided against the database clock and bumps a fencing epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0 so a peer takes over on its next tick. - app/channels/leadership.py: the holder's local monotonic deadline (send time of the last confirmed renewal + lease_seconds); holds() turns false one renewal interval early, so the transport stops at least grace_seconds before a peer may connect. DB calls are bounded. - ChannelService: start fence in _start_channel (no DB call) behind every start path; fixed-rate supervisor renews/acquires and runs transport starts/stops as per-provider tasks; takeover reloads the provider's saved config; failed starts back off; shutdown releases only leases whose transport stopped. get_status reports leader/leader_here and the holder's transport state on every instance. - Notification worker leaves rows of providers another instance holds. - Gate: the enabled=false refusal now recommends auto/true; an explicit true with database.backend memory is refused when a channel is enabled. - Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated. * docs(changelog): link the channel leader lease entry to #6843 * fix(channels): see peer-held leases for providers absent from local config Review on #6843: a provider connected from the web UI on a peer never enters this instance's channel config, so the notification worker here still claimed its rows (and parked them for 900s) and GET /api/channels/ reported no leader for it. Every supervision round, and startup, now also reads the whole lease table once (bounded), and notification exclusions and status use those observations: a provider whose live lease a peer holds is left to it even without local config. Running that read concurrently with renewals exposed that, through SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on another connection can hand back one of its own bind values instead of the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do not show it). A lease must never take such a row for a confirmed renewal, so acquire/renew/release now decide by affected-row count and acquire reads its epoch back in the same transaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-14 13:51:42 +08:00
Totoro	5ae95e6a34	feat: add Feishu group notifications and fix mention caret placement (#6762) ... * feat(feishu): deliver scheduled-task notifications to bound users * feat: add Feishu group notifications and fix mention caret placement * fix(frontend): preserve group targets while capabilities load --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>	2026-10-13 07:17:56 +08:00
Totoro	4b36e680c3	feat(feishu): deliver scheduled-task notifications to bound users (#6753) ... Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>	2026-10-12 20:08:16 +08:00
王一帆	1de47b56ed	fix(skillscan): detect modern sk API tokens (#6394) ... * fix(skillscan): detect modern sk API tokens * fix(skillscan): skip placeholders and align notification redaction	2026-10-07 17:10:45 +08:00
Totoro	35cdcab1c5	feat(scheduled-tasks): signals, honest notices and fair queueing for server-started runs (RFC #6340, 2/2) (#6393) ... * feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks - Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal streak boundary) and stop_condition (the user's stop rule, own column). - Stop rule is normalized (stop_rule.py) and appended only at launch; the stored prompt never contains it. Wording adapts when scheduler.tool_enabled is off. - Willem 1: page-created tasks get goal evaluation, <standing_notes>, the previous-run reference and own-schedule stop; request_stop and the scheduled capability no longer require an origin conversation. - Repository: single reactivation choke point raising ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop condition/note changes (not on resume); goal-check failures neither count nor reset the streak; pause of a finished task returns "finished"; trials on a paused task keep the host pause marker. - New repository reads: automatic_runs_used_for, active_run_status_for, list_manageable_from_thread, thread_relation/thread_run on thread tasks, task_ids_for_thread, run_number, and run_number/total_tokens/summary on run history rows. - REST: goal/max_runs/end_at/stop_condition on create and PATCH (null clears, naive end_at = task timezone), resume recomputes from now with an optional renewal body, once_time_passed, task_finished, limits_exhausted, scheduler_not_running on create, trigger returns outcome/existing/thread_id, task responses add automatic_runs_used and active_run_status. - Coded errors {code, message, params} for routes, shared validation and the chat capability (scheduled_task_errors.py), pinned by contracts/scheduled_task_errors_contract.json; goal notes contract v2 with check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py). - /api/features adds scheduled_tasks {available, running, tool_enabled, min_interval_seconds}; ScheduledTaskService.is_running. - Docs: six README scheduler sections, API.md scheduled tasks section (breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs. - Tests: migration 0031, stop rule, error contract, run listing, lifecycle over HTTP, repository, service and capability cases; updated tests for the new error shape, finished-task pause and head revision. * feat(scheduler): conversation update/resume, run-conversation authority and readable launches - schedule_task: update and resume actions, stop_condition and clear_fields parameters, plain parameter types (value rules come back as coded results), rewritten docstrings for schedule_task and stop_scheduled_task. - Capability: update/resume share the REST validation and error codes; interactive turns in a task's run conversation may manage that task; scheduled runs keep only own-stop; grants require a running scheduler. - Results: compact JSON-safe task views with local times, display card/text, last run in list, timezone_source/browser_timezone, now_local; no ISO, user, lease or origin fields; stop_instructions removed. - Timezone: explicit > browser (context.client_timezone, validated in start_run and never forwarded to the run config) > not needed for intervals and offset run_at > timezone_required. - Trial gate accepts natural direct requests after acknowledgements and particles; trial results report thread_id and existing. - Scheduled launches: stable message id scheduled-<task_run_id>, server-owned deerflow_scheduled_origin metadata with the user-language parts, run thread title for fresh-thread tasks. - Worker records goal continuations in the verdict; lead prompt language rule covers stored task text and scheduled runs. - Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs. * feat(frontend): scheduled-task core types, formatting, errors and glossary - throwGatewayApiError now throws GatewayApiError (status, code, params, rawMessage); message stays the string detail or detail.message - scheduled-task types gain stop_condition, automatic_runs_used, active_run_status, origin/notes, run_number/total_tokens/summary, thread relation, trigger result, renewal and scheduled origin; isTaskBusy and hasActiveRun helpers - api: fetchScheduledTask, resume with optional renewal body, typed trigger result and update payload with null clears - hooks: polling at 3 s while a run is active (list, thread list, run history), useScheduledTask with live gating and final 404, useInView, localized error toasts; run history refreshes tasks when a run finishes - new pure modules: errors.ts (contract code to localized copy), format.ts (relative task times, viewer time, display zone), status.ts, actions.ts, run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked goal checks, describeTaskOutcome and parseStopRunId - features: scheduled_tasks block with defaults when missing - buildRunContext sends client_timezone - titleOfThread takes a localized untitled label; breadcrumb names scheduled tasks and agents; reference chips use the localized untitled label - schedule input: shared browserTimeZone, labelled selects, aria-pressed type toggles, interval minimum from the hint template - i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes, recipe instructions moved into i18n - e2e mocks follow the new task, trigger, resume, thread-task and features shapes; expectNoRawIdentifiers helpers for e2e and DOM tests - tests for errors, format, cron, status, actions, goal outcome, run errors, hooks, run context, thread titles and features; scheduled e2e copy updated * feat(frontend): list-first scheduled tasks page with outcomes and editable goals - page.tsx is now a thin composition of components/workspace/scheduled-tasks: status tabs (All/Active/Paused/Finished) and search over the task list, ?task_id= deep link with router.replace on selection, ?thread_id= chip with Show all, empty state, scheduler-off notice and unavailable state - task list rows show the schedule in words and one status line (next run, running now, paused by agent, auto-paused, finished by its cap) - task detail: status badge, actions from availableActions with reasons, More actions menu (Duplicate, Copy task ID, Delete), outcome notice, Runs / Stops when / Each run's goal / Does / Notes from chat / History, origin chat link; success toasts for every action - run rows: run number or Trial run, summary, goal and assumption badges, continuations, tokens, Open chat link; raw errors and the evaluator reason only behind Details - create/edit/duplicate dialog with goal, stop condition, safety caps and advanced agent/context options; PATCH sends only changed fields, null clears a cap - renew dialog raises or removes caps and resumes in one request; a limits_exhausted Resume opens it - removed i18n keys the old page used; added feedback.resumedNoTime - recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for the new page; DOM tests for the detail and the form dialog - CHANGELOG entries and the frontend AGENTS.md scheduled-task note * feat(frontend): live schedule card, run conversation provenance and header entry - schedule_task results (create, update, pause, resume, trial, delete) become an assistant:scheduled-task message group rendered as a live card: schedule, next run, stop condition with the safety cap, goal, where results go, Run once now, Pause/Resume and Open task; the card polls the task only while it is on screen and shows deleted, scheduler-off and limits-exhausted states - one card per task per assistant turn; the card group is not a turn boundary for usage, branching, editing, outline or skill attribution - a human message with deerflow_scheduled_origin stays a human group with scheduledOrigin and renders as a "Scheduled run / Trial run" block with the task instructions, stop condition and notes collapsed; the launched text is never shown and the message is not editable - readable step labels for schedule_task actions and stop_scheduled_task - ThreadScheduledTasksButton replaces the header link: hidden without tasks, shows the count, links to the task (or the chat filter), and a run's chat links to its task - recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card, run block, header button, step labels and reconnect hydration - i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and frontend AGENTS.md notes * fix(scheduler): address verification findings Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check. * fix(scheduler): address review findings * fix(gateway): no-transform on SSE responses All four text/event-stream responses (thread run create stream, join, existing-run GET/POST stream, stateless create stream) now take their headers from app/gateway/sse_headers.py and send Cache-Control: no-cache, no-transform, so compressing proxies such as Next's rewrite proxy stop buffering streams. Content-Location and X-Accel-Buffering are unchanged; scheduled and MCP-notification runs keep stream_mode=None. Tests pin the headers on every SSE route and fail on a literal no-cache header left in routers. Docs: API.md SSE headers note, CHANGELOGs. * feat(scheduler): per-owner run cap and fair queue selection New scheduler.max_concurrent_runs_per_user (default 2, 0 = off, effective min(cap, max_concurrent_runs)) caps launching/running scheduled runs per task owner. claim_queued_run checks it right after the global count under the same budget lock, so it is atomic across workers and Pods. list_queued_runs ranks same-thread FIFO heads per owner with ROW_NUMBER() OVER (PARTITION BY owner) and orders the batch by that rank, so the drain is owner-fair even with the cap off; owners at their cap drop out before the batch limit. Orphan rows (task deleted while queued) are never held back by the cap. Gateway lifespan passes the startup value; the service constructor default stays 0. Docs: config.example.yaml (config_version 54), CONFIGURATION.md, the six READMEs, backend AGENTS.md, CHANGELOGs (behavior change flagged). The live-task quota is untouched. * feat(scheduler): durable lifecycle events for the originating chat When a chat-created schedule is paused by the agent, auto-paused or finishes, finalization now reports task_stopped / task_paused / task_finished, only on a live -> paused/finished transition. The new finish_task_at_end_condition replaces the three inline "mark completed" blocks (admission rejected as ended, claim-time skip, complete_if_ended), so every site emits exactly once. The scheduler's finalization observer is always installed. It writes scheduled_task_events rows for the originating chat in the same transaction as the state change, deduplicated on (task_id, anchor, event), with a title snapshot, stop condition, run number, last run outcome and the chat's newest run as placement anchor. Lookups fall back on bad stored data; only database errors propagate. Manual trials drop only run_* events. The IM step keeps PR1's selection until B4. New GET /api/threads/{thread_id}/scheduled-task-events (owner check, caller's rows, oldest first); rows survive task deletion and go with the chat. Migration 0032_activity_and_task_events adds the events table plus the run-origin column/index and read-state tables used later, each object guarded on its own. Contract scheduled_goal_notes v3 pins the lifecycle vocabulary. Docs: API.md, migrations chain, backend AGENTS.md, six READMEs, CHANGELOGs. * feat(scheduler): honest, localized and self-contained IM notices Every scheduled-task IM notice is now enqueued by the finalization observer in the transaction that records the outcome, one message per occurrence (task_stopped > task_paused > task_finished > run_failed > run_unmet > run_completed; a once task sends only its run event). The merged auto-pause replaces the separate unmet + pause messages, agent stops and finishes now notify, and recovery-finalized runs notify exactly once. Idle finishes use a deterministic idle: key derived from the lifecycle anchor. The old non-transactional enqueue in handle_run_completion is removed. New app/channels/capabilities.py holds CHANNEL_CAPABILITIES with a proactive_notifications flag (adds the missing qq entry); only push-capable providers (WeCom) get rows, /api/channels/providers returns the flag, and the worker ends legacy rows for other providers as terminal failed. New app/scheduler/notification_text.py renders en-US/zh-CN text: title, what happened (plus the last run's outcome), the tasks page's one-line run summary (redacted) when the agent replied, and a closing line. No IDs, no links, no public_base_url. Locale is the owner's new `locale` preference, else the new channel_connections.notification_locale (default en-US). Docs: config.example.yaml (config_version 55), CONFIGURATION.md, API.md, channels AGENTS.md, six READMEs, CHANGELOGs (behavior change flagged). * feat(gateway): thread origin, activity feed and read markers for server-created threads Runs started by the server for a user (scheduler, IM channels, GitHub agents, extension handles, MCP notifications) now carry a server-owned metadata.deerflow_origin {kind, provider?, namespace?} (new harness module deerflow.runtime.run_origin, pinned by contracts/thread_origin_contract.json). It is stamped only from request.state.run_origin or by the internal channel caller; run admission drops client copies from both metadata and config.metadata, and thread create/patch strip it. RunRepository derives runs.origin_kind on put and on create_thread_operation_atomic (agent runs only), and gains latest_change(). New ThreadReadRepository keeps a per-user, never-decreasing read position (thread_read_markers) and a per-user read clock (thread_read_versions). POST /api/threads/{id}/read marks a thread read, thread search items carry unread, and thread delete removes the markers. GET /api/thread-activity pages the caller's run changes over the run-change clock, returns only threads changed by server-originated runs plus read_version, and 503s without SQL; /api/features reports thread_activity.available. Docs: API.md, gateway AGENTS.md (condensed to stay under its budget), CHANGELOGs. * fix(scheduler): adapt PR2 to PR1 follow-ups PR1 moved the scheduled-task lifecycle rules out of backend/AGENTS.md into the persistence guide and grew the sandbox AGENTS.md chain to within 267 bytes of its hard limit. The per-owner queue-fairness detail now lives in the persistence guide next to the lifecycle rules (backend/AGENTS.md keeps the cap and a link), so check_agent_guidance passes again. PR1's goal-outcome unit test pinned scheduled_goal_notes_contract version 2; PR2 bumps it to 3 (lifecycle vocabulary only, the v2 keys are unchanged). * fix(scheduled-tasks): polish from live acceptance - zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒, and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2 days) instead of 每 1 分钟 or 每 24 小时; several units keep the number - zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between the interpolated time and the Chinese after it; a copy test guards every scheduled-task zh string against a placeholder glued to Chinese - run summary: when the first line of the final reply only leads in to a list (ends with ":" or "："), the list items follow it as plain text (bullets, numbering, checkboxes, bold, code and links stripped), joined with "；" for CJK text and "; " otherwise, within the 160-character cap - a schedule_task result (create, update, pause, resume, trial, delete) that arrives in a chat invalidates that chat's task list, the tasks list and the named task, so the header button and cards update without waiting for the idle poll; results already in the loaded history do not refetch - e2e and unit fixtures are now the live acceptance recordings (DeepSeek Flash, demo data, fictional names): live-weekday.json (create, trial, edit, pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1 and run 2, paused by the agent), fetched with GET from the demo Gateway and trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card, run-thread and tasks-page specs run in both locales over the recorded data * docs(changelog): reference #6378 for the scheduled-tasks entries * docs(agents): shorten the scheduled-task pointer to stay under the guidance limit * test(scheduled-tasks): pin the live takes to their recorded time zone The live fixtures were recorded in Asia/Shanghai, and an interval task reads in the viewer's zone, so the 16:48 assertions failed on UTC CI runners. Pin the browser zone in the two live-take e2e specs and the zh live-take unit test. * feat(frontend): thread origin, activity polling, read markers and task events core Frontend core for server-created threads and schedule lifecycle events (PR2 F1); the sidebar markers (F2) and the chat event line (F3) build on it. - core/threads/origin.ts: threadOriginOf reads metadata.deerflow_origin, then channel_source (github -> GitHub origin), then a legacy scheduled_task_id; key and kinds pinned to contracts/thread_origin_contract.json. labelOfThreadOrigin and the now exported labelOfChannelProvider(provider, t) use the new localized threads.origin.providers names (飞书, 企业微信, ...), with the English map (gaining GitHub) only as the fallback. - core/threads/activity.ts: useThreadActivity polls /api/thread-activity every 15 s (focus refetch, never in the background, off unless features.thread_activity.available). The first response only seeds the cursor; server-origin threads, a truncated page or a higher read_version invalidate the thread lists, a scheduled thread also ["scheduled-tasks"]; an invalid cursor re-seeds. useMarkThreadRead debounces POST /read by 1 s, skips a thread whose cached unread is false, patches the search and infinite caches optimistically and stores the returned read_version so a tab's own read does not refetch its lists. AgentThread gains unread. - core/scheduled-tasks/events.ts: ScheduledTaskEvent, the events query (["scheduled-tasks","thread-events",id], one fetch per saved chat, not gated on the task list so a deleted task's lines stay, refetched when a task of the chat changes status or finishes a run), describeTaskEvent (en/zh copy, title and condition as segments, last-run suffix, See that run / Open task) and placeTaskEvents (end of the after_run_id turn, else the tail, ordered by created_at); lifecycle names pinned to contract v3. - features: thread_activity (default unavailable); channels: proactive_notifications on providers. - Locale sync: the I18nProvider reports the UI language and a signed-in preference session saves it as the new `locale` preference when the account's differs (and on every switch), so IM notices follow the UI language; nothing is written with auth disabled. - i18n: threads.* (unread, origin labels, provider names), scheduledTasks.events.*, history.waitingForSlot (now shown for queued run rows), channels.scheduledUpdates.*. - e2e mocks: thread activity feed and controller, POST /read, the events route, providers with proactive_notifications, search unread, features thread_activity (off by default). Preference e2e mocks return the UI locale so they keep asserting only their own edits. - Docs: frontend AGENTS.md, CHANGELOGs. * fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in - The edit, duplicate and renew dialogs keep a stored end time's instant while its field still shows it. A minute-precision wall value cannot keep seconds or name the later 01:30 of a DST fall-back, so a title-only edit moved the cap an hour earlier, and raising the run limit re-sent the end time without its seconds. - Creating or duplicating a task from a chat's task view drops the chat filter when the new task does not belong to that chat, instead of showing "This task isn't one of this chat's tasks". * feat(frontend): origin markers and unread state in the thread lists Server-created threads (PR2 F2) now say who created them and whether they are new, in the sidebar and on the Chats page, and reach an open sidebar without a reload. - ThreadOriginIcon (thread-channel-source.tsx): clock for a scheduled run, the provider mark for IM threads, the GitHub mark (new in ChannelProviderIcon) for GitHub threads, a puzzle piece for extensions; the localized "Scheduled run" / "From Feishu" / "From GitHub" / "From an extension" label is its accessible name and tooltip. ThreadChannelIcon stays exported as a wrapper, and the provider chip tooltip is localized (no more "Feishu channel"). - Unread dot after the title (aria-hidden dot plus sr-only "Unread"); the row aria-label becomes "{title}, unread" / "{title}，未读". Never shown for the open thread. - WorkspaceSidebar mounts useThreadActivity once. - useMarkOpenThreadRead (components/workspace/chats): the default and custom-agent chat pages mark the thread read once its metadata has loaded, when its cached unread flips to true while open, when the tab becomes visible, and from onFinish (a send or a joined run ended). Each trigger goes through useMarkThreadRead, so it is debounced and a no-op while the cached unread is false. - Tests: thread-origin-icon and recent-chat-list DOM tests (en/zh, active row, no raw identifiers), use-mark-open-thread-read DOM tests, e2e thread-activity.spec.ts (new scheduled run appears unread within one fake-clock poll without a reload, opening it posts /read and clears the dot, a read on another device clears it, IM and GitHub markers, zh labels, the Chats page, no polling when unavailable); thread-history.spec.ts now finds the localized origin icon. - Docs: frontend AGENTS.md, CHANGELOGs. * feat(frontend): schedule event line in the originating chat and channel notice support The chat that created a schedule now says when the schedule paused or ended, and the channel cards say which apps receive scheduled task updates (PR2 F3). - ScheduledTaskEventLine (mockup m2, marker 4): a dashed, muted row with the describeTaskEvent text (title in <strong>, stop condition clamped to one line with the full text in its tooltip, last-run suffix), the time via formatTaskTime and a right-aligned "See that run" (run chat) or "Open task" link. PauseCircle for paused, CircleCheck for finished and once-done, CircleX for once-failed. role="note", aria-label "Scheduled task update" / "定时任务通知", data-task-id and data-event-id only. - MessageList takes scheduledTaskEvents and renders placeTaskEvents output through a new VirtualMessageList.renderAfterGroup, so each line sits at the end of its after_run_id turn inside the group's measured row (also when the list virtualizes); lines with nowhere to go render at the tail. - Both chat pages load useThreadScheduledTaskEvents (not gated on the task list, so the lines stay after a task is deleted). The custom-agent page passes its agent so "See that run" opens the run on that agent's route. - ChannelScheduledUpdates: "Scheduled task updates: sent here" / "... not available for this app yet" (定时任务通知：会发送到这里 / 此应用暂不支持) on the Settings provider cards and under each sidebar channel row; nothing when an older backend does not report proactive_notifications. - e2e mock controller: setScheduledTasks. - Tests: scheduled-task-event-line and channel-scheduled-updates DOM tests (every kind in en and zh, hrefs, clamp, no raw identifiers, Chinese title), MessageList placement DOM tests; e2e scheduled-task-chat-events.spec.ts (the line appears with the card's "Paused by agent" without a reload, See that run opens the run chat, anchored vs tail placement, two events in time order, the line survives task deletion and Open task shows the task is gone; en-3 and zh-1) and channels.spec.ts (WeCom sent here, Feishu not yet; en and zh). - Docs: frontend AGENTS.md, CHANGELOGs. * fix(scheduled-tasks): tie pause notices to the pause they describe - Paused by agent links and times only the run named in last_error. When that run is not loaded, the notice shows no link and the task's own time instead of an earlier run that also asked to stop. - The outcome notice reads the latest history page whichever page is shown, so browsing older runs no longer swaps the auto-pause reason and "Open latest run" to an earlier miss. It shares that page's request. * fix(scheduler): address PR2 review findings - Event lines: hold an event whose anchor run is on an unloaded history page instead of placing it at the bottom (hasMoreHistory); tail fallback only for fully loaded (branched/pruned) history or no anchor. - Idle-finish anchor follows its reason: end:<iso> only for an end_at finish, seq:<n> otherwise, so a later end-time finish is not deduplicated away by an earlier max_runs finish. - Observer takes FOR SHARE on the originating threads_meta row, so a racing thread delete cannot leave an orphan event line. - One taskPagePath (core), re-exported by the card. - Notification locales derived from NotificationLocale; /preferences uses it. - IM notices drop generic reasons (goal_not_met_yet, unknown) instead of repeating the sentence. - Localized provider names on Settings cards, sidebar rows and setup dialog. - zh event lines: no space after a Chinese title (Latin-aware gap). - Unread row label keeps the origin marker; chats page keeps the time as a description. - Sidebar shows the scheduled-updates line only for connected apps; Settings says "available after you connect" for unconnected push apps. - Event line keeps a real space before the time. * fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes - The edit form compares a cron in the form its schedule input emits, so an untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only edit no longer re-arms or refuses a finished task. - Chat resume asks for a timezone, as update does, when a naive end_at would land in the UTC placeholder of a zone-free task. - Re-arming a task clears a host pause marker, as resume does, so a later manual pause no longer reads as "Paused by agent". * test(scheduler): consolidate PR2 backend test setup - Fold the repeated task, origin-chat and observer setup of the lifecycle-event repository tests into one lifecycle_task builder. - Build the per-owner-cap services through one _capped_service builder that keeps every cap explicit at the call site. - Table the five SSE route header checks into one parametrized test with the same routes, bodies and Content-Location expectations. - Reuse bind_wecom and hoist repeated local imports in the event tests. Every case and assertion is kept; collected cases are unchanged. * test(frontend): consolidate PR2 thread activity and origin test setup Extract shared builders and helpers in the frontend tests PR2 adds, with every case, input and assertion kept: - activity.test.ts: seededClient/apply helpers and one LIST_KEYS constant replace the per-test seed, spy and apply boilerplate. - mark-read.dom.test.tsx: newClient() and advance() replace the repeated QueryClient wrapper and fake-timer act blocks. - origin.test.ts: the English and Chinese label checks become tables; the no-English-name check now runs on every Chinese row. - thread-activity.spec.ts: a scheduledRun() builder replaces four inline copies of the scheduled run thread. - channels.spec.ts: the PR2 block uses setLocaleCookie and one card helper. Collected cases are unchanged (unit 211, e2e 60); no test was removed. * fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages useLatestScheduledTaskRuns kept React Query's default refetch on mount, focus and reconnect, so going offline and back while an older history page was shown fetched the latest page again. Refreshing stays with the history, which refreshes the latest page only while it is shown. * fix(scheduler): two gaps found in live acceptance - A fresh scheduled run thread is named when it launches. The worker copied the title to the thread list only when the run ended, so the sidebar, which now shows the new run thread at once, listed it as untitled while it ran. - A task capped at one run finishes with "its one run is done" / "唯一一次运行已完成" in the chat event line and the IM notice, instead of "all 1 runs are done". * test(scheduled-tasks): let the viewer's time carry its own day label Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days, so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the fixed "01:00 · 09:00 your time" assertion failed depending on the hour. * test(scheduler): expect the one-run finish copy in service notices * fix(gateway): create a scheduled run thread with its title Pre-setting the name after start_run raced the thread row, which the run worker creates asynchronously: live, one of two run threads still reached the sidebar untitled. Run admission now creates a new thread with the title its run input carries, which the worker would copy only at run end. * docs(changelog): reference #6393 for the scheduled-task signal entries * fix(scheduled-tasks): route event-line run links by the agent that ran them The originating chat's agent was used for "See that run", but a task's agent can change after the chat created it, so the link could open the run chat on another agent's route. Lifecycle events now record run_agent_name, the task's agent when the run finished (edits are refused while a run is active), and the line routes by it. * fix(scheduled-tasks): load up to the route's maximum of chat event lines The chat loads its lifecycle events once, and the route's default of 50 would silently drop the oldest lines of a chat with many schedules. Ask for the maximum, 200. * fix(threads): keep activity reads and the first cursor in step with the lists - Accepting the first activity cursor refreshes the thread lists once. The lists load on their own, possibly before a run the seed already counts, and later polls start after it, so such a thread never appeared. - A local read whose returned read clock jumps past that one read also covers reads on other devices; the lists are refreshed instead of the clock silently skipping those threads' stale unread dots. - An invalid cursor re-seeds through the same path, so it refreshes once. * docs(agents): bring the middlewares guidance chain back under the hard limit #6378 and #6363 each fit the budget, but together the middlewares AGENTS.md chain on main is 98,425 bytes, 121 over the 98,304 hard limit, so Lint Check fails on main. Move the scheduled-task recovery, launch-update and queue-timeout rules from backend/AGENTS.md into the persistence guide's lifecycle section, which backend/AGENTS.md already points to. The chain is now 98,067 bytes. * fix(helm): bump the chart's config_version to 55 config.example.yaml moved to 55 in this PR, so the Helm chart's validate-chart drift check failed. The chart's embedded config needs no field changes: the new scheduler and channel keys have defaults. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-07 10:22:15 +08:00
Gumayusy	d9d9ff9708	fix: fence stale notification delivery claims (#6211) ... * fix: fence stale notification delivery claims * test: cover notification claim migration upgrades --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-04 10:43:05 +08:00
Totoro	b57fa6fb9e	feat(scheduler): push scheduled task results to IM channels via a notification outbox (#6135) ... * feat(persistence): add notification delivery outbox for scheduled task IM push New notification_deliveries table and repository (migration 0012) backing the outbox pattern for issue #4254: idempotent enqueue keyed on task_run_id plus event plus provider plus target, atomic claim of due rows, and exponential-backoff retry state moving pending to sending to sent or failed. Execution status stays in scheduled_task_runs; delivery status lives here. * feat(scheduler): enqueue and deliver scheduled-task notifications over IM channels Closes the loop for issue #4254. On run completion the scheduler writes outbox rows for the task owner's connected channel identities (success and failure events; interrupts stay silent), and a new NotificationDeliveryWorker polls due rows and pushes them through the channel's proactive send path. Channel gains send_notification; WeCom implements it via the WS client's send_message with errcode checking, the base class raises for providers without proactive push. Delivery is best-effort and never shadows execution status: enqueue and send failures are logged and recorded in the outbox, which retries with backoff. Channel liveness is re-checked at delivery time, and the worker stops before the channel service on shutdown. Feature activates only when channel connections are enabled; no new dependencies. Verified end to end against a live WeCom bot: scheduled run completion produced an outbox row that was delivered in under one second. * feat(scheduler): include scheduled run result summary in IM notifications The v1 notification only carried the run outcome skeleton (status, task id, run id). Completed runs now include the agent's final answer so the IM push is actually useful headlessly. - The delivery worker accepts an optional resolve_run_summary(run_id, user_id) resolver; for run_completed events it enriches the rendered text with a "Result:" line (bounded to 1000 chars). The lookup is best-effort: resolver failures or missing summaries degrade to the skeleton text and never block or fail the delivery. - The summary is read at delivery time from runs.last_ai_message (the convenience column written on run completion), not snapshotted into the outbox payload; retried deliveries re-read the freshest value and the outbox row payload stays skeleton-only. The claimed outbox row is never mutated in place. - Gateway wiring scopes the read to the outbox row's owner_user_id so a stale row cannot pull another user's run content. - Failed runs keep the error-only skeleton: a partial answer on a failed run would be misleading. Tests: 9 new worker cases (rendering incl. truncation, sync/async resolvers, failure/missing fallbacks, no resolver call for failed runs). Full scheduled + outbox + channels regression green (441 passed), ruff check + format clean. * fix(scheduler): address review feedback on notification delivery reliability Review follow-ups for the scheduled-task IM notification outbox: 1. Crash recovery for "sending" rows: claim now stamps updated_at and the worker reconciles rows stuck in "sending" past a 10-minute lease back to "pending" before every claim. Per-row try/except isolates a crashing delivery from the rest of the claimed batch. 2. Double-send race under multiple workers: claim returns exactly the rows its own UPDATE...RETURNING flipped (guarded by status='pending'), so a rival worker's concurrent claim can no longer be handed out twice. 3. Restore non-ASCII comments and drop the BOM accidentally introduced in test_scheduled_task_service.py. 4. Manual "run now" completions no longer enqueue IM notifications; the user is already watching the UI. Missing trigger metadata fails safe towards notifying. 5. Channel-outage failures no longer consume the retry budget: they are parked with count_attempt=False on a flat 15-minute backoff so an hours-long outage cannot exhaust retries and drop the notification. Tests: repository 16, worker 21, scheduled-task service 31; targeted regression across notification/scheduler/channel suites: 511 passed. * fix(scheduler): address round-2 review on IM notification delivery Park WeCom transport/outage failures without burning retries, bound worker stop, include task titles, align enqueue with delivery, and document the IM push behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(scheduler): complete IM notification docs and drop unused list API Add backend/channels/migrations AGENTS notes, CHANGELOG and config.example hints for the outbox push path, and remove the unused list_by_task_run repository method flagged in review. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scheduler): address round-3 review on IM notification delivery Cap channel-down parking by row age and parked attempts, redact IM egress content, narrow WeCom transport exceptions, add lifespan wiring tests, and run ruff format so lint-backend passes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scheduler): redact entire PEM blocks in IM egress text The previous BEGIN-only pattern left key material and the END marker in notifications. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(persistence): align 0014 parked_attempts with bootstrap schema tests Backfill existing rows with a temporary server default, then drop it so create_all matches Alembic, and move HEAD anchors to 0014. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(persistence): format notification tests and shorten alembic revision id ruff format --check failed on test_notification_delivery_worker.py. 0015_notification_delivery_parked_attempts exceeds alembic_version.version_num (VARCHAR 32) and truncates on Postgres CI; use 0015_parked_attempts instead. * test: fix post-merge CI failures after migration renumber Update migration 0015 test HEAD expectation to 0017_parked_attempts and add queue_timeout_seconds to the lifespan notification worker mock scheduler config. * fix(persistence): chain the notification outbox migrations after 0026 Main's head moved from 0016 to 0026_mcp_task_lease_tokens, so the two outbox revisions are renumbered and re-parented: - 0017_notification_deliveries -> 0027_notification_deliveries - 0018_parked_attempts -> 0028_parked_attempts Only the revision ids, parents and the migration guide change. * fix(scheduler): adapt the notification hook and its docs to main - tests: the completion tests assert on the atomic complete_run record; new cases cover a completion that was not recorded, a task deleted meanwhile, a failed task read, and the outbox being off. - AGENTS.md: drop the root note and shorten the backend note to a pointer; the two notes pushed two effective guidance chains past the 96 KiB limit. - CHANGELOG: move the entry from the released 2.1.0 section to Unreleased and add its link references. * test(persistence): move the chain-head pin to the outbox migrations test_migration_0026 pinned 0026 as the head, so it failed once the outbox revisions chained after it. It now checks chain membership like the older migration tests, and a new test file owns the head pin, checks the parents and the 32-character revision id limit, and runs the upgrade and downgrade. * fix(gateway): wire scheduled-task notifications after the channel service starts The outbox wiring sat in the scheduler block and asked for the running channel service there. #5035 moved the channel service start to after the scheduler, and the two changes merged without a conflict, so on every startup the lookup returned None and the feature disabled itself with the "write-only outbox" warning. The lifespan tests patched the lookup to a constant and kept passing. The wiring now runs in its own step after the channel service start. The scheduler is built as on main, and ScheduledTaskService gains attach_notification_outbox() so the enqueue side is enabled once the delivery worker is running. The guard, the worker and the shutdown order are unchanged. Tests: the lifespan helper can model the real accessor (None until the channel service has started); a new case fails without this fix. * fix(scheduler): enqueue from scheduler start and keep the push when the title read fails Follow-up to the startup-order fix, from review. - The enqueue side is wired with the scheduler again (it only needs the durable table), and only the delivery worker waits for the channel service. Attaching it later left a window after the scheduler started in which a finished run wrote no outbox row. If the channel service is missing or the worker cannot start, the enqueue side is detached, so the outbox still never becomes write-only. - A failed task read after complete_run no longer drops the notification: the title is optional and the message falls back to the task id. A task deleted meanwhile still stays silent. * fix(channels): park WeCom notifications while the SDK socket is closed The WeCom SDK raises a plain RuntimeError when its socket is not open, and send_notification only mapped ConnectionError, TimeoutError and OSError to ChannelUnavailable. A disconnect therefore spent the counted retry budget and settled the row as failed after about 15 minutes, although the docs say transport outages park without consuming retries. send_notification now asks the client's is_connected first. The new test uses a real, never-connected SDK client: it raised RuntimeError after three retries before, and parks now. * docs(scheduler): describe which outcomes notify; mirror the changelog entry - README (en/zh) and CONFIGURATION.md: only runs that finish as success or failed notify; occurrences that end without a finished run (launch error, queue timeout, restart recovery) do not. Parking lasts up to about a day. - CHANGELOG: cite the issue as text like the rest of the file, and add the entry and link reference to CHANGELOG_zh.md. - AGENTS.md: drop the backend pointer so this branch adds nothing to the guidance chains closest to their size limit; the scheduler-side rules and the lifespan order now live in app/channels/AGENTS.md. * docs(changelog): cite #6135 next to #4843 for the notification outbox entry * fix(scheduler,channels): re-check the IM binding at delivery time; park WeCom transport errors, count its rejections Review round 1 on #6135. The delivery worker now takes resolve_connections (the Gateway wires ChannelConnectionRepository.list_connections) and, before the channel liveness check and the summary read, requires the row's owner to still hold a connected (provider, external_account_id). A disconnected target is finalized with mark_failed(terminal=True); a lookup failure or an unexpected result shape parks the row. Terminal outbox rows ignore later status writes. WeCom: the SDK raises RuntimeError from send_message for a socket that is not open and for a dropped connection, but also for a platform rejection ("Reply ack error: errcode=..."). Only that prefix keeps counting; other RuntimeErrors, ConnectionError/TimeoutError/OSError and websockets' ConnectionClosed park the row as ChannelUnavailable. * fix(channels): treat a WeCom ACK rejection as proof the transport is up; pin the rejection prefix through the retry branch and the real connection-row shape --------- Co-authored-by: GaoTianyuanhr <75788039+GaoTianyuanhr@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: GaoTianyuanhr <GaoTianyuanhr@users.noreply.github.com> Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>	2026-10-01 17:59:30 +08:00