mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 13:16:24 +00:00
feat(scheduled-tasks): show goal outcomes and end conditions in run history (#6326)
* docs(scheduler): document tool_enabled, goal notices and goal-evaluation change Follow-up to #6229: - CONFIGURATION.md: add scheduler.tool_enabled to the example and notes. - CONFIGURATION.md, README (en/zh/pt): unmet goal runs send a goal-unmet notice instead of a completion notice, and three unmet runs add an auto-pause notice; the docs still said only success/failed notify. - CHANGELOG (en/zh): the #6229 entry now states the default-on goal evaluation change for scheduled/webhook/autonomous goal runs and that evaluator usage counts toward run usage. * feat(scheduled-tasks): show goal outcomes and end conditions in run history Follow-up to #6229. The API already returns these fields; the tasks page did not show them, so a met goal read as plain "Success", an unmet reason code rendered as a red execution error, and the goal-unmet IM notice pointed to a workspace showing neither objective nor verdict. - Task detail shows the per-run goal, max automatic runs and end time when a conversation-created task has them. - Goal runs show "Goal met" (or met relying on stated assumptions); unmet runs show a localized reason plus the host code in neutral styling. The run whose agent asked to stop the schedule is marked. - Runs without a goal, and failed runs, render as before. Read-only and frontend-only: no API change. * docs(changelog): add #6326 scheduled-task goal outcome entry * fix(scheduled-tasks): show readable unmet reasons instead of raw codes Unmet runs now show only the localized reason; the host code moves to the tooltip so it can still be matched with the IM notice. Codes without a label are still shown verbatim. * fix(scheduler): show readable goal reasons instead of internal codes Task detail now localizes the host-written last_error values from goal and stop finalization (agent stop, automatic pause, unmet reason codes) under a matching label, and Chinese copy is tightened. Goal-unmet and auto-pause IM notices state the reason in words instead of printing the raw code; unknown codes are still not forwarded. * docs(changelog): describe readable goal reasons in #6326 entry * test(scheduler): pin host goal strings with a cross-language contract Address review on #6326: - Add contracts/scheduled_goal_notes_contract.json listing the task last_error strings and unmet reason codes the tasks page and IM notices translate. A backend test checks finalization constants, blocker and stand-down codes against it; a frontend test checks every listed value has a label. - Move the two last_error strings in finalization.py into constants. - Frontend reason lookup strips the blocked: prefix like the IM map and now covers goal_not_met_yet, with en/zh labels. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Totoro
GitHub
parent
78150e9b30
commit
cdbcc0d9e9
16
CHANGELOG.md
16
CHANGELOG.md
@ -17,10 +17,23 @@ This release closes that milestone with **301 merged pull requests**.
|
||||
support bounded automatic launches and per-occurrence goals, and let a scheduled
|
||||
agent request stopping its own schedule. Unmet goals and automatic pause use
|
||||
the existing notification outbox; explicit notes and authorized previous-run
|
||||
references carry context forward without changing the goal lifecycle. ([#6229])
|
||||
references carry context forward without changing the goal lifecycle.
|
||||
Independent of these tools, existing scheduled, webhook and autonomous goal
|
||||
runs may now accept disclosed low-risk, reversible assumptions, recorded as
|
||||
`relied_on_assumption`; interactive goal evaluation stays strict. Goal
|
||||
evaluator calls and tokens now count toward run usage. ([#6229])
|
||||
|
||||
#### Scheduler
|
||||
|
||||
- **scheduler:** The tasks page shows the per-run goal and end conditions of
|
||||
conversation-created tasks. Run history shows whether a goal was met,
|
||||
including when it relied on stated assumptions; an unmet run shows a readable
|
||||
reason in neutral styling instead of a raw code in the error style; and the
|
||||
run whose agent asked to stop the schedule is marked. Agent-stop,
|
||||
auto-pause and unmet values in a task's last error appear as text in the UI
|
||||
language, and goal-unmet/auto-pause IM notices state the reason in words.
|
||||
Tasks without a goal render as before; no API change. ([#6326])
|
||||
|
||||
- **scheduler:** Scheduled tasks can be searched by title or prompt. Finding a
|
||||
task previously meant scanning every title or opening details to read its
|
||||
prompt. A search field above the existing filters matches a literal, trimmed,
|
||||
@ -7775,3 +7788,4 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
||||
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
||||
[#6319]: https://github.com/bytedance/deer-flow/pull/6319
|
||||
[#6326]: https://github.com/bytedance/deer-flow/pull/6326
|
||||
|
||||
@ -16,10 +16,19 @@
|
||||
|
||||
- **调度器：** 按需启用对话工具创建及管理属主绑定的定时任务，支持自动启动上限、
|
||||
每次执行的目标评估，以及 Agent 请求停止自身调度。目标未达成与自动暂停复用
|
||||
现有通知 outbox；明确备注和获授权的上次执行引用延续上下文，不增加 goal 状态。([#6229])
|
||||
现有通知 outbox；明确备注和获授权的上次执行引用延续上下文，不增加 goal 状态。
|
||||
与这些工具是否启用无关：已有的定时、webhook 和 autonomous goal 运行现在可以接受
|
||||
已声明的低风险、可逆假设，并记录为 `relied_on_assumption`；交互式 goal 评估仍保持
|
||||
严格。goal 评估器的调用和 token 现在计入运行用量。([#6229])
|
||||
|
||||
#### 调度器
|
||||
|
||||
- **调度器：** 任务页显示对话创建任务的每次执行目标和结束条件。执行记录显示目标
|
||||
是否达成（含依赖已声明假设的情况）；未达成的执行以中性样式显示可读的原因，不再
|
||||
按执行错误标红显示原始代码；请求停止调度的那次执行会被标出。任务详情里由 Agent
|
||||
停止、自动暂停和目标未达成写入的信息按界面语言显示；目标未达成和自动暂停的 IM
|
||||
通知改为用文字说明原因。没有目标的任务显示不变；没有 API 变更。([#6326])
|
||||
|
||||
- **调度器：** 定时任务现在可以按标题或 prompt 搜索。此前找一个任
|
||||
务只能逐个扫标题或打开详情读 prompt。现有过滤器上方新增的搜索框
|
||||
会匹配两个字段中任一的字面子串（先去首尾空白、大小写不敏感，包括
|
||||
@ -6484,3 +6493,4 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子
|
||||
[#6306]: https://github.com/bytedance/deer-flow/pull/6306
|
||||
[#6307]: https://github.com/bytedance/deer-flow/pull/6307
|
||||
[#6319]: https://github.com/bytedance/deer-flow/pull/6319
|
||||
[#6326]: https://github.com/bytedance/deer-flow/pull/6326
|
||||
|
||||
@ -2396,7 +2396,7 @@ Current MVP capabilities:
|
||||
- Pause, resume, trigger, inspect history, and delete tasks
|
||||
- Search task titles or prompts, combined with status/type filters and the current thread scope.
|
||||
- Execute scheduled work through the normal DeerFlow run lifecycle
|
||||
- When `channel_connections.enabled: true`, push a summary to the task owner's connected IM identities when a scheduled run finishes as success or failed (outbox + delivery worker). Manual "run now" and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park deliveries without exhausting retries, for up to about a day; platform rejections retry for roughly 15 minutes before settling as `failed`. An identity you disconnect while a delivery is waiting is never pushed to: the row is dropped as `failed`. Proactive push is currently implemented for WeCom; other connected providers enqueue but fail until they grow a `send_notification` path.
|
||||
- When `channel_connections.enabled: true`, push a summary to the task owner's connected IM identities when a scheduled run finishes as success or failed (outbox + delivery worker). Goal tasks send a goal-unmet notice for an `unmet` occurrence instead, plus an auto-pause notice when three unmet occurrences pause the task. Manual "run now" and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park deliveries without exhausting retries, for up to about a day; platform rejections retry for roughly 15 minutes before settling as `failed`. An identity you disconnect while a delivery is waiting is never pushed to: the row is dropped as `failed`. Proactive push is currently implemented for WeCom; other connected providers enqueue but fail until they grow a `send_notification` path.
|
||||
- Browse execution history in pages of 50; older pages pause automatic refresh, with an explicit return to the latest runs. Counts appear only after a successful read; loading and failed reads are not reported as zero runs.
|
||||
|
||||
**Filter execution history through the API**
|
||||
|
||||
@ -1506,7 +1506,7 @@ Capacidades atuais do MVP:
|
||||
- Pausar, retomar, disparar, inspecionar o histórico e excluir tarefas
|
||||
- Buscar em títulos ou prompts de tarefas, em combinação com filtros de status/tipo e com o escopo da thread atual.
|
||||
- Executar o trabalho agendado pelo ciclo de vida normal de runs do DeerFlow
|
||||
- Com `channel_connections.enabled: true`, enviar um resumo às identidades de IM conectadas do dono da tarefa quando um run agendado termina como sucesso ou falha (outbox + worker de entrega). O "run now" manual e as interrupções ficam em silêncio, assim como as ocorrências que terminam sem um run concluído (erro de lançamento, timeout de fila, recuperação após reinício). Indisponibilidades de canal/transporte deixam as entregas estacionadas sem esgotar as tentativas, por até cerca de um dia; rejeições da plataforma são tentadas de novo por aproximadamente 15 minutos antes de se fixarem como `failed`. Uma identidade que você desconecta enquanto uma entrega está em espera nunca recebe o envio: a linha é descartada como `failed`. O push proativo está implementado hoje para o WeCom; outros provedores conectados enfileiram, mas falham até ganharem um caminho `send_notification`.
|
||||
- Com `channel_connections.enabled: true`, enviar um resumo às identidades de IM conectadas do dono da tarefa quando um run agendado termina como sucesso ou falha (outbox + worker de entrega). Tarefas com goal enviam, em vez disso, um aviso de goal não atingido para uma ocorrência `unmet`, além de um aviso de pausa automática quando três ocorrências não atingidas pausam a tarefa. O "run now" manual e as interrupções ficam em silêncio, assim como as ocorrências que terminam sem um run concluído (erro de lançamento, timeout de fila, recuperação após reinício). Indisponibilidades de canal/transporte deixam as entregas estacionadas sem esgotar as tentativas, por até cerca de um dia; rejeições da plataforma são tentadas de novo por aproximadamente 15 minutos antes de se fixarem como `failed`. Uma identidade que você desconecta enquanto uma entrega está em espera nunca recebe o envio: a linha é descartada como `failed`. O push proativo está implementado hoje para o WeCom; outros provedores conectados enfileiram, mas falham até ganharem um caminho `send_notification`.
|
||||
- Navegar pelo histórico de execuções em páginas de 50; páginas mais antigas pausam a atualização automática, com um retorno explícito às execuções mais recentes. As contagens só aparecem depois de uma leitura bem-sucedida; carregamento e leituras com falha não são reportados como zero execuções.
|
||||
|
||||
**Filtrar o histórico de execuções pela API**
|
||||
|
||||
@ -925,7 +925,7 @@ DeerFlow 现在在 workspace 里内置了一个一等的定时任务（scheduled
|
||||
- 当某次执行处于 `queued`、`launching` 或 `running` 时冻结任务定义，避免持久化的执行意外换用新的 prompt、thread 或调度；将任务切换为暂停或删除任务会取消已在等待的执行，而 `launching`/`running` 执行结束后才能重试这些变更；显式手动触发在调度已暂停时仍可等待并执行，且不会自动恢复调度
|
||||
- 支持暂停、恢复、手动触发、查看历史和删除任务
|
||||
- 定时任务通过正常的 DeerFlow run 生命周期执行
|
||||
- 当 `channel_connections.enabled: true` 时，定时运行以成功或失败结束时，会把摘要推送到任务所有者已绑定的 IM 身份（outbox + 投递 worker）。手动「立即运行」和中断不推送；没有跑完一次运行就结束的执行记录（启动失败、排队超时、重启恢复）也不推送。渠道/传输宕机时会停车且不耗尽重试，最长约一天；平台拒绝大约重试 15 分钟后落为 `failed`。等待期间被你解绑的身份不会再收到推送：该条投递直接落为 `failed`。目前主动推送由企业微信实现；其它已绑定渠道会入队，但在实现 `send_notification` 前会失败。
|
||||
- 当 `channel_connections.enabled: true` 时，定时运行以成功或失败结束时，会把摘要推送到任务所有者已绑定的 IM 身份（outbox + 投递 worker）。带目标的任务在某次执行为 `unmet` 时改发“目标未达成”通知；连续三次未达成导致任务自动暂停时，再发一条自动暂停通知。手动「立即运行」和中断不推送；没有跑完一次运行就结束的执行记录（启动失败、排队超时、重启恢复）也不推送。渠道/传输宕机时会停车且不耗尽重试，最长约一天；平台拒绝大约重试 15 分钟后落为 `failed`。等待期间被你解绑的身份不会再收到推送：该条投递直接落为 `failed`。目前主动推送由企业微信实现；其它已绑定渠道会入队，但在实现 `send_notification` 前会失败。
|
||||
- 按每页 50 条浏览执行历史；历史页暂停自动刷新，可随时返回最新记录。 仅在读取成功后显示条数，加载中或失败不会误显示为零条。
|
||||
|
||||
**通过 API 筛选执行历史**
|
||||
|
||||
@ -86,6 +86,26 @@ def redact_egress_text(text: str) -> str:
|
||||
return redacted
|
||||
|
||||
|
||||
# Readable text for host-defined goal reason codes (``blocked:`` prefix
|
||||
# removed). Keys mirror the goal blockers and stand-down reasons.
|
||||
_GOAL_REASON_TEXT = {
|
||||
"missing_evidence": "the goal check found evidence missing",
|
||||
"needs_user_input": "it needs your input",
|
||||
"run_failed": "the goal check found the work unfinished",
|
||||
"external_wait": "it is waiting on something external",
|
||||
"goal_not_met_yet": "the goal is not met yet",
|
||||
"no_verdict": "no goal verdict was recorded",
|
||||
"consecutive_unmet": "3 scheduled runs in a row did not meet the goal",
|
||||
"evaluator_failed": "the goal check could not run",
|
||||
"max_continuations_reached": "the continuation limit was reached",
|
||||
"no_progress_detected": "no progress was made between turns",
|
||||
"token_capped": "the token budget was reached",
|
||||
"no_durable_end_of_turn": "no final reply was saved",
|
||||
"thread_changed_after_evaluation": "the conversation changed during the goal check",
|
||||
"thread_changed_before_continuation": "the conversation changed during the goal check",
|
||||
}
|
||||
|
||||
|
||||
def render_notification_text(delivery: dict[str, Any]) -> str:
|
||||
"""Render a bounded markdown summary of the run outcome for IM push."""
|
||||
payload = delivery.get("payload") or {}
|
||||
@ -94,15 +114,12 @@ def render_notification_text(delivery: dict[str, Any]) -> str:
|
||||
if event in {"run_unmet", "task_paused"}:
|
||||
label = "Scheduled task goal was not met" if event == "run_unmet" else "Scheduled task automatically paused"
|
||||
lines = [f"**{label}**", f"Task: `{task_label}`"]
|
||||
# These are host-defined result codes, never model/provider error text.
|
||||
# Only host-defined result codes are translated; model/provider error
|
||||
# text and unknown codes are never forwarded.
|
||||
reason = payload.get("reason_code")
|
||||
blockers = {"missing_evidence", "needs_user_input", "run_failed", "external_wait", "goal_not_met_yet"}
|
||||
allowed = (
|
||||
blockers
|
||||
| {f"blocked:{blocker}" for blocker in blockers}
|
||||
| {"no_verdict", "consecutive_unmet", "evaluator_failed", "max_continuations_reached", "no_progress_detected", "token_capped", "no_durable_end_of_turn", "thread_changed_after_evaluation", "thread_changed_before_continuation"}
|
||||
)
|
||||
lines.append(f"Reason: `{reason if isinstance(reason, str) and reason in allowed else 'unknown'}`")
|
||||
if isinstance(reason, str):
|
||||
reason = reason.removeprefix("blocked:")
|
||||
lines.append(f"Reason: {_GOAL_REASON_TEXT.get(reason, 'unknown') if isinstance(reason, str) else 'unknown'}.")
|
||||
lines.append("See the DeerFlow workspace for the result and next steps.")
|
||||
elif event == "run_failed":
|
||||
lines = ["**Scheduled task failed**", f"Task: `{task_label}`"]
|
||||
|
||||
@ -492,6 +492,7 @@ The scheduled-task MVP adds a scheduler section to `config.yaml`:
|
||||
```yaml
|
||||
scheduler:
|
||||
enabled: false
|
||||
tool_enabled: false
|
||||
multi_instance: false
|
||||
poll_interval_seconds: 5
|
||||
lease_seconds: 120
|
||||
@ -504,6 +505,7 @@ scheduler:
|
||||
Notes:
|
||||
|
||||
- `enabled: false` keeps background polling off by default.
|
||||
- `tool_enabled: false` keeps conversation schedule tools off. Set it together with `enabled: true` and restart Gateway to offer `schedule_task` in authorized interactive conversations and `stop_scheduled_task` in their scheduled runs (see "Create schedules in a conversation" in the README).
|
||||
- `multi_instance: true` opts into lease-aware scheduler recovery across Gateway instances. It requires Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; otherwise startup fails fast. Leave it false for the default single-instance scheduler.
|
||||
- `max_concurrent_runs` is a shared global execution cap in multi-instance mode. Waiting `queued` rows do not consume capacity; an atomic `queued` → `launching` claim counts `launching`/`running` rows under a Postgres advisory lock so concurrent Pods cannot exceed the cap.
|
||||
- `queue_timeout_seconds` limits how long a persisted occurrence may wait for capacity or a reused thread to become available. Expired occurrences are marked `failed`; queued rows otherwise survive Gateway restarts.
|
||||
@ -522,7 +524,7 @@ Notes:
|
||||
- Create/update accept `once`, `cron`, and `interval`. Interval uses `schedule_spec.every_seconds` (UTC `now + N`, no missed-beat catch-up). N is at least `min_once_delay_seconds` (default 60) and at most 30 days.
|
||||
- Manual trigger uses the same scheduled-task resource and run lifecycle.
|
||||
- Scheduled task definitions and task-run history are persisted in the application database.
|
||||
- With `channel_connections.enabled: true`, the scheduler enqueues IM outcome notifications for the task owner's connected identities. A delivery worker (same poll cadence as the scheduler) pushes them. Only runs that finish as success or failed notify: manual triggers and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park rows without consuming the retry budget, for up to about a day; platform rejections exhaust ~15 minutes of counted retries then settle `failed`. The worker re-checks the binding right before sending: a target the owner has disconnected since enqueue is dropped as `failed`, never pushed. Only WeCom currently implements proactive `send_notification`.
|
||||
- With `channel_connections.enabled: true`, the scheduler enqueues IM outcome notifications for the task owner's connected identities. A delivery worker (same poll cadence as the scheduler) pushes them. Scheduled runs that finish as success or failed notify. For goal tasks, an `unmet` scheduled occurrence sends a goal-unmet notice instead, and an automatic pause after three unmet occurrences sends an auto-pause notice; a stop requested by the agent adds no separate notice. Manual triggers and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park rows without consuming the retry budget, for up to about a day; platform rejections exhaust ~15 minutes of counted retries then settle `failed`. The worker re-checks the binding right before sending: a target the owner has disconnected since enqueue is dropped as `failed`, never pushed. Only WeCom currently implements proactive `send_notification`.
|
||||
|
||||
### Agent Storage
|
||||
|
||||
|
||||
@ -19,6 +19,11 @@ if TYPE_CHECKING:
|
||||
|
||||
FinalizationObserver = Callable[..., Awaitable[None]]
|
||||
|
||||
# Host-written task ``last_error`` values that the tasks page translates;
|
||||
# contracts/scheduled_goal_notes_contract.json pins them for the frontend.
|
||||
AGENT_STOP_LAST_ERROR_PREFIX = "stopped by the agent in run "
|
||||
AUTO_PAUSE_LAST_ERROR = "paused after 3 unmet scheduled goal runs"
|
||||
|
||||
|
||||
def utc(value: datetime) -> datetime:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
@ -117,11 +122,11 @@ async def finalize_occurrence(
|
||||
task.next_run_at = None
|
||||
elif occurrence.stop_requested_run_id is not None and occurrence.stop_requested_run_id == run_id:
|
||||
task.status = "paused"
|
||||
task.last_error = f"stopped by the agent in run {run_id}"
|
||||
task.last_error = f"{AGENT_STOP_LAST_ERROR_PREFIX}{run_id}"
|
||||
elif occurrence.goal_objective is not None and await _unmet_streak(session, occurrence):
|
||||
was_paused = task.status == "paused"
|
||||
task.status = "paused"
|
||||
task.last_error = "paused after 3 unmet scheduled goal runs"
|
||||
task.last_error = AUTO_PAUSE_LAST_ERROR
|
||||
if not was_paused:
|
||||
events += ("task_paused",)
|
||||
task.updated_at = finished_at
|
||||
|
||||
57
backend/tests/test_scheduled_goal_notes_contract.py
Normal file
57
backend/tests/test_scheduled_goal_notes_contract.py
Normal file
@ -0,0 +1,57 @@
|
||||
"""Contract tests for host-written scheduled goal strings.
|
||||
|
||||
The tasks page (``frontend/src/core/scheduled-tasks/goal-outcome.ts``) and the
|
||||
IM notice renderer translate these values instead of showing them raw, so the
|
||||
backend must keep producing exactly what ``contracts/scheduled_goal_notes_contract.json``
|
||||
lists.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.scheduler.notification_delivery import render_notification_text
|
||||
from deerflow.persistence.scheduled_task_runs import finalization
|
||||
from deerflow.runtime.goal import DEFAULT_MAX_GOAL_CONTINUATIONS, DEFAULT_MAX_NO_PROGRESS_CONTINUATIONS, GOAL_BLOCKERS
|
||||
from deerflow.runtime.runs import worker
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
_CONTRACT = json.loads((_REPO_ROOT / "contracts" / "scheduled_goal_notes_contract.json").read_text(encoding="utf-8"))
|
||||
_CODES = set(_CONTRACT["unmet_reason_codes"])
|
||||
|
||||
|
||||
def test_last_error_strings_match_contract():
|
||||
assert finalization.AGENT_STOP_LAST_ERROR_PREFIX == _CONTRACT["agent_stop_last_error_prefix"]
|
||||
assert finalization.AUTO_PAUSE_LAST_ERROR == _CONTRACT["auto_pause_last_error"]
|
||||
|
||||
|
||||
def test_blocker_stand_down_codes_are_in_contract():
|
||||
# An unsatisfied verdict never carries the "none" blocker.
|
||||
assert {f"blocked:{blocker}" for blocker in GOAL_BLOCKERS - {"none"}} <= _CODES
|
||||
|
||||
|
||||
def test_cap_stand_down_reasons_are_in_contract():
|
||||
evaluation = {"satisfied": False, "blocker": "goal_not_met_yet", "reason": ""}
|
||||
at_cap = {"continuation_count": DEFAULT_MAX_GOAL_CONTINUATIONS}
|
||||
assert worker._stand_down_reason(at_cap, evaluation, 0) in _CODES
|
||||
assert worker._stand_down_reason({"continuation_count": 0}, evaluation, DEFAULT_MAX_NO_PROGRESS_CONTINUATIONS) in _CODES
|
||||
|
||||
|
||||
def test_literal_stand_down_reasons_and_no_verdict_are_in_contract():
|
||||
source = Path(worker.__file__).read_text(encoding="utf-8")
|
||||
literals = set(re.findall(r'stand_down_reason\s*=\s*"([a-z_:]+)"', source))
|
||||
assert literals, "pattern no longer matches worker.py; update this test"
|
||||
assert literals <= _CODES
|
||||
assert '"no_verdict"' in Path(finalization.__file__).read_text(encoding="utf-8")
|
||||
assert "no_verdict" in _CODES
|
||||
|
||||
|
||||
@pytest.mark.parametrize("code", sorted(_CODES))
|
||||
def test_im_notice_translates_every_contract_code(code):
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": code}})
|
||||
assert "Reason: unknown." not in text
|
||||
assert code not in text
|
||||
@ -141,9 +141,12 @@ def test_unmet_and_pause_notifications_have_distinct_safe_text():
|
||||
unmet = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": "blocked:needs_user_input", "error": "private raw failure"}})
|
||||
paused = render_notification_text({"event": "task_paused", "task_id": "task-a", "payload": {"reason_code": "consecutive_unmet"}})
|
||||
assert "goal was not met" in unmet
|
||||
assert "needs_user_input" in unmet
|
||||
assert "Reason: it needs your input." in unmet
|
||||
assert "needs_user_input" not in unmet
|
||||
assert "private raw failure" not in unmet
|
||||
assert "automatically paused" in paused
|
||||
assert "Reason: 3 scheduled runs in a row did not meet the goal." in paused
|
||||
assert "consecutive_unmet" not in paused
|
||||
assert "completed" not in unmet + paused
|
||||
|
||||
|
||||
@ -186,7 +189,25 @@ async def test_external_wait_does_not_advance_unmet_streak(scheduler, outcomes):
|
||||
assert status == ("paused" if index == 3 else "enabled")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reason", ["token_capped", "no_durable_end_of_turn", "thread_changed_after_evaluation", "thread_changed_before_continuation"])
|
||||
def test_known_host_stand_down_reasons_are_reported(reason):
|
||||
@pytest.mark.parametrize(
|
||||
("reason", "readable"),
|
||||
[
|
||||
("token_capped", "the token budget was reached"),
|
||||
("no_durable_end_of_turn", "no final reply was saved"),
|
||||
("thread_changed_after_evaluation", "the conversation changed during the goal check"),
|
||||
("thread_changed_before_continuation", "the conversation changed during the goal check"),
|
||||
("blocked:missing_evidence", "the goal check found evidence missing"),
|
||||
("missing_evidence", "the goal check found evidence missing"),
|
||||
("no_verdict", "no goal verdict was recorded"),
|
||||
],
|
||||
)
|
||||
def test_known_host_stand_down_reasons_are_reported_readably(reason, readable):
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": reason}})
|
||||
assert f"Reason: `{reason}`" in text
|
||||
assert f"Reason: {readable}." in text
|
||||
assert reason not in text
|
||||
|
||||
|
||||
def test_unknown_reason_code_is_not_forwarded():
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": "provider said: secret"}})
|
||||
assert "Reason: unknown." in text
|
||||
assert "secret" not in text
|
||||
|
||||
21
contracts/scheduled_goal_notes_contract.json
Normal file
21
contracts/scheduled_goal_notes_contract.json
Normal file
@ -0,0 +1,21 @@
|
||||
{
|
||||
"version": 1,
|
||||
"description": "Cross-language contract fixture for the host-written scheduled-task strings the tasks page and IM notices translate. The backend writes these values to scheduled_task_runs.error and scheduled_tasks.last_error during goal and stop finalization; the frontend (core/scheduled-tasks/goal-outcome.ts) and the IM notice renderer map them to readable text, so neither shows the raw value.",
|
||||
"agent_stop_last_error_prefix": "stopped by the agent in run ",
|
||||
"auto_pause_last_error": "paused after 3 unmet scheduled goal runs",
|
||||
"unmet_reason_codes": [
|
||||
"blocked:missing_evidence",
|
||||
"blocked:needs_user_input",
|
||||
"blocked:run_failed",
|
||||
"blocked:external_wait",
|
||||
"blocked:goal_not_met_yet",
|
||||
"max_continuations_reached",
|
||||
"no_progress_detected",
|
||||
"token_capped",
|
||||
"evaluator_failed",
|
||||
"no_durable_end_of_turn",
|
||||
"thread_changed_after_evaluation",
|
||||
"thread_changed_before_continuation",
|
||||
"no_verdict"
|
||||
]
|
||||
}
|
||||
@ -54,6 +54,7 @@ page zero polls or refreshes on focus/reconnect. Task switches reset to page zer
|
||||
and consumed AbortSignals cancel obsolete reads. Live offsets are not snapshots;
|
||||
explicit mutations or navigation may observe newly inserted runs.
|
||||
Run status `unmet` identifies a finished occurrence whose scheduled goal was not satisfied; keep it distinct from execution failure.
|
||||
`core/scheduled-tasks/goal-outcome.ts` maps goal verdicts and host reason codes for run history; show known codes as localized labels (raw code only in the tooltip), unknown codes verbatim, and leave runs without a goal unchanged. `contracts/scheduled_goal_notes_contract.json` pins the host strings it matches.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@ -37,6 +37,11 @@ import { listAgents } from "@/core/agents/api";
|
||||
import { useAgentsApiEnabled } from "@/core/agents/hooks";
|
||||
import { useI18n } from "@/core/i18n/hooks";
|
||||
import { hasScheduleSpec } from "@/core/scheduled-tasks/cron";
|
||||
import {
|
||||
describeGoalOutcome,
|
||||
describeTaskLastError,
|
||||
requestedScheduleStop,
|
||||
} from "@/core/scheduled-tasks/goal-outcome";
|
||||
import {
|
||||
useCreateScheduledTask,
|
||||
useUpdateScheduledTask,
|
||||
@ -193,6 +198,7 @@ export default function ScheduledTasksPage() {
|
||||
const selectedTask =
|
||||
filteredData.find((task) => task.id === selectedTaskId) ?? filteredData[0];
|
||||
const taskRunsQuery = useScheduledTaskRunHistory(selectedTask?.id);
|
||||
const lastNote = describeTaskLastError(selectedTask?.last_error ?? null);
|
||||
const createTask = useCreateScheduledTask();
|
||||
const updateTask = useUpdateScheduledTask(selectedTask?.id ?? "");
|
||||
const pauseTask = usePauseScheduledTask();
|
||||
@ -630,6 +636,28 @@ export default function ScheduledTasksPage() {
|
||||
{st.detail.schedule}:{" "}
|
||||
{scheduleTypeLabel(selectedTask.schedule_type)}
|
||||
</div>
|
||||
{selectedTask.goal_objective && (
|
||||
<div
|
||||
className="text-muted-foreground text-sm break-words whitespace-pre-wrap"
|
||||
data-testid="scheduled-task-goal"
|
||||
>
|
||||
{st.goal.objective}: {selectedTask.goal_objective}
|
||||
</div>
|
||||
)}
|
||||
{selectedTask.max_runs != null && (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.goal.maxRuns.replace(
|
||||
"{count}",
|
||||
String(selectedTask.max_runs),
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{selectedTask.end_at && (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.goal.endAt}:{" "}
|
||||
{formatTimestamp(selectedTask.end_at, locale)}
|
||||
</div>
|
||||
)}
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.detail.nextRun}:{" "}
|
||||
{formatTimestamp(selectedTask.next_run_at, locale)}
|
||||
@ -641,8 +669,20 @@ export default function ScheduledTasksPage() {
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.detail.lastRunId}: {selectedTask.last_run_id ?? NONE}
|
||||
</div>
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.detail.lastError}: {selectedTask.last_error ?? NONE}
|
||||
<div
|
||||
className="text-muted-foreground text-sm"
|
||||
data-testid="scheduled-task-last-error"
|
||||
title={
|
||||
lastNote
|
||||
? (selectedTask.last_error ?? undefined)
|
||||
: undefined
|
||||
}
|
||||
>
|
||||
{lastNote === null
|
||||
? `${st.detail.lastError}: ${selectedTask.last_error ?? NONE}`
|
||||
: lastNote.kind === "goalUnmet"
|
||||
? `${st.goal.lastUnmet}: ${st.goal.reasons[lastNote.reasonKey]}`
|
||||
: `${st.goal.lastPause}: ${lastNote.kind === "agentStop" ? st.goal.agentStopped : st.goal.autoPaused}`}
|
||||
</div>
|
||||
{editing ? (
|
||||
<div className="flex flex-col gap-2 rounded-lg border p-3">
|
||||
@ -826,25 +866,58 @@ export default function ScheduledTasksPage() {
|
||||
data-testid="scheduled-task-run-list"
|
||||
>
|
||||
{(taskRunsQuery.data ?? []).length > 0 ? (
|
||||
(taskRunsQuery.data ?? []).map((run) => (
|
||||
<div
|
||||
key={run.id}
|
||||
className="rounded-md border p-3 text-sm"
|
||||
>
|
||||
<div className="font-medium">{runSummary(run)}</div>
|
||||
<div className="text-muted-foreground text-xs">
|
||||
{run.run_id ?? NONE}
|
||||
</div>
|
||||
<div className="text-muted-foreground text-xs">
|
||||
{formatTimestamp(run.scheduled_for, locale)}
|
||||
</div>
|
||||
{run.error && (
|
||||
<div className="text-destructive text-xs">
|
||||
{run.error}
|
||||
(taskRunsQuery.data ?? []).map((run) => {
|
||||
const goal = describeGoalOutcome(run);
|
||||
return (
|
||||
<div
|
||||
key={run.id}
|
||||
className="rounded-md border p-3 text-sm"
|
||||
>
|
||||
<div className="font-medium">{runSummary(run)}</div>
|
||||
<div className="text-muted-foreground text-xs">
|
||||
{run.run_id ?? NONE}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
))
|
||||
<div className="text-muted-foreground text-xs">
|
||||
{formatTimestamp(run.scheduled_for, locale)}
|
||||
</div>
|
||||
{goal?.kind === "met" && (
|
||||
<div
|
||||
className="text-xs"
|
||||
data-testid="scheduled-run-goal"
|
||||
>
|
||||
{goal.reliedOnAssumption
|
||||
? st.goal.metAssumed
|
||||
: st.goal.met}
|
||||
</div>
|
||||
)}
|
||||
{goal?.kind === "unmet"
|
||||
? goal.code && (
|
||||
<div
|
||||
className="text-xs text-amber-700 dark:text-amber-400"
|
||||
data-testid="scheduled-run-goal"
|
||||
title={goal.code}
|
||||
>
|
||||
{goal.reasonKey
|
||||
? st.goal.reasons[goal.reasonKey]
|
||||
: goal.code}
|
||||
</div>
|
||||
)
|
||||
: run.error && (
|
||||
<div className="text-destructive text-xs">
|
||||
{run.error}
|
||||
</div>
|
||||
)}
|
||||
{requestedScheduleStop(run) && (
|
||||
<div
|
||||
className="text-muted-foreground text-xs"
|
||||
data-testid="scheduled-run-stop-requested"
|
||||
>
|
||||
{st.goal.stopRequested}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})
|
||||
) : !taskRunsQuery.isPending && !taskRunsQuery.isError ? (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
{st.detail.noRuns}
|
||||
|
||||
@ -944,6 +944,32 @@ export const enUS: Translations = {
|
||||
skipped: "Skipped",
|
||||
interrupted: "Interrupted",
|
||||
},
|
||||
goal: {
|
||||
objective: "Goal per run",
|
||||
maxRuns: "Automatic runs: up to {count}",
|
||||
endAt: "Ends at",
|
||||
met: "Goal met",
|
||||
metAssumed: "Goal met, relying on stated assumptions",
|
||||
stopRequested: "This run asked to stop the schedule",
|
||||
lastPause: "Last pause reason",
|
||||
agentStopped: "The agent stopped its own schedule",
|
||||
autoPaused: "3 scheduled runs in a row did not meet the goal",
|
||||
lastUnmet: "Last unmet reason",
|
||||
reasons: {
|
||||
missingEvidence: "Goal check: evidence missing",
|
||||
needsUserInput: "Needs your input",
|
||||
externalWait: "Goal check: waiting on something external",
|
||||
runFailed: "Goal check: the run did not finish the work",
|
||||
goalNotMetYet: "Goal check: not met yet",
|
||||
maxContinuations: "Continuation limit reached",
|
||||
noProgress: "No progress between turns",
|
||||
tokenCapped: "Token budget reached",
|
||||
evaluatorFailed: "Goal check could not run",
|
||||
noDurableEndOfTurn: "No final reply was saved",
|
||||
threadChanged: "Conversation changed during the goal check",
|
||||
noVerdict: "No goal verdict",
|
||||
},
|
||||
},
|
||||
recipes: {
|
||||
label: "Quick create",
|
||||
trending: {
|
||||
|
||||
@ -780,6 +780,32 @@ export interface Translations {
|
||||
skipped: string;
|
||||
interrupted: string;
|
||||
};
|
||||
goal: {
|
||||
objective: string;
|
||||
maxRuns: string;
|
||||
endAt: string;
|
||||
met: string;
|
||||
metAssumed: string;
|
||||
stopRequested: string;
|
||||
lastPause: string;
|
||||
agentStopped: string;
|
||||
autoPaused: string;
|
||||
lastUnmet: string;
|
||||
reasons: {
|
||||
missingEvidence: string;
|
||||
needsUserInput: string;
|
||||
externalWait: string;
|
||||
runFailed: string;
|
||||
goalNotMetYet: string;
|
||||
maxContinuations: string;
|
||||
noProgress: string;
|
||||
tokenCapped: string;
|
||||
evaluatorFailed: string;
|
||||
noDurableEndOfTurn: string;
|
||||
threadChanged: string;
|
||||
noVerdict: string;
|
||||
};
|
||||
};
|
||||
recipes: {
|
||||
label: string;
|
||||
trending: { title: string; desc: string };
|
||||
|
||||
@ -885,6 +885,32 @@ export const zhCN: Translations = {
|
||||
skipped: "跳过",
|
||||
interrupted: "已中断",
|
||||
},
|
||||
goal: {
|
||||
objective: "每次执行的目标",
|
||||
maxRuns: "自动执行上限：{count} 次",
|
||||
endAt: "截止时间",
|
||||
met: "目标已达成",
|
||||
metAssumed: "目标已达成（基于已声明的假设）",
|
||||
stopRequested: "本次执行请求停止该定时任务",
|
||||
lastPause: "上次暂停原因",
|
||||
agentStopped: "Agent 主动停止了该定时任务",
|
||||
autoPaused: "连续 3 次定时执行未达成目标",
|
||||
lastUnmet: "上次未达成原因",
|
||||
reasons: {
|
||||
missingEvidence: "目标检查：缺少证据",
|
||||
needsUserInput: "需要你补充信息",
|
||||
externalWait: "目标检查：正在等待外部条件",
|
||||
runFailed: "目标检查：本次执行未完成任务",
|
||||
goalNotMetYet: "目标检查：尚未达成",
|
||||
maxContinuations: "已达到续跑次数上限",
|
||||
noProgress: "连续多轮没有进展",
|
||||
tokenCapped: "已达到 token 预算",
|
||||
evaluatorFailed: "目标检查未能完成",
|
||||
noDurableEndOfTurn: "没有保存最终回复",
|
||||
threadChanged: "目标检查期间对话发生了变化",
|
||||
noVerdict: "未得到目标判定",
|
||||
},
|
||||
},
|
||||
recipes: {
|
||||
label: "快速创建",
|
||||
trending: {
|
||||
|
||||
100
frontend/src/core/scheduled-tasks/goal-outcome.ts
Normal file
100
frontend/src/core/scheduled-tasks/goal-outcome.ts
Normal file
@ -0,0 +1,100 @@
|
||||
import type { ScheduledTaskRun } from "./types";
|
||||
|
||||
export type GoalReasonKey =
|
||||
| "missingEvidence"
|
||||
| "needsUserInput"
|
||||
| "externalWait"
|
||||
| "runFailed"
|
||||
| "goalNotMetYet"
|
||||
| "maxContinuations"
|
||||
| "noProgress"
|
||||
| "tokenCapped"
|
||||
| "evaluatorFailed"
|
||||
| "noDurableEndOfTurn"
|
||||
| "threadChanged"
|
||||
| "noVerdict";
|
||||
|
||||
// Host-defined codes an `unmet` occurrence stores in `error`, keyed without
|
||||
// the `blocked:` prefix like the IM notice map (`_GOAL_REASON_TEXT`).
|
||||
// contracts/scheduled_goal_notes_contract.json pins the codes the backend
|
||||
// writes; unknown codes stay visible verbatim.
|
||||
const REASON_KEYS: Record<string, GoalReasonKey> = {
|
||||
missing_evidence: "missingEvidence",
|
||||
needs_user_input: "needsUserInput",
|
||||
external_wait: "externalWait",
|
||||
run_failed: "runFailed",
|
||||
goal_not_met_yet: "goalNotMetYet",
|
||||
max_continuations_reached: "maxContinuations",
|
||||
no_progress_detected: "noProgress",
|
||||
token_capped: "tokenCapped",
|
||||
evaluator_failed: "evaluatorFailed",
|
||||
no_durable_end_of_turn: "noDurableEndOfTurn",
|
||||
thread_changed_after_evaluation: "threadChanged",
|
||||
thread_changed_before_continuation: "threadChanged",
|
||||
no_verdict: "noVerdict",
|
||||
};
|
||||
|
||||
// Host-written task `last_error` values, pinned by the same contract.
|
||||
const AGENT_STOP_LAST_ERROR_PREFIX = "stopped by the agent in run ";
|
||||
const AUTO_PAUSE_LAST_ERROR = "paused after 3 unmet scheduled goal runs";
|
||||
|
||||
function reasonKeyOf(code: string): GoalReasonKey | null {
|
||||
return REASON_KEYS[code.replace(/^blocked:/, "")] ?? null;
|
||||
}
|
||||
|
||||
export type GoalOutcome =
|
||||
| { kind: "met"; reliedOnAssumption: boolean }
|
||||
| { kind: "unmet"; code: string | null; reasonKey: GoalReasonKey | null };
|
||||
|
||||
/** Goal result of a finished occurrence; null when the run had no goal outcome. */
|
||||
export function describeGoalOutcome(
|
||||
run: Pick<ScheduledTaskRun, "status" | "error" | "goal_verdict">,
|
||||
): GoalOutcome | null {
|
||||
if (run.status === "unmet") {
|
||||
const code = run.error ?? null;
|
||||
return { kind: "unmet", code, reasonKey: code ? reasonKeyOf(code) : null };
|
||||
}
|
||||
if (run.status === "success" && run.goal_verdict?.satisfied === true) {
|
||||
return {
|
||||
kind: "met",
|
||||
reliedOnAssumption: run.goal_verdict.relied_on_assumption === true,
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** True for the occurrence whose agent asked to stop its own schedule. */
|
||||
export function requestedScheduleStop(
|
||||
run: Pick<ScheduledTaskRun, "run_id" | "stop_requested_run_id">,
|
||||
): boolean {
|
||||
return Boolean(run.run_id) && run.stop_requested_run_id === run.run_id;
|
||||
}
|
||||
|
||||
export type TaskLastNote =
|
||||
| { kind: "agentStop" }
|
||||
| { kind: "autoPause" }
|
||||
| { kind: "goalUnmet"; reasonKey: GoalReasonKey };
|
||||
|
||||
/**
|
||||
* Recognize the host-written `last_error` values from goal and stop
|
||||
* finalization (backend `scheduled_task_runs/finalization.py`). Anything else
|
||||
* is a real error message and returns null.
|
||||
*/
|
||||
export function describeTaskLastError(
|
||||
lastError: string | null,
|
||||
): TaskLastNote | null {
|
||||
if (!lastError) {
|
||||
return null;
|
||||
}
|
||||
if (
|
||||
lastError.startsWith(AGENT_STOP_LAST_ERROR_PREFIX) &&
|
||||
/^\S+$/.test(lastError.slice(AGENT_STOP_LAST_ERROR_PREFIX.length))
|
||||
) {
|
||||
return { kind: "agentStop" };
|
||||
}
|
||||
if (lastError === AUTO_PAUSE_LAST_ERROR) {
|
||||
return { kind: "autoPause" };
|
||||
}
|
||||
const reasonKey = reasonKeyOf(lastError);
|
||||
return reasonKey ? { kind: "goalUnmet", reasonKey } : null;
|
||||
}
|
||||
@ -21,10 +21,23 @@ export type ScheduledTask = {
|
||||
last_thread_id: string | null;
|
||||
last_error: string | null;
|
||||
run_count: number;
|
||||
// Only conversation-created tasks set these; other tasks omit them or send null.
|
||||
goal_objective?: string | null;
|
||||
max_runs?: number | null;
|
||||
end_at?: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
// The verdict an occurrence finished with: from the goal evaluator, or from
|
||||
// the host when it stood the goal down (see `stand_down_reason`).
|
||||
export type ScheduledGoalVerdict = {
|
||||
satisfied?: boolean;
|
||||
blocker?: string;
|
||||
relied_on_assumption?: boolean;
|
||||
stand_down_reason?: string;
|
||||
};
|
||||
|
||||
export type ScheduledTaskRun = {
|
||||
id: string;
|
||||
task_id: string;
|
||||
@ -42,6 +55,9 @@ export type ScheduledTaskRun = {
|
||||
| "skipped"
|
||||
| "interrupted";
|
||||
error: string | null;
|
||||
goal_objective?: string | null;
|
||||
goal_verdict?: ScheduledGoalVerdict | null;
|
||||
stop_requested_run_id?: string | null;
|
||||
attempt_count: number;
|
||||
started_at: string | null;
|
||||
finished_at: string | null;
|
||||
|
||||
173
frontend/tests/e2e/scheduled-goal-history.spec.ts
Normal file
173
frontend/tests/e2e/scheduled-goal-history.spec.ts
Normal file
@ -0,0 +1,173 @@
|
||||
import { expect, test, type Page } from "@playwright/test";
|
||||
|
||||
import { mockLangGraphAPI } from "./utils/mock-api";
|
||||
|
||||
const baseTask = {
|
||||
thread_id: null,
|
||||
context_mode: "fresh_thread_per_run" as const,
|
||||
prompt: "Check the release checklist",
|
||||
schedule_type: "interval" as const,
|
||||
schedule_spec: { every_seconds: 3600 },
|
||||
timezone: "UTC",
|
||||
status: "paused" as const,
|
||||
next_run_at: null,
|
||||
last_run_at: null,
|
||||
last_run_id: null,
|
||||
last_error: null,
|
||||
run_count: 5,
|
||||
created_at: "2026-10-01T00:00:00Z",
|
||||
updated_at: "2026-10-01T00:00:00Z",
|
||||
};
|
||||
const goalTask = {
|
||||
...baseTask,
|
||||
id: "goal-task",
|
||||
title: "Goal task",
|
||||
goal_objective: "status.md lists every unchecked item",
|
||||
max_runs: 5,
|
||||
end_at: null,
|
||||
last_error: "stopped by the agent in run execution-stop",
|
||||
};
|
||||
const plainTask = {
|
||||
...baseTask,
|
||||
id: "plain-task",
|
||||
title: "Plain task",
|
||||
last_error: "boom",
|
||||
};
|
||||
|
||||
const run = (
|
||||
id: string,
|
||||
overrides: Record<string, unknown>,
|
||||
): Record<string, unknown> => ({
|
||||
id,
|
||||
task_id: "goal-task",
|
||||
thread_id: `thread-${id}`,
|
||||
run_id: `execution-${id}`,
|
||||
scheduled_for: "2026-10-01T00:00:00Z",
|
||||
trigger: "scheduled",
|
||||
status: "success",
|
||||
error: null,
|
||||
attempt_count: 1,
|
||||
started_at: null,
|
||||
finished_at: null,
|
||||
created_at: "2026-10-01T00:00:00Z",
|
||||
...overrides,
|
||||
});
|
||||
const goal = { goal_objective: goalTask.goal_objective };
|
||||
const goalRuns = [
|
||||
run("met", { ...goal, goal_verdict: { satisfied: true } }),
|
||||
run("assumed", {
|
||||
...goal,
|
||||
goal_verdict: { satisfied: true, relied_on_assumption: true },
|
||||
}),
|
||||
run("unmet", {
|
||||
...goal,
|
||||
status: "unmet",
|
||||
error: "blocked:missing_evidence",
|
||||
goal_verdict: {
|
||||
satisfied: false,
|
||||
blocker: "missing_evidence",
|
||||
stand_down_reason: "blocked:missing_evidence",
|
||||
},
|
||||
}),
|
||||
run("stop", {
|
||||
...goal,
|
||||
goal_verdict: { satisfied: true },
|
||||
stop_requested_run_id: "execution-stop",
|
||||
}),
|
||||
run("unknown", { ...goal, status: "unmet", error: "future_reason" }),
|
||||
run("failed", { ...goal, status: "failed", error: "boom" }),
|
||||
];
|
||||
const plainRuns = [
|
||||
run("plain-ok", { task_id: "plain-task" }),
|
||||
run("plain-failed", {
|
||||
task_id: "plain-task",
|
||||
status: "failed",
|
||||
error: "boom",
|
||||
}),
|
||||
];
|
||||
|
||||
async function openTask(
|
||||
page: Page,
|
||||
task: typeof goalTask | typeof plainTask,
|
||||
runs: Record<string, unknown>[],
|
||||
locale?: "zh-CN",
|
||||
) {
|
||||
mockLangGraphAPI(page, { threads: [], scheduledTasks: [task] });
|
||||
await page.route(/\/api\/scheduled-tasks\/[^/]+\/runs(?:\?|$)/, (route) =>
|
||||
route.fulfill({ json: runs }),
|
||||
);
|
||||
await page.goto("/workspace/scheduled-tasks");
|
||||
if (locale) {
|
||||
await page.evaluate((value) => {
|
||||
document.cookie = `locale=${value}; path=/; SameSite=Lax`;
|
||||
}, locale);
|
||||
await page.reload();
|
||||
}
|
||||
return page.getByTestId("scheduled-task-run-list");
|
||||
}
|
||||
|
||||
const rowOf = (list: ReturnType<Page["getByTestId"]>, runId: string) =>
|
||||
list.locator("div.rounded-md").filter({ hasText: runId });
|
||||
|
||||
test("goal runs show their outcome and the unmet reason is not styled as an error", async ({
|
||||
page,
|
||||
}) => {
|
||||
const list = await openTask(page, goalTask, goalRuns);
|
||||
const detail = page.getByTestId("scheduled-task-detail");
|
||||
await expect(page.getByTestId("scheduled-task-goal")).toHaveText(
|
||||
"Goal per run: status.md lists every unchecked item",
|
||||
);
|
||||
await expect(detail.getByText("Automatic runs: up to 5")).toBeVisible();
|
||||
await expect(page.getByTestId("scheduled-task-last-error")).toHaveText(
|
||||
"Last pause reason: The agent stopped its own schedule",
|
||||
);
|
||||
|
||||
await expect(rowOf(list, "execution-met")).toContainText("Goal met");
|
||||
await expect(rowOf(list, "execution-assumed")).toContainText(
|
||||
"Goal met, relying on stated assumptions",
|
||||
);
|
||||
const unmet = rowOf(list, "execution-unmet").getByTestId(
|
||||
"scheduled-run-goal",
|
||||
);
|
||||
await expect(unmet).toHaveText("Goal check: evidence missing");
|
||||
await expect(unmet).toHaveAttribute("title", "blocked:missing_evidence");
|
||||
await expect(unmet).not.toHaveClass(/text-destructive/);
|
||||
await expect(
|
||||
rowOf(list, "execution-unknown").getByTestId("scheduled-run-goal"),
|
||||
).toHaveText("future_reason");
|
||||
await expect(
|
||||
rowOf(list, "execution-stop").getByTestId("scheduled-run-stop-requested"),
|
||||
).toHaveText("This run asked to stop the schedule");
|
||||
await expect(list.getByTestId("scheduled-run-stop-requested")).toHaveCount(1);
|
||||
await expect(
|
||||
rowOf(list, "execution-failed").locator(".text-destructive"),
|
||||
).toHaveText("boom");
|
||||
});
|
||||
|
||||
test("tasks without a goal render their runs as before", async ({ page }) => {
|
||||
const list = await openTask(page, plainTask, plainRuns);
|
||||
await expect(list.getByText("execution-plain-ok")).toBeVisible();
|
||||
await expect(page.getByTestId("scheduled-task-goal")).toHaveCount(0);
|
||||
await expect(page.getByTestId("scheduled-task-last-error")).toHaveText(
|
||||
"Last error: boom",
|
||||
);
|
||||
await expect(list.getByTestId("scheduled-run-goal")).toHaveCount(0);
|
||||
await expect(list.getByTestId("scheduled-run-stop-requested")).toHaveCount(0);
|
||||
await expect(
|
||||
rowOf(list, "execution-plain-failed").locator(".text-destructive"),
|
||||
).toHaveText("boom");
|
||||
});
|
||||
|
||||
test("goal outcomes use the Chinese copy", async ({ page }) => {
|
||||
const list = await openTask(page, goalTask, goalRuns, "zh-CN");
|
||||
await expect(page.getByTestId("scheduled-task-goal")).toHaveText(
|
||||
"每次执行的目标: status.md lists every unchecked item",
|
||||
);
|
||||
await expect(rowOf(list, "execution-met")).toContainText("目标已达成");
|
||||
await expect(page.getByTestId("scheduled-task-last-error")).toHaveText(
|
||||
"上次暂停原因: Agent 主动停止了该定时任务",
|
||||
);
|
||||
await expect(
|
||||
rowOf(list, "execution-unmet").getByTestId("scheduled-run-goal"),
|
||||
).toHaveText("目标检查：缺少证据");
|
||||
});
|
||||
158
frontend/tests/unit/core/scheduled-tasks/goal-outcome.test.ts
Normal file
158
frontend/tests/unit/core/scheduled-tasks/goal-outcome.test.ts
Normal file
@ -0,0 +1,158 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
import { expect, test } from "@rstest/core";
|
||||
|
||||
import {
|
||||
describeGoalOutcome,
|
||||
describeTaskLastError,
|
||||
requestedScheduleStop,
|
||||
} from "@/core/scheduled-tasks/goal-outcome";
|
||||
import type { ScheduledTaskRun } from "@/core/scheduled-tasks/types";
|
||||
|
||||
type OutcomeInput = Pick<ScheduledTaskRun, "status" | "error" | "goal_verdict">;
|
||||
|
||||
type ContractFile = {
|
||||
agent_stop_last_error_prefix: string;
|
||||
auto_pause_last_error: string;
|
||||
unmet_reason_codes: string[];
|
||||
};
|
||||
|
||||
const CONTRACT = JSON.parse(
|
||||
readFileSync(
|
||||
resolve(
|
||||
__dirname,
|
||||
"../../../../../contracts/scheduled_goal_notes_contract.json",
|
||||
),
|
||||
"utf-8",
|
||||
),
|
||||
) as ContractFile;
|
||||
|
||||
const run = (overrides: Partial<OutcomeInput>): OutcomeInput => ({
|
||||
status: "success",
|
||||
error: null,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
test("a run without a goal verdict has no goal outcome", () => {
|
||||
expect(describeGoalOutcome(run({}))).toBeNull();
|
||||
expect(describeGoalOutcome(run({ goal_verdict: null }))).toBeNull();
|
||||
expect(
|
||||
describeGoalOutcome(run({ status: "failed", error: "boom" })),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
test("a satisfied verdict is met, with the assumption flag kept", () => {
|
||||
expect(
|
||||
describeGoalOutcome(run({ goal_verdict: { satisfied: true } })),
|
||||
).toEqual({ kind: "met", reliedOnAssumption: false });
|
||||
expect(
|
||||
describeGoalOutcome(
|
||||
run({
|
||||
goal_verdict: { satisfied: true, relied_on_assumption: true },
|
||||
}),
|
||||
),
|
||||
).toEqual({ kind: "met", reliedOnAssumption: true });
|
||||
});
|
||||
|
||||
test("a success with an unsatisfied verdict is not reported as met", () => {
|
||||
expect(
|
||||
describeGoalOutcome(run({ goal_verdict: { satisfied: false } })),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
test.each([
|
||||
["blocked:missing_evidence", "missingEvidence"],
|
||||
["blocked:needs_user_input", "needsUserInput"],
|
||||
["blocked:external_wait", "externalWait"],
|
||||
["blocked:run_failed", "runFailed"],
|
||||
["blocked:goal_not_met_yet", "goalNotMetYet"],
|
||||
["max_continuations_reached", "maxContinuations"],
|
||||
["no_progress_detected", "noProgress"],
|
||||
["token_capped", "tokenCapped"],
|
||||
["evaluator_failed", "evaluatorFailed"],
|
||||
["no_durable_end_of_turn", "noDurableEndOfTurn"],
|
||||
["thread_changed_after_evaluation", "threadChanged"],
|
||||
["thread_changed_before_continuation", "threadChanged"],
|
||||
["no_verdict", "noVerdict"],
|
||||
])("unmet reason %s maps to %s", (code, reasonKey) => {
|
||||
expect(describeGoalOutcome(run({ status: "unmet", error: code }))).toEqual({
|
||||
kind: "unmet",
|
||||
code,
|
||||
reasonKey,
|
||||
});
|
||||
});
|
||||
|
||||
test("an unknown unmet code stays visible without a label", () => {
|
||||
expect(
|
||||
describeGoalOutcome(run({ status: "unmet", error: "future_reason" })),
|
||||
).toEqual({ kind: "unmet", code: "future_reason", reasonKey: null });
|
||||
expect(describeGoalOutcome(run({ status: "unmet", error: null }))).toEqual({
|
||||
kind: "unmet",
|
||||
code: null,
|
||||
reasonKey: null,
|
||||
});
|
||||
});
|
||||
|
||||
test.each([
|
||||
["run-2", "run-2", true],
|
||||
["run-2", "run-1", false],
|
||||
["run-2", null, false],
|
||||
[null, null, false],
|
||||
])(
|
||||
"run %s with stop request %s reports a stop request: %s",
|
||||
(runId, stopRequestedRunId, expected) => {
|
||||
expect(
|
||||
requestedScheduleStop({
|
||||
run_id: runId,
|
||||
stop_requested_run_id: stopRequestedRunId,
|
||||
}),
|
||||
).toBe(expected);
|
||||
},
|
||||
);
|
||||
|
||||
test.each([
|
||||
[
|
||||
"stopped by the agent in run 3f2a9c1e-8b47-4d2a-9e61-5c0b7a1d4e93",
|
||||
{ kind: "agentStop" },
|
||||
],
|
||||
["paused after 3 unmet scheduled goal runs", { kind: "autoPause" }],
|
||||
[
|
||||
"blocked:missing_evidence",
|
||||
{ kind: "goalUnmet", reasonKey: "missingEvidence" },
|
||||
],
|
||||
["no_verdict", { kind: "goalUnmet", reasonKey: "noVerdict" }],
|
||||
])("host-written last_error %s is recognized", (lastError, expected) => {
|
||||
expect(describeTaskLastError(lastError)).toEqual(expected);
|
||||
});
|
||||
|
||||
test.each([
|
||||
[null],
|
||||
[""],
|
||||
["Connection reset by peer"],
|
||||
["stopped by the agent in run "],
|
||||
["Paused after 3 unmet scheduled goal runs"],
|
||||
["stopped by the agent in run abc; then failed"],
|
||||
])("other last_error %s stays a plain error message", (lastError) => {
|
||||
expect(describeTaskLastError(lastError)).toBeNull();
|
||||
});
|
||||
|
||||
test("every contract reason code has a label in run history and task detail", () => {
|
||||
for (const code of CONTRACT.unmet_reason_codes) {
|
||||
const outcome = describeGoalOutcome(run({ status: "unmet", error: code }));
|
||||
expect(outcome?.kind).toBe("unmet");
|
||||
expect(outcome?.kind === "unmet" ? outcome.reasonKey : null).not.toBeNull();
|
||||
expect(describeTaskLastError(code)?.kind).toBe("goalUnmet");
|
||||
}
|
||||
});
|
||||
|
||||
test("contract last_error strings are recognized", () => {
|
||||
expect(
|
||||
describeTaskLastError(
|
||||
`${CONTRACT.agent_stop_last_error_prefix}3f2a9c1e-8b47-4d2a-9e61-5c0b7a1d4e93`,
|
||||
),
|
||||
).toEqual({ kind: "agentStop" });
|
||||
expect(describeTaskLastError(CONTRACT.auto_pause_last_error)).toEqual({
|
||||
kind: "autoPause",
|
||||
});
|
||||
});
|
||||
Loading…
x
Reference in New Issue
Block a user