feat(scheduled-tasks): preview upcoming cron occurrences (#5381)

Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
This commit is contained in:
tiammomo 2026-09-12 14:59:04 +08:00 committed by GitHub
parent a06688ad82
commit 444bfb72ce
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 255 additions and 2 deletions

View File

@ -1592,6 +1592,18 @@ Scheduled runs use `scheduler.recursion_limit` in `config.yaml` (default `1000`,
The background scheduler is single-instance by default. For a multi-pod deployment, set `scheduler.multi_instance: true` and use shared Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; startup and periodic recovery then preserve live peer runs, atomically return expired launch claims to the queue, take over only expired run leases, and fence stale launch writes. `max_concurrent_runs` is a shared global cap across Pods for `launching`/`running` occurrences; waiting `queued` rows do not consume it. Without those settings, enable the scheduler on exactly one Gateway pod. These scheduler fields are startup-only; restart all Gateway Pods together when changing them.
### Preview cron occurrences through the API
Authenticated clients with `threads:read` can call `POST /api/scheduled-tasks/preview-cron` before creating a task:
```json
{"cron":"0 9 * * 1-5","timezone":"Asia/Shanghai","count":3,"start_at":"2026-09-12T00:00:00Z"}
```
The response contains normalized `cron`, `timezone`, the effective UTC `start_at`, and `occurrences` with UTC `run_at` and offset-bearing `local_time`. In this example the first occurrence is `2026-09-14T01:00:00Z` / `2026-09-14T09:00:00+08:00`.
`count` is an integer from 1 to 10 (default 5). `start_at` must include a timezone; omit it to capture server time once. Cron expressions use the scheduler's five-field syntax (maximum 256 characters); timezone names are at most 128 characters. Invalid inputs or schedules without the requested future occurrences return 422. Preview shares the scheduler's DST behavior, creates no task, thread or run, and does not reserve execution. This is an API capability; the workspace form does not yet display these occurrences.
### Upgrade Notes
- Occurrence ordering applies to rows admitted by upgraded Gateway instances, which project only sequenced occurrences onto the parent task and defer recovery while any occurrence is still live, whichever instance admitted it; a task whose history is entirely unsequenced keeps the previous timestamp ordering until its first sequenced admission. During a rolling upgrade, rows admitted by pre-upgrade instances are projected by those instances themselves, as before the upgrade, and the ordering guarantees hold once every Gateway writer runs the upgraded version. Existing history is not backfilled; the upgrade does not reconstruct past order or repair historical counts.

View File

@ -838,6 +838,18 @@ DeerFlow 现在在 workspace 里内置了一个一等的定时任务scheduled
后台调度器默认是单实例。多 Pod 部署时,请设置 `scheduler.multi_instance: true`,并使用共享 Postgres、`run_ownership.heartbeat_enabled: true``run_events.backend: db`;启动和周期性恢复会保留仍由对端持有的运行,把过期的 launch claim 原子退回队列,只接管过期的 run lease并隔离过期的 launch 写入。`max_concurrent_runs` 是跨 Pod 共享的全局上限,只计入 `launching` / `running` 的执行;等待中的 `queued` 行不占用该配额。没有这些配置时,请只在一个 Gateway Pod 上启用调度器。这些 scheduler 字段只在启动时生效;修改后需要一起重启所有 Gateway Pod。
### 通过 API 预览 cron 执行时间
已认证且具有 `threads:read` 权限的客户端,可在创建任务前调用 `POST /api/scheduled-tasks/preview-cron`
```json
{"cron":"0 9 * * 1-5","timezone":"Asia/Shanghai","count":3,"start_at":"2026-09-12T00:00:00Z"}
```
响应包含规范化的 `cron``timezone`、生效的 UTC `start_at`,以及 `occurrences` 列表中的 UTC `run_at` 和带偏移量的 `local_time`。此例的首次执行时间为 `2026-09-14T01:00:00Z` / `2026-09-14T09:00:00+08:00`
`count` 为 110 的整数,默认 5。`start_at` 必须带时区省略时只读取一次服务器当前时间。cron 沿用调度器的五字段语法,最长 256 字符;时区名称最长 128 字符。输入无效或无法计算所需未来时间时返回 422。预览沿用实际调度器的夏令时语义不创建任务、thread 或 run也不预留执行资源。此能力目前通过 API 提供workspace 表单尚未展示这些时间。
### 升级说明
- 升级 `GATEWAY_WORKERS > 1``scheduler.enabled: true` 的部署前,要么只在一个 Gateway worker 上启用调度器,要么配置 `scheduler.multi_instance: true`,并同时使用共享 Postgres、`run_ownership.heartbeat_enabled: true``run_events.backend: db`。升级后的 Gateway 会在启动时拒绝这种不安全组合,而不是静默启动。

View File

@ -19,7 +19,8 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu
- The background scheduler is single-instance by default. `scheduler.multi_instance=true` opts into lease-aware recovery across Gateway instances and requires shared Postgres, `run_ownership.heartbeat_enabled=true`, and `run_events.backend=db`; otherwise startup rejects the configuration. Live scheduled runs are preserved when a peer starts; expired launch claims return to the durable queue, expired run leases are atomically taken over, stale launch writes are fenced by lease ownership, and the Postgres advisory-locked budget makes `max_concurrent_runs` a shared global cap for `launching`/`running` rows.
- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
- MCP task notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 are part of that same contract — see [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
- Scheduled-task dispatch enforces at most one non-terminal occurrence per task through `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). `queued` is durable and survives restart; `launching` carries a short owner/expiry lease and is the only state that may call the normal Gateway launch path; `running` is associated with the durable run. Each occurrence also supplies a stable run-admission idempotency key, so a recovered launch retry reuses the same durable run. A reused-thread `ConflictError` moves `launching` back to `queued`, while non-conflict launch errors become terminal `failed`. Waiting rows do not consume `max_concurrent_runs`; the atomic queue claim enforces the budget. Repeated triggers coalesce on the one active row, and same-thread FIFO treats older `queued`, `launching`, and `running` rows as blockers. The task definition stays immutable for all three active states because queue admission, PATCH/resume, pause, and delete serialize on the parent task row before touching the occurrence row. Pause/delete atomically interrupt existing `queued` rows and reject `launching`/`running` rows; PATCH/resume reject every active state, and mutation errors advertise pause cancellation only for `queued` work. A manual trigger may queue and run while the parent schedule remains paused. Recovery and multi-instance reconciliation lock task/run pairs in deterministic task-id/run-id order and must reconstruct `run_id`, `started_at`, and the live error state before releasing the short launch claim. Launch/failure/timeout bookkeeping changes the occurrence and its parent task in one parent-first transaction so a peer cannot claim the released task between those writes. Queue timeout marks the occurrence failed and advances a scheduled occurrence so it cannot immediately requeue forever; repository write boundaries coerce serialized task timestamps before binding SQL `DateTime` fields.
- Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs`, excluding waiting rows. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused. Recovery locks task/run pairs in task-id/run-id order and restores `run_id`, `started_at` and live errors before releasing launch claims. Launch/failure/timeout updates use one parent-first transaction to prevent interleaved claims. Queue timeout fails the occurrence and advances scheduled work to prevent immediate requeue. Repository boundaries coerce serialized timestamps before SQL `DateTime` binding.
- `POST /api/scheduled-tasks/preview-cron` requires authenticated `threads:read`. Bounded cron previews call the shared scheduler calculator in `asyncio.to_thread`, preserving its DST semantics. Capture the optional aware reference once; return UTC and offset-bearing local occurrences without acquiring task/thread/run stores or dispatching work. This advisory API does not reserve execution.
- `extensions_config.json` is written at runtime by the Gateway (`PUT`/`PATCH /api/mcp/config`, the MCP enable switch, skill updates), so the production compose mounts it read-write while `config.yaml` stays `:ro`; Helm copies its ConfigMap seed into a writable home-volume directory before Gateway starts. Every read-modify-write holds both `extensions_config_write_lock` and the sidecar advisory `extensions_config_file_lock`, because the process-local lock alone loses updates across workers. Docker mounts the compose file as its own mount point, and Linux refuses `rename()` over a mount point with `EBUSY` even when the mount is writable — so `atomic_write_extensions_config` keeps the temp-file-plus-rename path and falls back to an in-place overwrite only on `EBUSY`. That fallback is deliberately non-atomic (a crash mid-write truncates the file); it exists because the alternative is a write that can never succeed, and only its first occurrence per target is logged at warning level. Any other `errno` still propagates. Pinned by `tests/test_compose_extensions_config_writable.py`, `tests/test_extensions_config_atomic_write.py`, and `tests/test_helm_extensions_config_writable.py`.
**Project Structure**:

View File

@ -4,9 +4,10 @@ import asyncio
import uuid
from datetime import UTC, datetime
from typing import Any
from zoneinfo import ZoneInfo
from fastapi import APIRouter, HTTPException, Query, Request
from pydantic import BaseModel, Field
from pydantic import AwareDatetime, BaseModel, Field
from app.gateway.authz import require_permission
from app.gateway.deps import (
@ -122,6 +123,55 @@ class ScheduledTaskUpdateRequest(BaseModel):
timezone: str | None = None
class CronPreviewRequest(BaseModel):
cron: str = Field(min_length=1, max_length=256)
timezone: str = Field(min_length=1, max_length=128)
count: int = Field(default=5, ge=1, le=10, strict=True)
start_at: AwareDatetime | None = None
class CronPreviewOccurrence(BaseModel):
run_at: datetime
local_time: datetime
class CronPreviewResponse(BaseModel):
cron: str
timezone: str
start_at: datetime
occurrences: list[CronPreviewOccurrence]
def _preview_cron(body: CronPreviewRequest, reference: datetime) -> CronPreviewResponse:
"""Calculate advisory occurrences with the same semantics as scheduling."""
try:
cron = normalize_cron_expression(body.cron)
zone = ZoneInfo(validate_timezone(body.timezone))
reference = reference.astimezone(UTC)
cursor = reference
occurrences = []
for _ in range(body.count):
upcoming = compute_next_run_at("cron", {"cron": cron}, body.timezone, now=cursor)
if upcoming is None or upcoming <= cursor:
raise ValueError("Cron expression did not produce a future occurrence")
occurrences.append(CronPreviewOccurrence(run_at=upcoming, local_time=upcoming.astimezone(zone)))
cursor = upcoming
except (ValueError, OverflowError) as exc:
raise HTTPException(status_code=422, detail=f"Cannot preview cron schedule: {exc}") from exc
return CronPreviewResponse(cron=cron, timezone=body.timezone, start_at=reference, occurrences=occurrences)
@router.post("/scheduled-tasks/preview-cron", response_model=CronPreviewResponse)
@require_permission("threads", "read")
async def preview_cron_schedule(request: Request, body: CronPreviewRequest):
"""Preview future cron instants without creating or dispatching a task."""
user = await get_optional_user_from_request(request)
if user is None:
raise HTTPException(status_code=401, detail="Authentication required")
reference = body.start_at if body.start_at is not None else datetime.now(UTC)
return await asyncio.to_thread(_preview_cron, body, reference)
@router.get("/scheduled-tasks")
@require_permission("threads", "read")
async def list_scheduled_tasks(request: Request):

View File

@ -0,0 +1,178 @@
from datetime import UTC, datetime
from types import SimpleNamespace
from unittest.mock import Mock
from zoneinfo import ZoneInfo
import httpx
import pytest
import pytest_asyncio
from fastapi import FastAPI
from app.gateway.authz import AuthContext
from app.gateway.routers import scheduled_tasks
from deerflow.scheduler.schedules import next_run_at
URL = "/api/scheduled-tasks/preview-cron"
PAYLOAD = {"cron": "0 9 * * 1-5", "timezone": "Asia/Shanghai", "count": 3, "start_at": "2026-09-12T00:00:00Z"}
@pytest_asyncio.fixture
async def client(monkeypatch):
app = FastAPI()
app.include_router(scheduled_tasks.router)
@app.middleware("http")
async def authenticate(request, call_next):
user = None if request.headers.get("x-test-auth") == "anonymous" else SimpleNamespace(id="preview-user")
permissions = [] if request.headers.get("x-test-auth") == "denied" else ["threads:read"]
request.state.auth = AuthContext(user=user, permissions=permissions)
return await call_next(request)
async def user_from_request(request):
return request.state.auth.user
monkeypatch.setattr(scheduled_tasks, "get_optional_user_from_request", user_from_request)
# A preview must remain usable without a task database, thread store or worker.
for name in ("get_scheduled_task_repo", "get_scheduled_task_run_repo", "get_thread_store", "get_scheduled_task_service", "get_config"):
monkeypatch.setattr(scheduled_tasks, name, Mock(side_effect=AssertionError(f"Preview accessed {name}")))
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as session:
yield session
@pytest.mark.asyncio
async def test_preview_returns_exact_utc_and_local_times_without_persistence(client):
response = await client.post(URL, json={**PAYLOAD, "cron": " 0\t9 * * 1-5 "})
assert response.status_code == 200, response.text
assert response.json() == {
"cron": "0 9 * * 1-5",
"timezone": "Asia/Shanghai",
"start_at": "2026-09-12T00:00:00Z",
"occurrences": [
{"run_at": "2026-09-14T01:00:00Z", "local_time": "2026-09-14T09:00:00+08:00"},
{"run_at": "2026-09-15T01:00:00Z", "local_time": "2026-09-15T09:00:00+08:00"},
{"run_at": "2026-09-16T01:00:00Z", "local_time": "2026-09-16T09:00:00+08:00"},
],
}
@pytest.mark.asyncio
@pytest.mark.parametrize(("auth", "status"), [("anonymous", 401), ("denied", 403)])
async def test_preview_enforces_authentication_and_read_permission(client, auth, status):
response = await client.post(URL, json=PAYLOAD, headers={"x-test-auth": auth})
assert response.status_code == status
@pytest.mark.asyncio
@pytest.mark.parametrize(
"override",
[
{"count": 0},
{"count": 11},
{"count": True},
{"count": "3"},
{"cron": ""},
{"cron": " " * 257},
{"cron": "0 0 * * * *"},
{"cron": "61 * * * *"},
{"cron": "0 0 31 2 *"},
{"timezone": "Not/A_Zone"},
{"timezone": "../UTC"},
{"timezone": "/UTC"},
{"timezone": ""},
{"timezone": "x" * 129},
{"start_at": "2026-09-12T00:00:00"},
{"start_at": "invalid"},
{"start_at": "9999-12-31T23:59:59Z"},
],
)
async def test_preview_rejects_invalid_or_unfulfillable_inputs(client, override):
response = await client.post(URL, json={**PAYLOAD, **override})
assert response.status_code == 422, response.text
@pytest.mark.asyncio
async def test_preview_default_reference_is_captured_once_and_count_defaults_to_five(client, monkeypatch):
class FixedDatetime(datetime):
calls = 0
@classmethod
def now(cls, tz=None):
cls.calls += 1
return datetime(2026, 9, 12, 0, 0, tzinfo=UTC)
monkeypatch.setattr(scheduled_tasks, "datetime", FixedDatetime)
response = await client.post(URL, json={"cron": "0 * * * *", "timezone": "UTC"})
assert response.status_code == 200, response.text
data = response.json()
assert FixedDatetime.calls == 1
assert data["start_at"] == "2026-09-12T00:00:00Z"
assert [row["run_at"] for row in data["occurrences"]] == [f"2026-09-12T0{hour}:00:00Z" for hour in range(1, 6)]
@pytest.mark.asyncio
async def test_preview_normalizes_offset_reference_and_excludes_reference_instant(client):
response = await client.post(URL, json={**PAYLOAD, "cron": "0 * * * *", "count": 1, "start_at": "2026-09-12T08:00:00+08:00"})
assert response.status_code == 200, response.text
assert response.json()["start_at"] == "2026-09-12T00:00:00Z"
assert response.json()["occurrences"][0]["run_at"] == "2026-09-12T01:00:00Z"
@pytest.mark.asyncio
@pytest.mark.parametrize("start_at", ["2026-03-07T12:00:00Z", "2026-10-31T12:00:00Z"])
async def test_preview_preserves_scheduler_dst_semantics(client, start_at):
cron = "30 2 * * *"
zone = "America/New_York"
cursor = datetime.fromisoformat(start_at)
expected = []
for _ in range(10):
cursor = next_run_at("cron", {"cron": cron}, zone, now=cursor)
expected.append(cursor)
response = await client.post(URL, json={"cron": cron, "timezone": zone, "count": 10, "start_at": start_at})
assert response.status_code == 200, response.text
rows = response.json()["occurrences"]
assert [datetime.fromisoformat(row["run_at"]) for row in rows] == expected
assert [row["local_time"] for row in rows] == [instant.astimezone(ZoneInfo(zone)).isoformat() for instant in expected]
@pytest.mark.asyncio
async def test_preview_calculates_off_the_request_thread(client, monkeypatch):
import threading
request_thread = threading.get_ident()
observed = []
original = scheduled_tasks.compute_next_run_at
def calculate(*args, **kwargs):
observed.append(threading.get_ident())
return original(*args, **kwargs)
monkeypatch.setattr(scheduled_tasks, "compute_next_run_at", calculate)
response = await client.post(URL, json=PAYLOAD)
assert response.status_code == 200, response.text
assert len(observed) == 3
assert all(ident != request_thread for ident in observed)
@pytest.mark.asyncio
async def test_preview_first_occurrence_matches_task_creation_at_same_clock(client, monkeypatch):
from unittest.mock import AsyncMock
from _router_auth_helpers import call_unwrapped
class FixedDatetime(datetime):
@classmethod
def now(cls, tz=None):
return datetime(2026, 9, 12, 0, 0, tzinfo=UTC)
monkeypatch.setattr(scheduled_tasks, "datetime", FixedDatetime)
response = await client.post(URL, json={"cron": PAYLOAD["cron"], "timezone": PAYLOAD["timezone"], "count": 1})
assert response.status_code == 200, response.text
# Only the separate creation below is allowed to acquire persistence dependencies.
repo = SimpleNamespace(create=AsyncMock())
monkeypatch.setattr(scheduled_tasks, "get_scheduled_task_repo", lambda request: repo)
monkeypatch.setattr(scheduled_tasks, "get_thread_store", lambda request: None)
monkeypatch.setattr(scheduled_tasks, "get_config", lambda: SimpleNamespace())
request = SimpleNamespace(state=SimpleNamespace(auth=AuthContext(user=SimpleNamespace(id="preview-user"))))
body = scheduled_tasks.ScheduledTaskCreateRequest(title="Example", prompt="Example", schedule_type="cron", schedule_spec={"cron": PAYLOAD["cron"]}, timezone=PAYLOAD["timezone"])
await call_unwrapped(scheduled_tasks.create_scheduled_task, request=request, body=body)
assert repo.create.await_args.kwargs["next_run_at"] == datetime.fromisoformat(response.json()["occurrences"][0]["run_at"])