mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-12 23:19:36 +00:00
feat(scheduled-tasks): preview upcoming cron occurrences (#5381)
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
This commit is contained in:
parent
a06688ad82
commit
444bfb72ce
12
README.md
12
README.md
@ -1592,6 +1592,18 @@ Scheduled runs use `scheduler.recursion_limit` in `config.yaml` (default `1000`,
|
||||
|
||||
The background scheduler is single-instance by default. For a multi-pod deployment, set `scheduler.multi_instance: true` and use shared Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; startup and periodic recovery then preserve live peer runs, atomically return expired launch claims to the queue, take over only expired run leases, and fence stale launch writes. `max_concurrent_runs` is a shared global cap across Pods for `launching`/`running` occurrences; waiting `queued` rows do not consume it. Without those settings, enable the scheduler on exactly one Gateway pod. These scheduler fields are startup-only; restart all Gateway Pods together when changing them.
|
||||
|
||||
### Preview cron occurrences through the API
|
||||
|
||||
Authenticated clients with `threads:read` can call `POST /api/scheduled-tasks/preview-cron` before creating a task:
|
||||
|
||||
```json
|
||||
{"cron":"0 9 * * 1-5","timezone":"Asia/Shanghai","count":3,"start_at":"2026-09-12T00:00:00Z"}
|
||||
```
|
||||
|
||||
The response contains normalized `cron`, `timezone`, the effective UTC `start_at`, and `occurrences` with UTC `run_at` and offset-bearing `local_time`. In this example the first occurrence is `2026-09-14T01:00:00Z` / `2026-09-14T09:00:00+08:00`.
|
||||
|
||||
`count` is an integer from 1 to 10 (default 5). `start_at` must include a timezone; omit it to capture server time once. Cron expressions use the scheduler's five-field syntax (maximum 256 characters); timezone names are at most 128 characters. Invalid inputs or schedules without the requested future occurrences return 422. Preview shares the scheduler's DST behavior, creates no task, thread or run, and does not reserve execution. This is an API capability; the workspace form does not yet display these occurrences.
|
||||
|
||||
### Upgrade Notes
|
||||
|
||||
- Occurrence ordering applies to rows admitted by upgraded Gateway instances, which project only sequenced occurrences onto the parent task and defer recovery while any occurrence is still live, whichever instance admitted it; a task whose history is entirely unsequenced keeps the previous timestamp ordering until its first sequenced admission. During a rolling upgrade, rows admitted by pre-upgrade instances are projected by those instances themselves, as before the upgrade, and the ordering guarantees hold once every Gateway writer runs the upgraded version. Existing history is not backfilled; the upgrade does not reconstruct past order or repair historical counts.
|
||||
|
||||
12
README_zh.md
12
README_zh.md
@ -838,6 +838,18 @@ DeerFlow 现在在 workspace 里内置了一个一等的定时任务(scheduled
|
||||
|
||||
后台调度器默认是单实例。多 Pod 部署时,请设置 `scheduler.multi_instance: true`,并使用共享 Postgres、`run_ownership.heartbeat_enabled: true` 和 `run_events.backend: db`;启动和周期性恢复会保留仍由对端持有的运行,把过期的 launch claim 原子退回队列,只接管过期的 run lease,并隔离过期的 launch 写入。`max_concurrent_runs` 是跨 Pod 共享的全局上限,只计入 `launching` / `running` 的执行;等待中的 `queued` 行不占用该配额。没有这些配置时,请只在一个 Gateway Pod 上启用调度器。这些 scheduler 字段只在启动时生效;修改后需要一起重启所有 Gateway Pod。
|
||||
|
||||
### 通过 API 预览 cron 执行时间
|
||||
|
||||
已认证且具有 `threads:read` 权限的客户端,可在创建任务前调用 `POST /api/scheduled-tasks/preview-cron`:
|
||||
|
||||
```json
|
||||
{"cron":"0 9 * * 1-5","timezone":"Asia/Shanghai","count":3,"start_at":"2026-09-12T00:00:00Z"}
|
||||
```
|
||||
|
||||
响应包含规范化的 `cron`、`timezone`、生效的 UTC `start_at`,以及 `occurrences` 列表中的 UTC `run_at` 和带偏移量的 `local_time`。此例的首次执行时间为 `2026-09-14T01:00:00Z` / `2026-09-14T09:00:00+08:00`。
|
||||
|
||||
`count` 为 1–10 的整数,默认 5。`start_at` 必须带时区,省略时只读取一次服务器当前时间。cron 沿用调度器的五字段语法,最长 256 字符;时区名称最长 128 字符。输入无效或无法计算所需未来时间时返回 422。预览沿用实际调度器的夏令时语义,不创建任务、thread 或 run,也不预留执行资源。此能力目前通过 API 提供,workspace 表单尚未展示这些时间。
|
||||
|
||||
### 升级说明
|
||||
|
||||
- 升级 `GATEWAY_WORKERS > 1` 且 `scheduler.enabled: true` 的部署前,要么只在一个 Gateway worker 上启用调度器,要么配置 `scheduler.multi_instance: true`,并同时使用共享 Postgres、`run_ownership.heartbeat_enabled: true` 和 `run_events.backend: db`。升级后的 Gateway 会在启动时拒绝这种不安全组合,而不是静默启动。
|
||||
|
||||
@ -19,7 +19,8 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu
|
||||
- The background scheduler is single-instance by default. `scheduler.multi_instance=true` opts into lease-aware recovery across Gateway instances and requires shared Postgres, `run_ownership.heartbeat_enabled=true`, and `run_events.backend=db`; otherwise startup rejects the configuration. Live scheduled runs are preserved when a peer starts; expired launch claims return to the durable queue, expired run leases are atomically taken over, stale launch writes are fenced by lease ownership, and the Postgres advisory-locked budget makes `max_concurrent_runs` a shared global cap for `launching`/`running` rows.
|
||||
- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
|
||||
- MCP task notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 are part of that same contract — see [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
|
||||
- Scheduled-task dispatch enforces at most one non-terminal occurrence per task through `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). `queued` is durable and survives restart; `launching` carries a short owner/expiry lease and is the only state that may call the normal Gateway launch path; `running` is associated with the durable run. Each occurrence also supplies a stable run-admission idempotency key, so a recovered launch retry reuses the same durable run. A reused-thread `ConflictError` moves `launching` back to `queued`, while non-conflict launch errors become terminal `failed`. Waiting rows do not consume `max_concurrent_runs`; the atomic queue claim enforces the budget. Repeated triggers coalesce on the one active row, and same-thread FIFO treats older `queued`, `launching`, and `running` rows as blockers. The task definition stays immutable for all three active states because queue admission, PATCH/resume, pause, and delete serialize on the parent task row before touching the occurrence row. Pause/delete atomically interrupt existing `queued` rows and reject `launching`/`running` rows; PATCH/resume reject every active state, and mutation errors advertise pause cancellation only for `queued` work. A manual trigger may queue and run while the parent schedule remains paused. Recovery and multi-instance reconciliation lock task/run pairs in deterministic task-id/run-id order and must reconstruct `run_id`, `started_at`, and the live error state before releasing the short launch claim. Launch/failure/timeout bookkeeping changes the occurrence and its parent task in one parent-first transaction so a peer cannot claim the released task between those writes. Queue timeout marks the occurrence failed and advances a scheduled occurrence so it cannot immediately requeue forever; repository write boundaries coerce serialized task timestamps before binding SQL `DateTime` fields.
|
||||
- Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs`, excluding waiting rows. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused. Recovery locks task/run pairs in task-id/run-id order and restores `run_id`, `started_at` and live errors before releasing launch claims. Launch/failure/timeout updates use one parent-first transaction to prevent interleaved claims. Queue timeout fails the occurrence and advances scheduled work to prevent immediate requeue. Repository boundaries coerce serialized timestamps before SQL `DateTime` binding.
|
||||
- `POST /api/scheduled-tasks/preview-cron` requires authenticated `threads:read`. Bounded cron previews call the shared scheduler calculator in `asyncio.to_thread`, preserving its DST semantics. Capture the optional aware reference once; return UTC and offset-bearing local occurrences without acquiring task/thread/run stores or dispatching work. This advisory API does not reserve execution.
|
||||
- `extensions_config.json` is written at runtime by the Gateway (`PUT`/`PATCH /api/mcp/config`, the MCP enable switch, skill updates), so the production compose mounts it read-write while `config.yaml` stays `:ro`; Helm copies its ConfigMap seed into a writable home-volume directory before Gateway starts. Every read-modify-write holds both `extensions_config_write_lock` and the sidecar advisory `extensions_config_file_lock`, because the process-local lock alone loses updates across workers. Docker mounts the compose file as its own mount point, and Linux refuses `rename()` over a mount point with `EBUSY` even when the mount is writable — so `atomic_write_extensions_config` keeps the temp-file-plus-rename path and falls back to an in-place overwrite only on `EBUSY`. That fallback is deliberately non-atomic (a crash mid-write truncates the file); it exists because the alternative is a write that can never succeed, and only its first occurrence per target is logged at warning level. Any other `errno` still propagates. Pinned by `tests/test_compose_extensions_config_writable.py`, `tests/test_extensions_config_atomic_write.py`, and `tests/test_helm_extensions_config_writable.py`.
|
||||
|
||||
**Project Structure**:
|
||||
|
||||
@ -4,9 +4,10 @@ import asyncio
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import AwareDatetime, BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import (
|
||||
@ -122,6 +123,55 @@ class ScheduledTaskUpdateRequest(BaseModel):
|
||||
timezone: str | None = None
|
||||
|
||||
|
||||
class CronPreviewRequest(BaseModel):
|
||||
cron: str = Field(min_length=1, max_length=256)
|
||||
timezone: str = Field(min_length=1, max_length=128)
|
||||
count: int = Field(default=5, ge=1, le=10, strict=True)
|
||||
start_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class CronPreviewOccurrence(BaseModel):
|
||||
run_at: datetime
|
||||
local_time: datetime
|
||||
|
||||
|
||||
class CronPreviewResponse(BaseModel):
|
||||
cron: str
|
||||
timezone: str
|
||||
start_at: datetime
|
||||
occurrences: list[CronPreviewOccurrence]
|
||||
|
||||
|
||||
def _preview_cron(body: CronPreviewRequest, reference: datetime) -> CronPreviewResponse:
|
||||
"""Calculate advisory occurrences with the same semantics as scheduling."""
|
||||
try:
|
||||
cron = normalize_cron_expression(body.cron)
|
||||
zone = ZoneInfo(validate_timezone(body.timezone))
|
||||
reference = reference.astimezone(UTC)
|
||||
cursor = reference
|
||||
occurrences = []
|
||||
for _ in range(body.count):
|
||||
upcoming = compute_next_run_at("cron", {"cron": cron}, body.timezone, now=cursor)
|
||||
if upcoming is None or upcoming <= cursor:
|
||||
raise ValueError("Cron expression did not produce a future occurrence")
|
||||
occurrences.append(CronPreviewOccurrence(run_at=upcoming, local_time=upcoming.astimezone(zone)))
|
||||
cursor = upcoming
|
||||
except (ValueError, OverflowError) as exc:
|
||||
raise HTTPException(status_code=422, detail=f"Cannot preview cron schedule: {exc}") from exc
|
||||
return CronPreviewResponse(cron=cron, timezone=body.timezone, start_at=reference, occurrences=occurrences)
|
||||
|
||||
|
||||
@router.post("/scheduled-tasks/preview-cron", response_model=CronPreviewResponse)
|
||||
@require_permission("threads", "read")
|
||||
async def preview_cron_schedule(request: Request, body: CronPreviewRequest):
|
||||
"""Preview future cron instants without creating or dispatching a task."""
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
reference = body.start_at if body.start_at is not None else datetime.now(UTC)
|
||||
return await asyncio.to_thread(_preview_cron, body, reference)
|
||||
|
||||
|
||||
@router.get("/scheduled-tasks")
|
||||
@require_permission("threads", "read")
|
||||
async def list_scheduled_tasks(request: Request):
|
||||
|
||||
178
backend/tests/test_scheduled_task_cron_preview.py
Normal file
178
backend/tests/test_scheduled_task_cron_preview.py
Normal file
@ -0,0 +1,178 @@
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.gateway.authz import AuthContext
|
||||
from app.gateway.routers import scheduled_tasks
|
||||
from deerflow.scheduler.schedules import next_run_at
|
||||
|
||||
URL = "/api/scheduled-tasks/preview-cron"
|
||||
PAYLOAD = {"cron": "0 9 * * 1-5", "timezone": "Asia/Shanghai", "count": 3, "start_at": "2026-09-12T00:00:00Z"}
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def client(monkeypatch):
|
||||
app = FastAPI()
|
||||
app.include_router(scheduled_tasks.router)
|
||||
|
||||
@app.middleware("http")
|
||||
async def authenticate(request, call_next):
|
||||
user = None if request.headers.get("x-test-auth") == "anonymous" else SimpleNamespace(id="preview-user")
|
||||
permissions = [] if request.headers.get("x-test-auth") == "denied" else ["threads:read"]
|
||||
request.state.auth = AuthContext(user=user, permissions=permissions)
|
||||
return await call_next(request)
|
||||
|
||||
async def user_from_request(request):
|
||||
return request.state.auth.user
|
||||
|
||||
monkeypatch.setattr(scheduled_tasks, "get_optional_user_from_request", user_from_request)
|
||||
# A preview must remain usable without a task database, thread store or worker.
|
||||
for name in ("get_scheduled_task_repo", "get_scheduled_task_run_repo", "get_thread_store", "get_scheduled_task_service", "get_config"):
|
||||
monkeypatch.setattr(scheduled_tasks, name, Mock(side_effect=AssertionError(f"Preview accessed {name}")))
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test") as session:
|
||||
yield session
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_returns_exact_utc_and_local_times_without_persistence(client):
|
||||
response = await client.post(URL, json={**PAYLOAD, "cron": " 0\t9 * * 1-5 "})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json() == {
|
||||
"cron": "0 9 * * 1-5",
|
||||
"timezone": "Asia/Shanghai",
|
||||
"start_at": "2026-09-12T00:00:00Z",
|
||||
"occurrences": [
|
||||
{"run_at": "2026-09-14T01:00:00Z", "local_time": "2026-09-14T09:00:00+08:00"},
|
||||
{"run_at": "2026-09-15T01:00:00Z", "local_time": "2026-09-15T09:00:00+08:00"},
|
||||
{"run_at": "2026-09-16T01:00:00Z", "local_time": "2026-09-16T09:00:00+08:00"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(("auth", "status"), [("anonymous", 401), ("denied", 403)])
|
||||
async def test_preview_enforces_authentication_and_read_permission(client, auth, status):
|
||||
response = await client.post(URL, json=PAYLOAD, headers={"x-test-auth": auth})
|
||||
assert response.status_code == status
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"override",
|
||||
[
|
||||
{"count": 0},
|
||||
{"count": 11},
|
||||
{"count": True},
|
||||
{"count": "3"},
|
||||
{"cron": ""},
|
||||
{"cron": " " * 257},
|
||||
{"cron": "0 0 * * * *"},
|
||||
{"cron": "61 * * * *"},
|
||||
{"cron": "0 0 31 2 *"},
|
||||
{"timezone": "Not/A_Zone"},
|
||||
{"timezone": "../UTC"},
|
||||
{"timezone": "/UTC"},
|
||||
{"timezone": ""},
|
||||
{"timezone": "x" * 129},
|
||||
{"start_at": "2026-09-12T00:00:00"},
|
||||
{"start_at": "invalid"},
|
||||
{"start_at": "9999-12-31T23:59:59Z"},
|
||||
],
|
||||
)
|
||||
async def test_preview_rejects_invalid_or_unfulfillable_inputs(client, override):
|
||||
response = await client.post(URL, json={**PAYLOAD, **override})
|
||||
assert response.status_code == 422, response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_default_reference_is_captured_once_and_count_defaults_to_five(client, monkeypatch):
|
||||
class FixedDatetime(datetime):
|
||||
calls = 0
|
||||
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
cls.calls += 1
|
||||
return datetime(2026, 9, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
monkeypatch.setattr(scheduled_tasks, "datetime", FixedDatetime)
|
||||
response = await client.post(URL, json={"cron": "0 * * * *", "timezone": "UTC"})
|
||||
assert response.status_code == 200, response.text
|
||||
data = response.json()
|
||||
assert FixedDatetime.calls == 1
|
||||
assert data["start_at"] == "2026-09-12T00:00:00Z"
|
||||
assert [row["run_at"] for row in data["occurrences"]] == [f"2026-09-12T0{hour}:00:00Z" for hour in range(1, 6)]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_normalizes_offset_reference_and_excludes_reference_instant(client):
|
||||
response = await client.post(URL, json={**PAYLOAD, "cron": "0 * * * *", "count": 1, "start_at": "2026-09-12T08:00:00+08:00"})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["start_at"] == "2026-09-12T00:00:00Z"
|
||||
assert response.json()["occurrences"][0]["run_at"] == "2026-09-12T01:00:00Z"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("start_at", ["2026-03-07T12:00:00Z", "2026-10-31T12:00:00Z"])
|
||||
async def test_preview_preserves_scheduler_dst_semantics(client, start_at):
|
||||
cron = "30 2 * * *"
|
||||
zone = "America/New_York"
|
||||
cursor = datetime.fromisoformat(start_at)
|
||||
expected = []
|
||||
for _ in range(10):
|
||||
cursor = next_run_at("cron", {"cron": cron}, zone, now=cursor)
|
||||
expected.append(cursor)
|
||||
response = await client.post(URL, json={"cron": cron, "timezone": zone, "count": 10, "start_at": start_at})
|
||||
assert response.status_code == 200, response.text
|
||||
rows = response.json()["occurrences"]
|
||||
assert [datetime.fromisoformat(row["run_at"]) for row in rows] == expected
|
||||
assert [row["local_time"] for row in rows] == [instant.astimezone(ZoneInfo(zone)).isoformat() for instant in expected]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_calculates_off_the_request_thread(client, monkeypatch):
|
||||
import threading
|
||||
|
||||
request_thread = threading.get_ident()
|
||||
observed = []
|
||||
original = scheduled_tasks.compute_next_run_at
|
||||
|
||||
def calculate(*args, **kwargs):
|
||||
observed.append(threading.get_ident())
|
||||
return original(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(scheduled_tasks, "compute_next_run_at", calculate)
|
||||
response = await client.post(URL, json=PAYLOAD)
|
||||
assert response.status_code == 200, response.text
|
||||
assert len(observed) == 3
|
||||
assert all(ident != request_thread for ident in observed)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_first_occurrence_matches_task_creation_at_same_clock(client, monkeypatch):
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
from _router_auth_helpers import call_unwrapped
|
||||
|
||||
class FixedDatetime(datetime):
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
return datetime(2026, 9, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
monkeypatch.setattr(scheduled_tasks, "datetime", FixedDatetime)
|
||||
response = await client.post(URL, json={"cron": PAYLOAD["cron"], "timezone": PAYLOAD["timezone"], "count": 1})
|
||||
assert response.status_code == 200, response.text
|
||||
# Only the separate creation below is allowed to acquire persistence dependencies.
|
||||
repo = SimpleNamespace(create=AsyncMock())
|
||||
monkeypatch.setattr(scheduled_tasks, "get_scheduled_task_repo", lambda request: repo)
|
||||
monkeypatch.setattr(scheduled_tasks, "get_thread_store", lambda request: None)
|
||||
monkeypatch.setattr(scheduled_tasks, "get_config", lambda: SimpleNamespace())
|
||||
request = SimpleNamespace(state=SimpleNamespace(auth=AuthContext(user=SimpleNamespace(id="preview-user"))))
|
||||
body = scheduled_tasks.ScheduledTaskCreateRequest(title="Example", prompt="Example", schedule_type="cron", schedule_spec={"cron": PAYLOAD["cron"]}, timezone=PAYLOAD["timezone"])
|
||||
await call_unwrapped(scheduled_tasks.create_scheduled_task, request=request, body=body)
|
||||
assert repo.create.await_args.kwargs["next_run_at"] == datetime.fromisoformat(response.json()["occurrences"][0]["run_at"])
|
||||
Loading…
x
Reference in New Issue
Block a user