diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 7b947ed96..ab346e9a8 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -17,7 +17,8 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu - Gateway streams `write_file` and `str_replace` argument deltas in bounded batches when clients also subscribe to `values`; messages-only consumers retain the original per-chunk contract, while `values` preserves the complete tool call. - With `stream_subgraphs`, subgraph frames keep their namespace in the SSE event name (`values|`, LangGraph Platform style) instead of impersonating root frames — a delegated subagent inherits the parent checkpoint namespace, so publishing its `values` snapshot as bare `values` replaces the whole thread view in SDK clients (#4399). Root-only consumers (file-tool chunk batcher, subagent event persistence, LLM error-fallback detection) ignore namespaced frames. The web frontend does not request subgraph streaming; subtask progress rides root-namespace `task_*` custom events. - Scheduled-task executions must reuse that same Gateway run lifecycle. The scheduler may decide *when* work runs, but it must dispatch through the existing run path rather than introducing a parallel execution stack. -- Scheduled-task dispatch enforces "at most one active run per task when `overlap_policy=skip`" at the DB layer via the partial unique index `uq_scheduled_task_run_active` (`scheduled_task_runs.task_id WHERE status IN ('queued','running')`). `ScheduledTaskService.dispatch_task`'s `has_active_runs` check is a non-atomic fast path (its own session, separated from the `create()` insert by `await` points), so two concurrent dispatches — a manual `POST /scheduled-tasks/{id}/trigger` racing the poller, a double-click, or a client retry — can both pass it; the index is the atomic arbiter, and the losing `create` surfaces as `ActiveScheduledRunConflict` (translated from `IntegrityError` in the repository) and collapses to the same outcome as the fast path (manual → 409 conflict, scheduled → a `"skipped"` tombstone). The scheduled-skip tombstone is created directly as terminal `"skipped"` (not a transient `"queued"`) so it never occupies the active slot the pre-existing run still holds. Sibling of the `runs` table's `uq_runs_thread_active` (PR #4003), which keys on `thread_id` and so does not cover the default `fresh_thread_per_run` context where every dispatch gets a new thread. Index is status-only, not `overlap_policy`-conditional (the policy is fixed to `"skip"` in the MVP). +- Scheduled-task dispatch enforces "at most one active run per task when `overlap_policy=skip`" at the DB layer via the partial unique index `uq_scheduled_task_run_active` (`scheduled_task_runs.task_id WHERE status IN ('queued','running')`). `ScheduleService.dispatch_task`'s `has_active` check is a non-atomic fast path (its own session, separated from the queued-record insert by `await` points), so two concurrent dispatches — a manual `POST /scheduled-tasks/{id}/trigger` racing the poller, a double-click, or a client retry — can both pass it; the index is the atomic arbiter, and the losing insert surfaces as `ActiveRunConflictError` (translated from `IntegrityError` in the `SqlScheduledRunRepository` adapter) and collapses to the same outcome as the fast path (manual → 409 conflict, scheduled → a `"skipped"` tombstone). The scheduled-skip tombstone is created directly as terminal `"skipped"` (not a transient `"queued"`) so it never occupies the active slot the pre-existing run still holds. Sibling of the `runs` table's `uq_runs_thread_active` (PR #4003), which keys on `thread_id` and so does not cover the default `fresh_thread_per_run` context where every dispatch gets a new thread. Index is status-only, not `overlap_policy`-conditional (the policy is fixed to `"skip"` in the MVP). +- The dispatch path and the run-completion hook write to the same `scheduled_tasks` row concurrently, so **the two writes own disjoint field sets and neither may write through the whole aggregate**. A run that fails fast reaches its completion hook before the dispatch path's own bookkeeping write lands, so the hook is holding a snapshot taken *before* it. The launch write (`record_launch`) owns the schedule — `next_run_at`, `last_run_at`, `last_run_id`, `last_thread_id`, `run_count`, the claim — and the completion write (`record_completion`) owns only the verdict: the terminal status and `last_error`. Expressing the completion as a read-modify-write through the aggregate replays the stale snapshot and rolls the launch write back, restoring an elapsed `next_run_at` and leaving the task in `running` with no live claim — a shape neither `claim_due` branch nor `cancel_stuck_once_tasks` can reach, i.e. a cron task that is permanently unschedulable with no recovery path. `record_launch`'s `protect_terminal` flag closes the mirror-image race in the other direction (the hook's terminal status must survive a launch write landing after it). Pinned by the `TestRecordCompletion` contract cases (both the in-memory double and real sqlite) and by `test_a_cron_task_survives_a_launch_write_landing_mid_completion`. **Project Structure**: ``` @@ -86,7 +87,6 @@ When making code changes, you MUST update the relevant documentation: ```bash make check # Check system requirements make install # Install all dependencies (frontend + backend) -make detect-thread-boundaries # Inventory backend executor/thread/event-loop boundaries make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight make start # Start production services locally make stop # Stop all services @@ -97,46 +97,13 @@ make stop # Stop all services make install # Install backend dependencies make dev # Run Gateway API with reload (port 8001) make gateway # Run Gateway API only (port 8001) -make test # Run offline backend tests (excludes live external-API tests) -make test-live # Explicitly run live DeerFlowClient tests with real APIs +make test # Run all backend tests make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/ make lint # Lint with ruff make format # Format code with ruff make migrate-rev MSG="..." # Autogenerate a new alembic revision (see Schema Migrations section) ``` -The root `detect-thread-boundaries` target statically inventories execution -boundaries under `backend/app/` and `backend/packages/harness/deerflow/`. It -prints a concise count by execution domain and writes the complete, versioned -JSON payload to `.deer-flow/thread-boundary-inventory.json`. Every finding has -a stable `boundary_kind`: `asyncio_default_executor`, `dedicated_executor`, -`anyio_worker_thread`, `direct_event_loop_blocking`, `separate_event_loop`, or -`unresolved_dynamic_boundary`. - -The AST inventory covers `asyncio.to_thread`, default and explicit -`run_in_executor` submissions, imported aliases, simple same-module helper -wrappers (after pre-registering dedicated executor targets), `set_default_executor`, -`ThreadPoolExecutor` construction/submission, -additional event loops, synchronous LangChain tools, and direct -`BaseChatModel` fallback inheritance. It remains read-only and does not alter -executor routing or sizing. - -To supplement the static scan with configured runtime types, run: - -```bash -python scripts/detect_thread_boundaries.py \ - --runtime-config config.yaml \ - --json-output .deer-flow/thread-boundary-inventory.json -``` - -Runtime inspection imports configured tool objects and model classes so it can -record concrete tool names/types/modules, sync functions, async coroutines, -and `_agenerate`/`_astream` ownership. It does not invoke tools, instantiate -models, or call external services; import failures remain in the JSON as -`unresolved_dynamic_boundary` records. The detector implementation and focused -coverage live in `tests/support/detectors/thread_boundaries.py` and -`tests/test_detect_thread_boundaries.py`. - The `detect-blocking-io` target parses `app/`, `packages/harness/deerflow/`, and `scripts/` with AST. By default it reports only blocking IO candidates that are inside async code, reachable from async code in the same file, or reachable @@ -485,8 +452,8 @@ Localhost persistence deliberately reads the direct request `Host` and ignores ` |--------|-----------| | **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details | | **Features** (`/api/features`) | `GET /` - report config-gated feature availability (`agents_api.enabled`, `browser_control.enabled`) for frontend UI gating | -| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). All priced models must use one currency; mixed currencies disable cost reporting and leave cost/currency fields null instead of producing invalid aggregates. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | -| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - replace the full config with whole-payload stdio validation; `PATCH /config` - toggle one server while preserving the raw extensions config and validating only an enabled target; both writes reload config and reset the process-local MCP cache | +| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | +| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) | | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes | | **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants | | **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data | @@ -683,7 +650,6 @@ that cannot tell sibling branches apart. ### Subagent System (`packages/harness/deerflow/subagents/`) **Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist) -**Benefit-based routing policy**: Enabling subagents exposes delegation as an optimization, not a default response to complexity. The lead prompt defaults to direct execution and permits `task` only when parallel latency, specialist capability, or context-isolation benefit clearly exceeds startup, duplicate-discovery, synthesis, state-conflict, and side-effect costs. Inter-agent output dependencies and overlapping mutable state are hard vetoes for parallel dispatch, while duplicate discovery and a cheap direct path remain costs rather than categorical vetoes; a bounded sequential chain may run in one subagent when specialist or context-isolation benefit clearly wins. Parallel scopes must be independent and non-overlapping, the lead uses the fewest useful subagents, and every later batch is re-evaluated while retaining any within-batch parallel benefit. When the enforced per-response limit is 1, the rendered prompt removes parallel and multi-batch benefit guidance and permits delegation only for material specialist or context-isolation benefit. Keep this policy aligned across `lead_agent/prompt.py`, the `task` tool description, and both built-in role descriptions; routing regressions are pinned in `tests/test_subagent_routing_prompt.py`, `tests/test_subagent_prompt_security.py`, and `tests/test_lead_agent_prompt.py`. **User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog. **Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers) **Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box) @@ -752,7 +718,7 @@ E2B output sync records remote file versions and actual host file metadata in a add a parallel routing middleware for PR1-style preference hints. - **Stdio file outputs**: Persistent stdio sessions are scoped by `user_id:thread_id`. For stdio transports only, DeerFlow pins the subprocess default `cwd` to the thread workspace and `TMPDIR`/`TMP`/`TEMP` to `workspace/.mcp/tmp/`, unless the operator explicitly configured `cwd` or temp env values. SSE/HTTP transports skip this filesystem prep entirely. - **Stdio path translation**: MCP-returned local file references are not copied. If a `ResourceLink` or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to `/mnt/user-data/...`; paths outside that tree remain unchanged. -- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (`PUT /api/mcp/config` keeps whole-payload validation, while `PATCH /api/mcp/config` changes only one server's `enabled` field, normalizes the same `type`/MCP-spec `transport` alias as the runtime config model, and validates the target only when enabling it; either endpoint's reset clears the cache only in its own worker). MCP, skill, and embedded-client writers share `atomic_write_extensions_config()`, which writes and fsyncs a same-directory temporary file before `os.replace()` and preserves an existing file's mode and symlink target; failed serialization or replacement leaves the prior config intact and cleans up the temporary file. +- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via the resolved-path + content-signature check above, so multi-worker / stale-mtime deployments still pick up an added/removed MCP server without a restart (the `PUT /api/mcp/config` reset only clears the cache in its own worker) ### Skills System (`packages/harness/deerflow/skills/`) @@ -777,7 +743,7 @@ Lets a caller pass per-request, short-lived end-user credentials (e.g. an ERP to - **Declare**: a skill lists the secrets it needs in `SKILL.md` frontmatter — `required-secrets:` as a string list or `{name, optional}` mappings. `name` is both the lookup key and the env var name exposed to scripts. Parsed by `skills/parser.py::parse_required_secrets` into `Skill.required_secrets` (`SecretRequirement`); malformed entries are dropped with a warning. - **Carry**: the caller sends values out-of-band in the run request's `context.secrets` mapping (never a message). `runtime/secret_context.py` owns the contract (`SECRETS_CONTEXT_KEY`, `extract_request_secrets`). The existing `context` passthrough carries it to `runtime.context` without mirroring into `configurable`. `build_run_config` still sets `configurable.thread_id` on the context path — the checkpointer requires it. MCP tool interceptors can read the same live carrier from `langgraph.config.get_config()["context"]["secrets"]`; see `docs/MCP_SERVER.md`. -- **Admission and redaction ownership**: `services.py::start_run()` validates both legacy request mappings, `metadata.auth_token` and `config.metadata.auth_token`, before any run or thread persistence. `runtime/secret_context.py::redact_config_secrets()` also removes nested config metadata secrets from observable and persisted config copies; historical `RunResponse.kwargs` applies the same redaction non-mutatively, leaving stored `RunRecord` data unchanged. Keep callers on `config.context.secrets` rather than adding another credential carrier. Scheduled task definitions have no durable credential carrier: `ScheduledTaskService` supplies only `scheduled_task_id`, `scheduled_task_run_id`, and `scheduled_trigger` as run metadata. +- **Admission and redaction ownership**: `services.py::start_run()` validates both legacy request mappings, `metadata.auth_token` and `config.metadata.auth_token`, before any run or thread persistence. `runtime/secret_context.py::redact_config_secrets()` also removes nested config metadata secrets from observable and persisted config copies; historical `RunResponse.kwargs` applies the same redaction non-mutatively, leaving stored `RunRecord` data unchanged. Keep callers on `config.context.secrets` rather than adding another credential carrier. Scheduled task definitions have no durable credential carrier: the schedule dispatch path supplies only `scheduled_task_id`, `scheduled_task_run_id`, and `scheduled_trigger` as run metadata. - **Bind (point A+)**: `SkillActivationMiddleware._resolve_secret_bindings` recomputes the injection set (`runtime.context[__active_skill_secrets]`) on every model call from two unioned sources, then REPLACES the key. (1) *Slash*: the run's most recent `/skill` activation, persisted as a source on the run context (only the activated skill's **canonical container path**, never its declared secrets) so the whole tool loop after the activation call keeps the binding; a new activation replaces it. Slash reads the genuine user text via `get_original_user_content_text`; `InputSanitizationMiddleware` preserves it (`ORIGINAL_USER_CONTENT_KEY`), so activation fires even after sanitization. (2) *In-context* (autonomous invocation): skills the model actually loaded in this thread — `ThreadState.skill_context` entries. **Both sources resolve the live registry skill by normalized container path on every call** (`_resolve_registry_skill`) and bind only that skill's own declared secrets — enabled + allowlist checked for both; the `secrets-autonomous: false` opt-out (malformed values fail closed to `false`) additionally gates the in-context path but exempts explicit slash. Resolving by registry — not by trusting the source's stored data — is what makes a caller-forged `__slash_skill_secret_source` harmless (`runtime.context` is caller-mergeable; the gateway also strips caller `__`-keys in `build_run_config`), #3938. Authorization is three-gated regardless of activation style: skill **enabled** by the operator × values **supplied per-request** by the caller (`context.secrets`) × names **declared** in frontmatter (∩ semantics). Because the set is recomputed per call, a skill evicted from `skill_context` (capacity) or a caller that stops supplying a value loses injection on the next call. The injected value always comes from the caller's request, never the host environment (scrubbed first — see below), so a declared name that also exists in the host env is safe: the caller's value wins and the host value is dropped (the #3861 per-user-key-overrides-shared-key case). Missing required secrets are logged once per binding change, not injected; binding changes are recorded as a `middleware:skill_secrets` journal event (skill and secret names only, never values). - **Inject**: `bash_tool` reads the injection set and passes it as `execute_command(env=...)`. Scope is the activation turn/run only — a run without `/skill` activation injects nothing. - **AIO image requirement**: on `AioSandbox` the env path uses the `bash.exec` API (`POST /v1/bash/exec`), which upstream all-in-one-sandbox only ships since `1.9.3` — older images (including a `latest` tag frozen on the `1.0.0.x` line) 404 the whole `/v1/bash/*` namespace. `AioSandbox` detects the 404, remembers the capability gap on the instance, and fails fast with an actionable upgrade error instead of letting the model retry raw 404s; there is deliberately **no** fallback through the legacy shell path because none keeps the secret values out of the command string (#3921). Regression tests: `tests/test_aio_sandbox.py::TestBashExecUnsupportedFailFast`. @@ -866,7 +832,7 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_ **GitHub event-driven agents** (webhook-driven IM channel): - Custom agents declare a `github:` block in their `config.yaml` to bind to repos and event triggers; the webhook route is fail-closed by default (mounted only when `GITHUB_WEBHOOK_SECRET` is set) and exempt from auth/CSRF because authenticity is enforced by HMAC. - Outbound is **log-only** by design: each agent posts its own reply mid-run via the `gh` CLI from its sandbox, so the manager uses `fire_and_forget=True` and `runs.create()` returns once pending. -- **Follow-up buffering while busy** (issue #4121): because outbound is log-only, the pre-existing `THREAD_BUSY_MESSAGE` reply on a `ConflictError` was invisible to the commenter — a comment posted while a run was already active looked like it had been silently ignored. When `ChannelRunPolicy.buffer_followups_on_busy=True` (GitHub's default), a `ConflictError` on `runs.create()` now also appends the triggering message to a per-thread, in-memory buffer (`ChannelManager._followup_buffers`) — deduped by GitHub delivery id, capped at `FOLLOWUP_BUFFER_MAX_PER_THREAD` (20, oldest dropped with a WARNING log on overflow). The first successful `runs.create()` on a thread now captures its `run_id` and spawns a background watcher that subscribes to that run's `StreamBridge` stream; once it observes `END_SENTINEL`, the watcher drains up to `FOLLOWUP_DRAIN_BATCH_SIZE` (10) buffered entries into one ``-wrapped input and fires a follow-up `runs.create()` — itself watched the same way, so a backlog deeper than one batch chains into further drain cycles instead of growing one unbounded input. If that follow-up `runs.create()` itself hits `ConflictError` (e.g. a manual Web UI turn or a scheduled run raced onto the same thread), the batch is requeued rather than lost, and is retried whenever this manager next successfully creates and watches a run on that thread. Reactions/acknowledgment (e.g. GitHub's `eyes`/`confused` reaction API) on buffered comments are intentionally **out of scope** for this mechanism and left to a follow-up — comments are coalesced silently. **Plumbing**: the watcher needs the Gateway's `StreamBridge` singleton, which `ChannelManager` did not previously have access to; it is threaded from `app.py`'s lifespan (where `app.state.stream_bridge` is already set by `langgraph_runtime`) through `start_channel_service(get_stream_bridge=...)` → `ChannelService.__init__` → `ChannelManager.__init__`, as a zero-arg closure mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` pattern used for `ScheduledTaskService` in the same lifespan function. A `ChannelManager` constructed without it (e.g. directly in a test) still buffers safely — it just has no watcher to auto-drain. **Scope limitation**: the buffer and watcher state are per-process, in-memory. Under `GATEWAY_WORKERS>1` or multi-pod, a follow-up comment routed to a different worker process than the one running the busy thread's agent will not see that buffer. This is a known, deliberately deferred limitation with the same shape as the cross-pod gap described for issue #4120 (a shared buffer store or IM-leader election would be needed to close it) — single-process/single-pod deployments, the safe default, see no correctness issue from this, only the documented per-process scope. +- **Follow-up buffering while busy** (issue #4121): because outbound is log-only, the pre-existing `THREAD_BUSY_MESSAGE` reply on a `ConflictError` was invisible to the commenter — a comment posted while a run was already active looked like it had been silently ignored. When `ChannelRunPolicy.buffer_followups_on_busy=True` (GitHub's default), a `ConflictError` on `runs.create()` now also appends the triggering message to a per-thread, in-memory buffer (`ChannelManager._followup_buffers`) — deduped by GitHub delivery id, capped at `FOLLOWUP_BUFFER_MAX_PER_THREAD` (20, oldest dropped with a WARNING log on overflow). The first successful `runs.create()` on a thread now captures its `run_id` and spawns a background watcher that subscribes to that run's `StreamBridge` stream; once it observes `END_SENTINEL`, the watcher drains up to `FOLLOWUP_DRAIN_BATCH_SIZE` (10) buffered entries into one ``-wrapped input and fires a follow-up `runs.create()` — itself watched the same way, so a backlog deeper than one batch chains into further drain cycles instead of growing one unbounded input. If that follow-up `runs.create()` itself hits `ConflictError` (e.g. a manual Web UI turn or a scheduled run raced onto the same thread), the batch is requeued rather than lost, and is retried whenever this manager next successfully creates and watches a run on that thread. Reactions/acknowledgment (e.g. GitHub's `eyes`/`confused` reaction API) on buffered comments are intentionally **out of scope** for this mechanism and left to a follow-up — comments are coalesced silently. **Plumbing**: the watcher needs the Gateway's `StreamBridge` singleton, which `ChannelManager` did not previously have access to; it is threaded from `app.py`'s lifespan (where `app.state.stream_bridge` is already set by `langgraph_runtime`) through `start_channel_service(get_stream_bridge=...)` → `ChannelService.__init__` → `ChannelManager.__init__`, as a zero-arg closure mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` pattern used for the schedule composition root in the same lifespan function. A `ChannelManager` constructed without it (e.g. directly in a test) still buffers safely — it just has no watcher to auto-drain. **Scope limitation**: the buffer and watcher state are per-process, in-memory. Under `GATEWAY_WORKERS>1` or multi-pod, a follow-up comment routed to a different worker process than the one running the busy thread's agent will not see that buffer. This is a known, deliberately deferred limitation with the same shape as the cross-pod gap described for issue #4120 (a shared buffer store or IM-leader election would be needed to close it) — single-process/single-pod deployments, the safe default, see no correctness issue from this, only the documented per-process scope. - See [backend/docs/GITHUB_AGENTS.md](docs/GITHUB_AGENTS.md) for the architecture diagrams: webhook → fan-out → `InboundMessage` dispatch, `preferred_thread_id = UUID5(repo, number, agent_name)` thread determinism, mention-handle precedence chain, GH token lifecycle via `GH_TOKEN`/`GITHUB_TOKEN` per-call `extra_env`, and the narrow `ConflictError` (HTTP 409) thread-create race recovery. @@ -1219,7 +1185,7 @@ Config is env-driven like the others — `MonocleTracingConfig`, built in `get_t - `skills` - Map of skill name → state (enabled) - `middlewares` - Zero-argument `AgentMiddleware` class paths for lead and subagent runtime extension. `config.yaml -> extensions` can override these fields after validation; overrides are replace-per-field, not list concatenation. -Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; their `extensions_config.json` writes use the shared atomic replacement helper, while `middlewares` remains an operator-controlled config-file extension point. +Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and skill state at runtime; `middlewares` remains an operator-controlled config-file extension point. ### Embedded Client (`packages/harness/deerflow/client.py`) @@ -1254,13 +1220,7 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk **Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent. -**Tests**: `tests/test_client.py` (offline unit tests including -`TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, -requires a root `config.yaml`, valid API credentials, and explicit opt-in via -`make test-live` or `DEER_FLOW_RUN_LIVE_TESTS=1`). The live suite calls real -external APIs and may incur API costs or create local sandboxes, artifacts, and -files. It is marked `live`, excluded from `make test`, and skipped in default -CI. +**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml) **Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`. @@ -1271,27 +1231,19 @@ CI. **Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.** - Write tests in `backend/tests/` following the existing naming convention `test_.py` -- Run the full offline suite before and after your change: `make test` +- Run the full suite before and after your change: `make test` - Tests must pass before a feature is considered complete - For lightweight config/utility modules, prefer pure unit tests with no external dependencies - If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`) ```bash -# Run all offline tests +# Run all tests make test -# Explicit live integration tests (requires config.yaml and credentials; -# calls real APIs and may create local side effects) -make test-live - # Run a specific test file PYTHONPATH=. uv run pytest tests/test_.py -v ``` -Direct pytest collection or execution of `tests/test_client_live.py` remains -skipped unless `DEER_FLOW_RUN_LIVE_TESTS=1` is set. Do not add that opt-in to -default CI workflows. - ### Running the Full Application From the **project root** directory: diff --git a/backend/app/adapters/schedule/__init__.py b/backend/app/adapters/schedule/__init__.py new file mode 100644 index 000000000..bc75798ac --- /dev/null +++ b/backend/app/adapters/schedule/__init__.py @@ -0,0 +1,20 @@ +"""Adapters of the schedule context. + +Mostly secondary (driven) adapters -- implementations of the output ports the +domain declares, which the service calls out to: + + scheduled_task_repository.py owned persistence + scheduled_run_repository.py owned persistence + run_launcher.py anti-corruption layer over the run runtime + thread_lookup.py anti-corruption layer over the thread store + +One exception, and the name says so: + + run_completion.py PRIMARY (inbound) -- the run runtime calls it + +It lives here so the context stays in one place rather than beside the other +two primary adapters, which sit next to whatever drives them (the router under +`gateway/routers/schedule/`, the poller under `scheduler/`). Direction is +stated by each module's own first line; a file added here without one is a +file whose direction nobody decided. +""" diff --git a/backend/app/adapters/schedule/run_completion.py b/backend/app/adapters/schedule/run_completion.py new file mode 100644 index 000000000..83007371e --- /dev/null +++ b/backend/app/adapters/schedule/run_completion.py @@ -0,0 +1,102 @@ +"""Primary adapter (inbound) -- the run runtime's completion callback. + +The one file in this package that drives the domain rather than serving it. +Its siblings are secondary adapters the service calls out to; this one is +called *by* the run runtime, the same way the router is called by HTTP and the +poller by its clock. Kept here rather than beside those two so the context +stays in one place, with the direction stated by the name and by this line. + +Its job is the filtering the legacy completion hook did inline. Every run in +the process reaches that callback, so most of them are none of this context's +business, and producing no call at all says exactly that -- not an error, just +nothing to write back. That is why `ScheduleService.handle_run_completion` +carries no guard clauses and never imports ``RunRecord``. + +TODO(hexagonal): this depends on ``RunRecord``, a run-runtime type, rather +than on a contract published by the run context -- that context has not been +through a hexagonal slice yet. When it publishes one (a DTO, not its aggregate +and not its repository), replace ``_to_outcome``. Nothing else moves. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +from deerflow.domain.schedule.model import RunStatus +from deerflow.domain.schedule.ports import RunOutcome + +if TYPE_CHECKING: + from deerflow.domain.schedule.service import ScheduleService + from deerflow.runtime import RunRecord + +# The runtime reports four terminal states; the domain has three, because +# `timeout` and `error` are the same fact to a scheduled task while +# `interrupted` is deliberately not -- a cancel or same-thread takeover ends +# the task CANCELLED, not FAILED. +_TERMINAL_STATUSES = { + "success": RunStatus.SUCCESS, + "error": RunStatus.FAILED, + "timeout": RunStatus.FAILED, + "interrupted": RunStatus.INTERRUPTED, +} + +_INTERRUPTED_WITHOUT_ERROR = "run was interrupted before completion" + + +class ScheduleRunCompletionListener: + """Turns a finished run into the write-back use case, or into nothing. + + Deciding whether a run is ours and invoking the use case are one + responsibility, not two: "ignore this run" is only meaningful as "do not + call the service", so splitting them left the second half living in the + composition root, where behaviour is not asserted. + """ + + def __init__(self, service: ScheduleService) -> None: + self._service = service + + async def __call__(self, record: RunRecord) -> None: + outcome = self._to_outcome(record) + if outcome is None: + return + await self._service.handle_run_completion(outcome, now=datetime.now(UTC)) + + @staticmethod + def _to_outcome(record: RunRecord) -> RunOutcome | None: + """Translate into domain vocabulary, or `None` to ignore the run. + + `None` when the run is not a scheduled execution (no usable task + metadata, no owner) or has not reached a terminal state yet. + """ + metadata = record.metadata or {} + task_id = metadata.get("scheduled_task_id") + record_id = metadata.get("scheduled_task_run_id") + user_id = record.user_id + # `metadata` is a free-form dict a caller can influence, so the ids are + # type-checked rather than assumed; `user_id` is required because every + # task read is scoped by it. + if not isinstance(task_id, str) or not isinstance(record_id, str) or not user_id: + return None + + status = _TERMINAL_STATUSES.get(str(record.status.value)) + if status is None: + return None + + if status is RunStatus.SUCCESS: + # A stale error left on a successful record must not be written back + # as the task's last_error. + error = None + elif status is RunStatus.INTERRUPTED: + error = record.error or _INTERRUPTED_WITHOUT_ERROR + else: + error = record.error + + return RunOutcome( + task_id=task_id, + record_id=record_id, + run_id=record.run_id, + user_id=user_id, + status=status, + error=error, + ) diff --git a/backend/app/adapters/schedule/run_launcher.py b/backend/app/adapters/schedule/run_launcher.py new file mode 100644 index 000000000..d3c96b6af --- /dev/null +++ b/backend/app/adapters/schedule/run_launcher.py @@ -0,0 +1,89 @@ +"""Secondary adapter (anti-corruption layer) -- starting a run through Gateway. + +Implements ``RunLauncher`` from ``deerflow.domain.schedule.ports``. This context +owns no part of the run lifecycle: it asks the Gateway to start one and +translates whatever comes back into the two outcomes the domain distinguishes. + +That translation is the reason this file exists. The Gateway signals a busy +thread two different ways -- ``ConflictError`` from the run manager, or an +``HTTPException(409)`` from the route-level path -- and the legacy scheduler +service therefore imported ``fastapi`` to tell them apart. Both are the same +domain fact, and saying so here is what keeps the web framework and the run +runtime out of the inner ring. + +TODO(hexagonal): this depends on ``launch_scheduled_thread_run``, a Gateway +service function returning an untyped dict, rather than on a contract published +by the run context -- that context has not been through a hexagonal slice yet. +When it publishes one (a DTO, not its aggregate and not its repository), +replace the body of this class. The ``RunLauncher`` port does not move. +""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable, Mapping +from typing import Any + +from fastapi import HTTPException + +from deerflow.domain.schedule.exceptions import LaunchFailedError, ThreadBusyError +from deerflow.domain.schedule.ports import LaunchedRun, RunLauncher +from deerflow.runtime import ConflictError + +LaunchRun = Callable[..., Awaitable[Mapping[str, Any]]] + + +class GatewayRunLauncher(RunLauncher): + """Adapts the Gateway's scheduled-run launch path to the ``RunLauncher`` port. + + Takes the launch callable rather than importing it, because the production + one is bound to the FastAPI app (``launch_scheduled_thread_run(app=app, + ...)``) and that binding belongs to the composition root. + + Explicit inheritance is a readability aid only: a misspelled method would + still instantiate fine and silently inherit the Protocol's ``...`` body, + so the contract tests must call every port method and assert on what it + returns. + """ + + def __init__(self, launch_run: LaunchRun) -> None: + self._launch_run = launch_run + + async def launch( + self, + *, + thread_id: str, + assistant_id: str | None, + prompt: str, + owner_user_id: str | None, + metadata: dict[str, str], + ) -> LaunchedRun: + try: + result = await self._launch_run( + thread_id=thread_id, + assistant_id=assistant_id, + prompt=prompt, + owner_user_id=owner_user_id, + metadata=metadata, + ) + except ConflictError as exc: + raise ThreadBusyError(str(exc)) from exc + except HTTPException as exc: + if exc.status_code == 409: + raise ThreadBusyError(str(exc.detail)) from exc + raise LaunchFailedError(str(exc.detail)) from exc + except Exception as exc: + # Deliberately broad: the port promises the domain that nothing but + # its two errors escapes, so an unclassifiable failure has to become + # the "genuine failure" branch rather than unwinding the poll loop. + # `CancelledError` derives from BaseException and is not caught -- + # shutdown is control flow, not a launch outcome. + raise LaunchFailedError(str(exc)) from exc + + run_id = result.get("run_id") + launched_thread_id = result.get("thread_id") + if not isinstance(run_id, str) or not isinstance(launched_thread_id, str): + # The run path broke its own contract. Reporting it as a failure + # keeps the task's bookkeeping honest instead of recording a launch + # whose run can never be traced. + raise LaunchFailedError(f"run launch returned no usable identity: {result!r}") + return LaunchedRun(run_id=run_id, thread_id=launched_thread_id) diff --git a/backend/app/adapters/schedule/scheduled_run_repository.py b/backend/app/adapters/schedule/scheduled_run_repository.py new file mode 100644 index 000000000..7c55aa427 --- /dev/null +++ b/backend/app/adapters/schedule/scheduled_run_repository.py @@ -0,0 +1,176 @@ +"""Secondary adapter (owned persistence) -- the scheduled_task_runs table in SQL. + +Implements `ScheduledRunRepository` from `deerflow.domain.schedule.ports`. This +context owns the `scheduled_task_runs` table and writes its own queries. +Queries are migrated unchanged from the legacy repository. + +The load-bearing piece is the `IntegrityError` translation in `add`: the +partial unique index `uq_scheduled_task_run_active` is the atomic arbiter of +"at most one active run per task", and turning its rejection into +`ActiveRunConflictError` is what lets the service collapse a lost race into +exactly the same outcome as its own non-atomic fast path. +""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from sqlalchemy import func, select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker + +from deerflow.domain.schedule.exceptions import ActiveRunConflictError +from deerflow.domain.schedule.model import ACTIVE_RUN_STATUSES, TERMINAL_RUN_STATUSES, RunStatus, ScheduledRun, TriggerKind +from deerflow.domain.schedule.ports import ScheduledRunRepository + +# Transitional: the ORM row stays in the harness until engine, models, and +# migrations move into app/adapters together. +from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow + +_ACTIVE_STATUS_VALUES = tuple(str(status) for status in ACTIVE_RUN_STATUSES) + + +def _tz_aware(value: datetime | None) -> datetime | None: + """SQLite drops tzinfo on read; stored values are always UTC.""" + return value if value is None or value.tzinfo is not None else value.replace(tzinfo=UTC) + + +class SqlScheduledRunRepository(ScheduledRunRepository): + """SQL implementation of the `ScheduledRunRepository` port.""" + + def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None: + self._sf = session_factory + + @staticmethod + def _to_domain(row: ScheduledTaskRunRow) -> ScheduledRun: + return ScheduledRun( + record_id=row.id, + task_id=row.task_id, + thread_id=row.thread_id, + scheduled_for=_tz_aware(row.scheduled_for), + trigger=TriggerKind(row.trigger), + status=RunStatus(row.status), + run_id=row.run_id, + error=row.error, + started_at=_tz_aware(row.started_at), + finished_at=_tz_aware(row.finished_at), + created_at=_tz_aware(row.created_at), + ) + + async def add(self, run: ScheduledRun) -> ScheduledRun: + row = ScheduledTaskRunRow( + id=run.record_id, + task_id=run.task_id, + thread_id=run.thread_id, + run_id=run.run_id, + scheduled_for=run.scheduled_for, + trigger=str(run.trigger), + status=str(run.status), + error=run.error, + started_at=run.started_at, + finished_at=run.finished_at, + created_at=run.created_at, + ) + async with self._sf() as session: + session.add(row) + try: + await session.commit() + except IntegrityError: + await session.rollback() + # Only an active-status insert can trip the partial unique + # index; a terminal row (e.g. a skip tombstone) is outside its + # predicate and cannot conflict, so an IntegrityError there is + # a genuine fault and is re-raised untranslated. + if run.is_active: + raise ActiveRunConflictError(f"scheduled task {run.task_id!r} already has an active run") from None + raise + await session.refresh(row) + return self._to_domain(row) + + async def list_by_task(self, task_id: str, *, limit: int = 50, offset: int = 0) -> list[ScheduledRun]: + stmt = ( + select(ScheduledTaskRunRow) + .where(ScheduledTaskRunRow.task_id == task_id) + .order_by( + ScheduledTaskRunRow.created_at.desc(), + ScheduledTaskRunRow.id.desc(), + ) + .limit(limit) + .offset(offset) + ) + async with self._sf() as session: + result = await session.execute(stmt) + return [self._to_domain(row) for row in result.scalars()] + + async def count_active(self) -> int: + """Global count of active rows, used to bound cross-task concurrency.""" + stmt = select(func.count()).select_from(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(_ACTIVE_STATUS_VALUES)) + async with self._sf() as session: + result = await session.execute(stmt) + return int(result.scalar() or 0) + + async def has_active(self, task_id: str) -> bool: + stmt = ( + select(ScheduledTaskRunRow.id) + .where( + ScheduledTaskRunRow.task_id == task_id, + ScheduledTaskRunRow.status.in_(_ACTIVE_STATUS_VALUES), + ) + .limit(1) + ) + async with self._sf() as session: + result = await session.execute(stmt) + return result.scalars().first() is not None + + async def update_status( + self, + record_id: str, + *, + status: RunStatus, + run_id: str | None = None, + error: str | None = None, + started_at: datetime | None = None, + finished_at: datetime | None = None, + protect_terminal: bool = False, + ) -> None: + async with self._sf() as session: + row = await session.get(ScheduledTaskRunRow, record_id) + if row is None: + return + if protect_terminal and RunStatus(row.status) in TERMINAL_RUN_STATUSES: + # The launch-path "running" write lost the race against the + # completion hook; keep the terminal status/error and only + # backfill bookkeeping the completion write could not know. + if row.run_id is None and run_id is not None: + row.run_id = run_id + if row.started_at is None and started_at is not None: + row.started_at = started_at + await session.commit() + return + row.status = str(status) + row.run_id = run_id + row.error = error + if started_at is not None: + row.started_at = started_at + if finished_at is not None: + row.finished_at = finished_at + await session.commit() + + async def mark_stale_active(self, *, error: str) -> int: + """Fail-fast bookkeeping for runs orphaned by a process crash. + + Agent runs execute in-process, so any active row found at scheduler + startup belongs to a run whose process is gone. Only valid under the + single-scheduler-instance assumption. + """ + stmt = select(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(_ACTIVE_STATUS_VALUES)) + now = datetime.now(UTC) + async with self._sf() as session: + result = await session.execute(stmt) + rows = list(result.scalars()) + for row in rows: + row.status = str(RunStatus.INTERRUPTED) + row.error = error + row.finished_at = now + await session.commit() + return len(rows) diff --git a/backend/app/adapters/schedule/scheduled_task_repository.py b/backend/app/adapters/schedule/scheduled_task_repository.py new file mode 100644 index 000000000..6741bc573 --- /dev/null +++ b/backend/app/adapters/schedule/scheduled_task_repository.py @@ -0,0 +1,341 @@ +"""Secondary adapter (owned persistence) -- the scheduled_tasks table in SQL. + +Implements `ScheduledTaskRepository` from `deerflow.domain.schedule.ports`. +This context owns the `scheduled_tasks` table and writes its own queries, so +SQL/ORM vocabulary stops at this file: methods exchange domain objects and +normalize SQLite's tz-naive reads. + +Sibling of `scheduled_run_repository.py`. Translating the row is this class's +own private business, `_to_domain` / `_apply` and the two `schedule_spec` +helpers beside them -- the HTTP shape has its own translation next to the +router, and neither side imports the other. + +**The queries are migrated unchanged from the legacy repository.** The claim +statement's `FOR UPDATE SKIP LOCKED` and the `protect_terminal` conditional +write are the module's concurrency contract, not style choices -- rewriting +them is how the invariants silently break. +""" + +from __future__ import annotations + +import logging +import socket +import uuid +from datetime import UTC, datetime, timedelta + +from sqlalchemy import and_, or_, select +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker + +from deerflow.domain.schedule.exceptions import CorruptStoredScheduleError, ScheduleError +from deerflow.domain.schedule.model import TERMINAL_TASK_STATUSES, ContextMode, ScheduledTask, ScheduleSpec, ScheduleType, TaskStatus +from deerflow.domain.schedule.ports import ScheduledTaskRepository + +# Transitional: the ORM row stays in the harness until engine, models, and +# migrations move into app/adapters together. +from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow + +logger = logging.getLogger(__name__) + + +def _tz_aware(value: datetime | None) -> datetime | None: + """SQLite drops tzinfo on read; stored values are always UTC.""" + return value if value is None or value.tzinfo is not None else value.replace(tzinfo=UTC) + + +class SqlScheduledTaskRepository(ScheduledTaskRepository): + """SQL implementation of the `ScheduledTaskRepository` port. + + Explicit inheritance is a readability aid only: a missing method would + still instantiate fine (Protocol bodies are inherited), so the contract + test suite must cover every port method. + """ + + def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None: + self._sf = session_factory + # Diagnostics only. The domain deliberately does not carry this -- who + # claimed a task is an identity, not a rule, and nothing reads it back. + self._lease_owner = f"{socket.gethostname()}:{uuid.uuid4().hex}" + + # ------------------------------------------------------------ conversion + + @staticmethod + def _spec_from_row(row: ScheduledTaskRow) -> ScheduleSpec: + """The stored triple -> the value object. + + All this side owns is which two keys this table's JSON uses; the rule + for turning them into a schedule belongs to the domain, which is why + the router's own mapping can state the same thing without either + importing the other. + """ + stored = row.schedule_spec or {} + return ScheduleSpec.from_primitives( + row.schedule_type, + cron=stored.get("cron"), + run_at=stored.get("run_at"), + timezone=row.timezone, + ) + + @staticmethod + def _spec_to_column(spec: ScheduleSpec) -> dict[str, str]: + """The value object -> the stored JSON. + + Emits the normalized value rather than echoing the caller's bytes: a + trailing-Z input is stored as "+00:00". Both forms parse back, so this + is deliberate -- preferable to carrying the raw dict on the value + object just to preserve the exact input spelling. + """ + if spec.schedule_type is ScheduleType.CRON: + return {"cron": spec.cron or ""} + return {"run_at": spec.run_at.isoformat() if spec.run_at else ""} + + @staticmethod + def _to_domain(row: ScheduledTaskRow) -> ScheduledTask: + """ORM row -> aggregate. + + Raises CorruptStoredScheduleError for a row that can no longer be + rebuilt -- unparseable schedule, unknown enum value. The rebuild goes + through the aggregate's own validation, so a hand-damaged row blows up + here rather than flowing on; the translation to the dedicated error is + what keeps it off ``InvalidScheduleError``'s client-facing 422 mapping + (a stored fault is the server's problem, and PATCH -- which reads + before writing -- must not become unusable for the one row it could + repair). Single-row reads let it propagate; list reads skip and log, + so one bad row cannot take down an entire listing. + """ + try: + return ScheduledTask( + task_id=row.id, + user_id=row.user_id, + title=row.title, + prompt=row.prompt, + schedule=SqlScheduledTaskRepository._spec_from_row(row), + context_mode=ContextMode(row.context_mode), + thread_id=row.thread_id, + assistant_id=row.assistant_id, + status=TaskStatus(row.status), + overlap_policy=row.overlap_policy, + next_run_at=_tz_aware(row.next_run_at), + last_run_at=_tz_aware(row.last_run_at), + last_run_id=row.last_run_id, + last_thread_id=row.last_thread_id, + last_error=row.last_error, + run_count=row.run_count, + created_at=_tz_aware(row.created_at), + updated_at=_tz_aware(row.updated_at), + ) + except (ScheduleError, ValueError) as exc: + raise CorruptStoredScheduleError(f"stored scheduled task {row.id} cannot be rebuilt: {exc}") from exc + + @classmethod + def _to_domain_or_skip(cls, row: ScheduledTaskRow) -> ScheduledTask | None: + try: + return cls._to_domain(row) + except CorruptStoredScheduleError: + logger.warning("Skipping scheduled task %s: stored row cannot be rebuilt", row.id, exc_info=True) + return None + + @staticmethod + def _apply(row: ScheduledTaskRow, task: ScheduledTask) -> None: + """Aggregate -> ORM row. Explicit field list: a new column stays + invisible here until it is deliberately mapped.""" + row.user_id = task.user_id + row.title = task.title + row.prompt = task.prompt + row.schedule_type = str(task.schedule.schedule_type) + row.schedule_spec = SqlScheduledTaskRepository._spec_to_column(task.schedule) + row.timezone = task.schedule.timezone + row.context_mode = str(task.context_mode) + row.thread_id = task.thread_id + row.assistant_id = task.assistant_id + row.status = str(task.status) + row.overlap_policy = task.overlap_policy + row.next_run_at = task.next_run_at + row.last_run_at = task.last_run_at + row.last_run_id = task.last_run_id + row.last_thread_id = task.last_thread_id + row.last_error = task.last_error + row.run_count = task.run_count + + # ------------------------------------------------------------------ port + + async def add(self, task: ScheduledTask) -> ScheduledTask: + # The aggregate's construction instant is the one truth for both + # bookkeeping stamps on insert; the later write paths own updated_at. + row = ScheduledTaskRow(id=task.task_id, created_at=task.created_at, updated_at=task.updated_at) + self._apply(row, task) + async with self._sf() as session: + session.add(row) + await session.commit() + await session.refresh(row) + return self._to_domain(row) + + async def get(self, task_id: str, *, user_id: str) -> ScheduledTask | None: + async with self._sf() as session: + row = await session.get(ScheduledTaskRow, task_id) + if row is None or row.user_id != user_id: + return None + return self._to_domain(row) + + async def list_by_user(self, user_id: str) -> list[ScheduledTask]: + stmt = select(ScheduledTaskRow).where(ScheduledTaskRow.user_id == user_id).order_by(ScheduledTaskRow.created_at.desc(), ScheduledTaskRow.id.desc()) + async with self._sf() as session: + result = await session.execute(stmt) + return [task for task in (self._to_domain_or_skip(row) for row in result.scalars()) if task is not None] + + async def list_by_user_and_thread(self, user_id: str, thread_id: str) -> list[ScheduledTask]: + stmt = ( + select(ScheduledTaskRow) + .where( + ScheduledTaskRow.user_id == user_id, + ScheduledTaskRow.thread_id == thread_id, + ) + .order_by(ScheduledTaskRow.created_at.desc(), ScheduledTaskRow.id.desc()) + ) + async with self._sf() as session: + result = await session.execute(stmt) + return [task for task in (self._to_domain_or_skip(row) for row in result.scalars()) if task is not None] + + async def save(self, task: ScheduledTask) -> ScheduledTask | None: + async with self._sf() as session: + row = await session.get(ScheduledTaskRow, task.task_id) + if row is None or row.user_id != task.user_id: + return None + self._apply(row, task) + row.updated_at = datetime.now(UTC) + await session.commit() + await session.refresh(row) + return self._to_domain(row) + + async def delete(self, task_id: str, *, user_id: str) -> bool: + async with self._sf() as session: + row = await session.get(ScheduledTaskRow, task_id) + if row is None or row.user_id != user_id: + return False + await session.delete(row) + await session.commit() + return True + + async def claim_due(self, *, now: datetime, lease_seconds: int, limit: int) -> list[ScheduledTask]: + lease_expires_at = now + timedelta(seconds=lease_seconds) + stmt = ( + select(ScheduledTaskRow) + .where( + ScheduledTaskRow.next_run_at.is_not(None), + ScheduledTaskRow.next_run_at <= now, + or_( + and_( + ScheduledTaskRow.status == "enabled", + or_( + ScheduledTaskRow.lease_expires_at.is_(None), + ScheduledTaskRow.lease_expires_at < now, + ), + ), + # A task stuck in "running" with an expired lease means the + # claiming process died between claim and dispatch; it must + # stay reclaimable or the task is dead forever. + and_( + ScheduledTaskRow.status == "running", + ScheduledTaskRow.lease_expires_at.is_not(None), + ScheduledTaskRow.lease_expires_at < now, + ), + ), + ) + .order_by(ScheduledTaskRow.next_run_at.asc(), ScheduledTaskRow.id.asc()) + .limit(limit) + .with_for_update(skip_locked=True) + ) + async with self._sf() as session: + result = await session.execute(stmt) + rows = list(result.scalars()) + for row in rows: + row.lease_owner = self._lease_owner + row.lease_expires_at = lease_expires_at + row.status = "running" + row.updated_at = datetime.now(UTC) + await session.commit() + return [task for task in (self._to_domain_or_skip(row) for row in rows) if task is not None] + + async def record_launch( + self, + task_id: str, + *, + status: TaskStatus, + next_run_at: datetime | None, + last_run_at: datetime | None, + last_run_id: str | None, + last_thread_id: str | None, + last_error: str | None, + increment_run_count: bool, + protect_terminal: bool = False, + ) -> None: + async with self._sf() as session: + row = await session.get(ScheduledTaskRow, task_id) + if row is None: + return + if protect_terminal and TaskStatus(row.status) in TERMINAL_TASK_STATUSES: + # A fast-failing run can reach the completion hook (which + # finalizes a `once` task) before this launch-path write + # commits; keep the hook's status/error and only record the + # launch bookkeeping. + pass + else: + row.status = str(status) + row.last_error = last_error + row.next_run_at = next_run_at + row.last_run_at = last_run_at + row.last_run_id = last_run_id + row.last_thread_id = last_thread_id + if increment_run_count: + row.run_count += 1 + row.lease_owner = None + row.lease_expires_at = None + row.updated_at = datetime.now(UTC) + await session.commit() + + async def record_completion( + self, + task_id: str, + *, + user_id: str, + status: TaskStatus | None, + error: str | None, + ) -> None: + async with self._sf() as session: + row = await session.get(ScheduledTaskRow, task_id) + if row is None or row.user_id != user_id: + return + # Field-level on purpose: `record_launch` may commit on either side + # of this write, and the two must not undo each other. Assigning + # anything below `last_error` here is what reintroduces the race -- + # see the port docstring. + row.last_error = error + if status is not None: + row.status = str(status) + row.updated_at = datetime.now(UTC) + await session.commit() + + async def cancel_stuck_once_tasks(self, *, error: str) -> int: + """Reconcile `once` tasks orphaned in `running` by a process crash. + + A launched `once` task stays `running` until the in-process completion + hook moves it to a terminal status; its lease was cleared at launch, so + the claim query's expired-lease branch never sees it. After a crash the + hook is gone and the task would be stuck forever. Tasks still holding a + lease are left alone -- they were claimed but not launched, and + expired-lease reclaim recovers them safely. + """ + stmt = select(ScheduledTaskRow).where( + ScheduledTaskRow.schedule_type == "once", + ScheduledTaskRow.status == "running", + ScheduledTaskRow.lease_expires_at.is_(None), + ) + async with self._sf() as session: + result = await session.execute(stmt) + rows = list(result.scalars()) + now = datetime.now(UTC) + for row in rows: + row.status = "cancelled" + row.last_error = error + row.updated_at = now + await session.commit() + return len(rows) diff --git a/backend/app/adapters/schedule/thread_lookup.py b/backend/app/adapters/schedule/thread_lookup.py new file mode 100644 index 000000000..9192f6f40 --- /dev/null +++ b/backend/app/adapters/schedule/thread_lookup.py @@ -0,0 +1,46 @@ +"""Secondary adapter (anti-corruption layer) -- narrowing ThreadMetaStore to ThreadLookup. + +Implements ``ThreadLookup`` from ``deerflow.domain.schedule.ports``. This +context does not own the ``threads_meta`` table and writes no SQL against it: it +asks its one question through the store the thread context already provides. + +``require_existing=True`` is the load-bearing argument. The store's default +treats an absent row as accessible -- reasonable for a thread that has not been +written yet, wrong for binding a task to it, since the task would then reference +a thread that never existed. + +TODO(hexagonal): this depends on ``ThreadMetaStore``, an infrastructure +component, rather than on a contract published by the thread context -- that +context has not been through a hexagonal slice yet. When it publishes one (a +DTO, not its aggregate and not its repository), replace the body of this class. +The ``ThreadLookup`` port does not move. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from deerflow.domain.schedule.ports import ThreadLookup + +if TYPE_CHECKING: + from deerflow.persistence.thread_meta.base import ThreadMetaStore + + +class ThreadStoreThreadLookup(ThreadLookup): + """Adapts the wide ``ThreadMetaStore`` to the one question this context asks. + + Both halves of that question -- does the thread exist, and may this user use + it -- collapse into a single bool on purpose: reporting them separately + would let a caller probe for the existence of threads they cannot see. + + Explicit inheritance is a readability aid only: a misspelled method would + still instantiate fine and silently inherit the Protocol's ``...`` body, + so the contract tests must call every port method and assert on what it + returns. + """ + + def __init__(self, thread_store: ThreadMetaStore) -> None: + self._thread_store = thread_store + + async def exists_for_user(self, thread_id: str, user_id: str) -> bool: + return bool(await self._thread_store.check_access(thread_id, user_id, require_existing=True)) diff --git a/backend/app/channels/manager.py b/backend/app/channels/manager.py index 72be91337..fd50727ad 100644 --- a/backend/app/channels/manager.py +++ b/backend/app/channels/manager.py @@ -930,7 +930,7 @@ class ChannelManager: self._require_bound_identity = require_bound_identity # Zero-arg accessor for the FastAPI app's StreamBridge singleton, # threaded in from app.py's lifespan via start_channel_service() -> - # ChannelService.__init__ (mirrors how ScheduledTaskService gets a + # ChannelService.__init__ (mirrors how the schedule composition root gets a # launch_run closure over `app` in the same lifespan function). None # when not wired (e.g. a ChannelManager constructed directly in # tests) — follow-up buffering still works, but no watcher is diff --git a/backend/app/channels/service.py b/backend/app/channels/service.py index f46724beb..bab30d9cc 100644 --- a/backend/app/channels/service.py +++ b/backend/app/channels/service.py @@ -441,7 +441,7 @@ async def start_channel_service( ``ChannelRunPolicy.buffer_followups_on_busy`` (currently GitHub) can watch a run's completion and auto-drain buffered follow-ups. ``app.py``'s lifespan passes a closure over ``app.state.stream_bridge`` here, the same - pattern it already uses for ``ScheduledTaskService``'s ``launch_run``. + pattern it already uses for the schedule composition root's ``launch_run``. """ global _channel_service if _channel_service is not None: diff --git a/backend/app/composition.py b/backend/app/composition.py new file mode 100644 index 000000000..0f38cecee --- /dev/null +++ b/backend/app/composition.py @@ -0,0 +1,119 @@ +"""Composition root -- the one place adapters are instantiated and wired. + +Every application service is assembled here and nowhere else. Ports are +declared by the domain, implemented under ``app/adapters/``, and the two are +introduced to each other in this file; no router, middleware, or lifespan hook +constructs an adapter of its own. + +**Why this is a pure function rather than part of the lifespan.** Wiring used +to live inside ``deps.py::langgraph_runtime``, tangled with engine startup, +orphan recovery, and graceful shutdown -- so the one rule that actually +governs it ("no SQL backend means no service, and the routes answer 503") +could not be tested without driving a full application startup, and was held +up by a single comment. ``build_domain_services`` takes what it needs and +returns what it built, so that rule is an assertion in +``tests/test_composition.py`` instead. + +**What "no SQL backend" means.** ``session_factory is None`` is how a +``database.backend: memory`` deployment presents itself. The context owns +tables, so it cannot run on it; the service is ``None`` and the dependency +providers translate that into 503. This is deliberately not a silent +degradation to an in-memory implementation -- scheduled work that vanishes on +restart is worse than scheduled work that is refused. +""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable, Mapping +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +from deerflow.domain.schedule.model import SchedulePolicy +from deerflow.domain.schedule.service import ScheduleService + +if TYPE_CHECKING: + from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker + + from deerflow.config.scheduler_config import SchedulerConfig + from deerflow.persistence.thread_meta.base import ThreadMetaStore + + +@dataclass(frozen=True) +class DomainServices: + """Every application service the Gateway serves, or ``None`` where the + configured backend cannot support one.""" + + schedule: ScheduleService | None + + +def build_domain_services( + *, + session_factory: async_sessionmaker[AsyncSession] | None, + thread_store: ThreadMetaStore, + launch_run: Callable[..., Awaitable[Mapping[str, Any]]], + scheduler_config: SchedulerConfig, +) -> DomainServices: + """Wire the domain services from already-built infrastructure. + + Takes infrastructure rather than building it: engines, stores and the run + launcher have lifecycles (startup, recovery, shutdown) that belong to the + lifespan, while deciding *what is assembled from them* is this function's + only job. That split is what makes it callable from a test. + """ + if session_factory is None: + return DomainServices(schedule=None) + + # Imported here, not at module scope: these pull in SQLAlchemy and the + # Gateway run path, and the composition root is imported by tests that + # only want the memory-backend branch. + from app.adapters.schedule.run_launcher import GatewayRunLauncher + from app.adapters.schedule.scheduled_run_repository import SqlScheduledRunRepository + from app.adapters.schedule.scheduled_task_repository import SqlScheduledTaskRepository + from app.adapters.schedule.thread_lookup import ThreadStoreThreadLookup + + return DomainServices( + schedule=ScheduleService( + tasks=SqlScheduledTaskRepository(session_factory), + runs=SqlScheduledRunRepository(session_factory), + launcher=GatewayRunLauncher(launch_run), + threads=ThreadStoreThreadLookup(thread_store), + policy=build_schedule_policy(scheduler_config), + ), + ) + + +def build_run_completion_hook( + schedule_service: ScheduleService | None, +) -> Callable[[Any], Awaitable[None]] | None: + """Install the schedule context on the run runtime's completion callback. + + The inbound half of the wiring, and nothing more: which runs the context + cares about and what it does with them belongs to the adapter, not here. + This function's only decision is the same one the rest of this module + makes -- what to assemble, and what to leave unassembled. + + Returns ``None`` when there is no service, so the runtime installs no hook + at all rather than one that always declines. + """ + if schedule_service is None: + return None + + from app.adapters.schedule.run_completion import ScheduleRunCompletionListener + + return ScheduleRunCompletionListener(schedule_service) + + +def build_schedule_policy(scheduler_config: SchedulerConfig) -> SchedulePolicy: + """Project the operator's scheduler config onto the domain's policy. + + Separate from the wiring above because it is the whole of the + configuration-to-domain translation: the domain declares which thresholds + it needs, and this names where each one comes from. `poll_interval_seconds` + is deliberately absent -- how often to look is the poller's business, not a + rule any task is subject to. + """ + return SchedulePolicy( + min_once_delay_seconds=scheduler_config.min_once_delay_seconds, + max_concurrent_runs=scheduler_config.max_concurrent_runs, + lease_seconds=scheduler_config.lease_seconds, + ) diff --git a/backend/app/gateway/app.py b/backend/app/gateway/app.py index 705f9be7d..08f0d0246 100644 --- a/backend/app/gateway/app.py +++ b/backend/app/gateway/app.py @@ -30,13 +30,13 @@ from app.gateway.routers import ( memory, models, runs, - scheduled_tasks, skills, suggestions, thread_runs, threads, uploads, ) +from app.gateway.routers.schedule import router as schedule_router from app.gateway.trace_middleware import TraceMiddleware, resolve_trace_enabled from deerflow.config import app_config as deerflow_app_config from deerflow.logging_config import DEFAULT_LOG_DATE_FORMAT, DEFAULT_LOG_FORMAT, configure_logging @@ -287,7 +287,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: try: from app.channels.service import start_channel_service - # Closure over `app` (mirrors ScheduledTaskService's `launch_run` + # Closure over `app` (mirrors the schedule composition root's `launch_run` # below) rather than resolving `app.state.stream_bridge` here # directly: `stream_bridge` is a STARTUP_ONLY_FIELDS singleton set # once, above, by `langgraph_runtime(app, startup_config)`, so @@ -305,23 +305,23 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: logger.exception("No IM channels configured or channel service failed to start") try: - from app.gateway.services import launch_scheduled_thread_run - from app.scheduler import ScheduledTaskService + # The service itself was assembled by the composition root inside + # `langgraph_runtime`; all that is left here is the clock that + # drives it. It is None when the configured backend cannot support + # scheduling, in which case there is nothing to poll. + from app.scheduler.poller import SchedulePoller - if getattr(app.state, "scheduled_task_repo", None) is not None and getattr(app.state, "scheduled_task_run_repo", None) is not None: - scheduled_task_service = ScheduledTaskService( - task_repo=app.state.scheduled_task_repo, - task_run_repo=app.state.scheduled_task_run_repo, - launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs), + schedule_service = getattr(app.state, "schedule_service", None) + if schedule_service is not None: + schedule_poller = SchedulePoller( + schedule_service, poll_interval_seconds=startup_config.scheduler.poll_interval_seconds, - lease_seconds=startup_config.scheduler.lease_seconds, - max_concurrent_runs=startup_config.scheduler.max_concurrent_runs, ) - app.state.scheduled_task_service = scheduled_task_service + app.state.schedule_poller = schedule_poller if startup_config.scheduler.enabled: - await scheduled_task_service.start() + await schedule_poller.start() except Exception: - logger.exception("Failed to initialize scheduled task service") + logger.exception("Failed to start the scheduled task poller") yield @@ -346,11 +346,11 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: except Exception: logger.exception("Failed to stop channel service") - if getattr(app.state, "scheduled_task_service", None) is not None: + if getattr(app.state, "schedule_poller", None) is not None: try: - await app.state.scheduled_task_service.stop() + await app.state.schedule_poller.stop() except Exception: - logger.exception("Failed to stop scheduled task service") + logger.exception("Failed to stop the scheduled task poller") try: from deerflow.community.browser_automation import get_browser_session_manager @@ -598,7 +598,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for app.include_router(threads.router) # Scheduled tasks API is mounted at /api/scheduled-tasks - app.include_router(scheduled_tasks.router) + app.include_router(schedule_router.router) # Agents API is mounted at /api/agents app.include_router(agents.router) diff --git a/backend/app/gateway/deps.py b/backend/app/gateway/deps.py index 9c5540b80..b293ee863 100644 --- a/backend/app/gateway/deps.py +++ b/backend/app/gateway/deps.py @@ -22,13 +22,14 @@ import logging import os from collections.abc import AsyncGenerator, Callable from contextlib import AsyncExitStack, asynccontextmanager -from typing import TYPE_CHECKING, TypeVar, cast +from typing import TYPE_CHECKING, Annotated, TypeVar, cast -from fastapi import FastAPI, HTTPException, Request +from fastapi import Depends, FastAPI, HTTPException, Request from langgraph.types import Checkpointer from deerflow.community.browser_automation.session import browser_multi_worker_error from deerflow.config.app_config import AppConfig, get_app_config +from deerflow.domain.schedule.service import ScheduleService from deerflow.persistence.feedback import FeedbackRepository from deerflow.runtime import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, RunContext, RunManager, StreamBridge from deerflow.runtime.events.store.base import RunEventStore @@ -415,17 +416,25 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen from deerflow.persistence.thread_meta import make_thread_store app.state.thread_store = make_thread_store(sf, app.state.store) - if sf is not None: - from deerflow.persistence.scheduled_task_runs import ( - ScheduledTaskRunRepository, - ) - from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository - app.state.scheduled_task_repo = ScheduledTaskRepository(sf) - app.state.scheduled_task_run_repo = ScheduledTaskRunRepository(sf) - else: - app.state.scheduled_task_repo = None - app.state.scheduled_task_run_repo = None + # Hexagonal slices: every adapter is instantiated by the composition + # root and nowhere else. It is a pure function of the infrastructure + # built above, so what it decides — including "no SQL backend means no + # service, and the routes answer 503" — is covered by + # tests/test_composition.py rather than only by this call site. + from app.composition import build_domain_services, build_run_completion_hook + from app.gateway.services import launch_scheduled_thread_run + + domain_services = build_domain_services( + session_factory=sf, + thread_store=app.state.thread_store, + launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs), + scheduler_config=config.scheduler, + ) + app.state.schedule_service = domain_services.schedule + # Built once here rather than per request: it closes over the service, + # and every RunContext needs the same one. + app.state.run_completion_hook = build_run_completion_hook(domain_services.schedule) # Run event store. The store and the matching ``run_events_config`` are # both frozen at startup so ``get_run_context`` does not combine a @@ -556,25 +565,23 @@ def get_thread_store(request: Request) -> ThreadMetaStore: return val -def get_scheduled_task_repo(request: Request): - val = getattr(request.app.state, "scheduled_task_repo", None) +def get_schedule_service(request: Request) -> ScheduleService: + """The schedule context's input port. + + ``None`` here means the composition root refused to build it, which today + means a non-SQL backend -- see ``app/composition.py`` for why that is a + 503 rather than a degraded in-memory implementation. + """ + val = getattr(request.app.state, "schedule_service", None) if val is None: - raise HTTPException(status_code=503, detail="Scheduled task repo not available") + raise HTTPException(status_code=503, detail="Scheduled tasks require a SQL database backend") return val -def get_scheduled_task_run_repo(request: Request): - val = getattr(request.app.state, "scheduled_task_run_repo", None) - if val is None: - raise HTTPException(status_code=503, detail="Scheduled task run repo not available") - return val - - -def get_scheduled_task_service(request: Request): - val = getattr(request.app.state, "scheduled_task_service", None) - if val is None: - raise HTTPException(status_code=503, detail="Scheduled task service not available") - return val +# Declared as an alias so routes take the service without a default argument +# (leaving parameter order unconstrained) and so swapping the provider is a +# one-line change here rather than an edit to every handler signature. +ScheduleServiceDep = Annotated[ScheduleService, Depends(get_schedule_service)] def get_run_context(request: Request) -> RunContext: @@ -596,7 +603,7 @@ def get_run_context(request: Request) -> RunContext: checkpoint_snapshot_frequency=getattr(request.app.state, "checkpoint_snapshot_frequency", None), thread_store=get_thread_store(request), app_config=get_config(), - on_run_completed=getattr(request.app.state, "scheduled_task_service", None).handle_run_completion if getattr(request.app.state, "scheduled_task_service", None) is not None else None, + on_run_completed=getattr(request.app.state, "run_completion_hook", None), ) diff --git a/backend/app/gateway/routers/__init__.py b/backend/app/gateway/routers/__init__.py index a948c522b..27afb4341 100644 --- a/backend/app/gateway/routers/__init__.py +++ b/backend/app/gateway/routers/__init__.py @@ -5,7 +5,6 @@ from . import ( input_polish, mcp, models, - scheduled_tasks, skills, suggestions, thread_runs, @@ -20,7 +19,6 @@ __all__ = [ "input_polish", "mcp", "models", - "scheduled_tasks", "skills", "suggestions", "threads", diff --git a/backend/app/gateway/routers/schedule/__init__.py b/backend/app/gateway/routers/schedule/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/app/gateway/routers/schedule/models.py b/backend/app/gateway/routers/schedule/models.py new file mode 100644 index 000000000..e2cb833a7 --- /dev/null +++ b/backend/app/gateway/routers/schedule/models.py @@ -0,0 +1,263 @@ +"""The HTTP shapes of the scheduled-task API. + +The primary adapter's own model of what a client sends and receives -- the +counterpart to the domain aggregates, not a view of them. Two things follow +from that: + +**Responses are an allowlist, not a dump.** The pre-migration router returned +the ORM row's ``to_dict()``, which leaked ``user_id``, ``lease_owner``, +``lease_expires_at``, ``overlap_policy`` and ``assistant_id`` -- lease fields +are scheduler-internal bookkeeping, and the other three are server-owned. None +appear in the frontend's ``ScheduledTask`` type or anywhere in its code, so +naming the fields explicitly here closes the leak without a client change. A +field added to the aggregate from now on stays invisible until it is +deliberately published. + +**Timestamps keep the legacy spelling.** The legacy path emitted +``coerce_iso`` -> ``astimezone(UTC).isoformat()``, i.e. +``2026-08-01T09:00:00+00:00``. Pydantic v2 would serialize the same instant as +``...T09:00:00Z``, which is a silent wire change for every client parsing +these, so ``UtcTimestamp`` pins ``isoformat()`` explicitly. Both spellings are +valid ISO 8601 and JS ``Date`` accepts either -- the point is that changing it +is a decision, not a side effect of adopting a model. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Annotated, Any + +from pydantic import BaseModel, Field, PlainSerializer + +from deerflow.domain.schedule.commands import UNSET, ContextChange, CreateScheduledTask, UpdateScheduledTask +from deerflow.domain.schedule.model import ScheduledRun, ScheduledTask, ScheduleSpec, ScheduleType + +UtcTimestamp = Annotated[datetime, PlainSerializer(lambda value: value.isoformat(), return_type=str)] + + +def _utc(value: datetime | None) -> datetime | None: + """Match the legacy `coerce_iso` normalisation exactly.""" + if value is None: + return None + return value.astimezone(UTC) if value.tzinfo is not None else value.replace(tzinfo=UTC) + + +def _spec_to_wire(spec: ScheduleSpec) -> dict[str, str]: + """The value object -> the `schedule_spec` body field. + + Emits the normalized value rather than echoing the caller's bytes: the + frontend submits an already-UTC-aware ISO value (`zonedLocalToUtcIso`), so + a trailing-Z input comes back as "+00:00". Both forms parse on either side, + so the normalization is deliberate. + + All this side owns is which two keys the body uses. Its counterpart is + `SqlScheduledTaskRepository._spec_to_column`; the two are near-identical + today by coincidence, and are kept apart because a primary adapter must not + import a secondary one and because the day the API grows a field the column + does not have, they diverge without either having to be untangled. + """ + if spec.schedule_type is ScheduleType.CRON: + return {"cron": spec.cron or ""} + return {"run_at": spec.run_at.isoformat() if spec.run_at else ""} + + +class CreateScheduledTaskRequest(BaseModel): + """Body of ``POST /scheduled-tasks``. + + No identity field: ``user_id`` is resolved server-side and injected + through ``to_command``, never read off the wire. + """ + + thread_id: str | None = None + context_mode: str = "fresh_thread_per_run" + title: str = Field(min_length=1) + prompt: str = Field(min_length=1) + schedule_type: str + schedule_spec: dict[str, Any] + timezone: str + + def to_command(self, user_id: str) -> CreateScheduledTask: + """Wire shape -> command (transformation ① of the chain).""" + return CreateScheduledTask( + user_id=user_id, + title=self.title, + prompt=self.prompt, + schedule=self.to_schedule(), + context_mode=self.context_mode, + thread_id=self.thread_id, + ) + + def to_schedule(self) -> ScheduleSpec: + """Parse the submitted triple into the value object. + + Raises `InvalidScheduleError` -- a *domain* error out of a primary + adapter, on purpose: it is the vocabulary the outer ring uses to say + "this violates a domain rule", and the router maps that one family onto + 422. Structural problems (key missing, wrong type) and value problems + (5-field cron, resolvable timezone) both arrive as it. + """ + return ScheduleSpec.from_primitives( + self.schedule_type, + cron=self.schedule_spec.get("cron"), + run_at=self.schedule_spec.get("run_at"), + timezone=self.timezone, + ) + + +class UpdateScheduledTaskRequest(BaseModel): + """Body of ``PATCH /scheduled-tasks/{task_id}``. + + On the wire, ``None`` means "not supplied" -- an explicit ``null`` has + always meant that on this endpoint, and unbinding a thread is expressed + by switching ``context_mode``, not by nulling ``thread_id``. The + ``None``-to-``UNSET`` translation in ``to_command`` is where that wire + convention meets the command's unambiguous three-state fields. + """ + + context_mode: str | None = None + thread_id: str | None = None + title: str | None = Field(default=None, min_length=1) + prompt: str | None = Field(default=None, min_length=1) + schedule_spec: dict[str, Any] | None = None + timezone: str | None = None + + def changes_schedule(self) -> bool: + return self.schedule_spec is not None or self.timezone is not None + + def changes_context(self) -> bool: + return self.context_mode is not None or self.thread_id is not None + + def to_command(self, task_id: str, user_id: str, current: ScheduledTask | None) -> UpdateScheduledTask: + """Wire shape -> command (transformation ① of the chain). + + A schedule or context change needs the parts the client omitted, so + the router reads the current task once and passes it in -- this model + stays IO-free and only translates. ``current`` may be ``None`` when + neither composite field is being changed. + """ + schedule = UNSET + if current is not None and self.changes_schedule(): + schedule = self.to_schedule(current.schedule) + context = UNSET + if current is not None and self.changes_context(): + context = ContextChange( + context_mode=self.context_mode if self.context_mode is not None else str(current.context_mode), + thread_id=self.thread_id if self.thread_id is not None else current.thread_id, + ) + return UpdateScheduledTask( + task_id=task_id, + user_id=user_id, + title=self.title if self.title is not None else UNSET, + prompt=self.prompt if self.prompt is not None else UNSET, + schedule=schedule, + context=context, + ) + + def to_schedule(self, current: ScheduleSpec) -> ScheduleSpec: + """Build the replacement spec, taking what was omitted from `current`. + + The schedule *type* is not patchable; only its spec and its zone are. + Omitted parts are read straight off the current value object rather + than round-tripped through the wire shape and back. + """ + if self.schedule_spec is not None: + cron = self.schedule_spec.get("cron") + run_at = self.schedule_spec.get("run_at") + else: + cron = current.cron + run_at = current.run_at.isoformat() if current.run_at else None + return ScheduleSpec.from_primitives( + str(current.schedule_type), + cron=cron, + run_at=run_at, + timezone=self.timezone if self.timezone is not None else current.timezone, + ) + + +class ScheduledTaskResponse(BaseModel): + """One scheduled task as the client sees it. + + Mirrors the frontend's `ScheduledTask` type field for field. + """ + + id: str + thread_id: str | None + context_mode: str + title: str + prompt: str + schedule_type: str + schedule_spec: dict[str, str] + timezone: str + status: str + next_run_at: UtcTimestamp | None + last_run_at: UtcTimestamp | None + last_run_id: str | None + last_thread_id: str | None + last_error: str | None + run_count: int + created_at: UtcTimestamp + updated_at: UtcTimestamp + + @classmethod + def from_domain(cls, task: ScheduledTask) -> ScheduledTaskResponse: + return cls( + id=task.task_id, + thread_id=task.thread_id, + context_mode=str(task.context_mode), + title=task.title, + prompt=task.prompt, + schedule_type=str(task.schedule.schedule_type), + schedule_spec=_spec_to_wire(task.schedule), + timezone=task.schedule.timezone, + status=str(task.status), + next_run_at=_utc(task.next_run_at), + last_run_at=_utc(task.last_run_at), + last_run_id=task.last_run_id, + last_thread_id=task.last_thread_id, + last_error=task.last_error, + run_count=task.run_count, + created_at=_utc(task.created_at), + updated_at=_utc(task.updated_at), + ) + + +class ScheduledRunResponse(BaseModel): + """One execution record. Mirrors the frontend's `ScheduledTaskRun` type.""" + + id: str + task_id: str + thread_id: str + run_id: str | None + scheduled_for: UtcTimestamp + trigger: str + status: str + error: str | None + started_at: UtcTimestamp | None + finished_at: UtcTimestamp | None + created_at: UtcTimestamp + + @classmethod + def from_domain(cls, run: ScheduledRun) -> ScheduledRunResponse: + return cls( + id=run.record_id, + task_id=run.task_id, + thread_id=run.thread_id, + run_id=run.run_id, + scheduled_for=_utc(run.scheduled_for), + trigger=str(run.trigger), + status=str(run.status), + error=run.error, + started_at=_utc(run.started_at), + finished_at=_utc(run.finished_at), + created_at=_utc(run.created_at), + ) + + +class TriggerResponse(BaseModel): + id: str + triggered: bool + + +class DeleteResponse(BaseModel): + id: str + deleted: bool diff --git a/backend/app/gateway/routers/schedule/router.py b/backend/app/gateway/routers/schedule/router.py new file mode 100644 index 000000000..5443712c1 --- /dev/null +++ b/backend/app/gateway/routers/schedule/router.py @@ -0,0 +1,195 @@ +"""Primary adapter -- the scheduled-task HTTP API. + +Everything here is protocol translation. Parse the body into domain values, +call one service method, render the result, and map domain errors onto status +codes. What is deliberately *absent* is the giveaway: no cron normalisation, +no `next_run_at` arithmetic, no re-arm rule, no ownership checks written out +by hand. Those moved into the aggregate, where they are covered by domain +tests that never construct an HTTP request. + +The error mapping is one table rather than a `raise HTTPException` per branch, +so a new domain error surfaces as a 500 that has to be classified, instead of +being silently swallowed by whichever `except` happened to be nearest. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from functools import wraps +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Query, Request + +from app.gateway.authz import require_permission +from app.gateway.deps import ScheduleServiceDep, get_optional_user_from_request +from app.gateway.routers.schedule.models import ( + CreateScheduledTaskRequest, + DeleteResponse, + ScheduledRunResponse, + ScheduledTaskResponse, + TriggerResponse, + UpdateScheduledTaskRequest, +) +from deerflow.domain.schedule.commands import DeleteTask, PauseTask, ResumeTask, TriggerTask +from deerflow.domain.schedule.exceptions import ( + InvalidContextModeError, + InvalidScheduleError, + ScheduleError, + TaskNotFoundError, + TaskNotMutableError, + ThreadNotFoundError, +) +from deerflow.domain.schedule.model import DispatchOutcome + +router = APIRouter(prefix="/api", tags=["scheduled-tasks"]) + +# One family of domain errors, one place they become a protocol. +_STATUS_BY_ERROR: dict[type[ScheduleError], int] = { + # "Not yours" and "does not exist" are both 404 by design -- the service + # already refuses to distinguish them, and so must the status code. + TaskNotFoundError: 404, + ThreadNotFoundError: 404, + InvalidScheduleError: 422, + InvalidContextModeError: 422, + TaskNotMutableError: 409, +} + + +def _map_domain_errors(handler): + """Translate domain errors raised by the service into HTTP responses. + + An unclassified `ScheduleError` becomes a 500 on purpose: a new domain + error is a new protocol decision, and defaulting it to 4xx would let it + ship as a client error nobody chose. + """ + + @wraps(handler) + async def wrapper(*args, **kwargs): + try: + return await handler(*args, **kwargs) + except ScheduleError as exc: + status = _STATUS_BY_ERROR.get(type(exc)) + if status is None: + raise + raise HTTPException(status_code=status, detail=str(exc)) from exc + + return wrapper + + +async def _require_user_id(request: Request) -> str: + user = await get_optional_user_from_request(request) + if user is None: + raise HTTPException(status_code=401, detail="Authentication required") + return str(user.id) + + +@router.get("/scheduled-tasks", response_model=list[ScheduledTaskResponse]) +@require_permission("threads", "read") +@_map_domain_errors +async def list_scheduled_tasks(request: Request, service: ScheduleServiceDep): + user = await get_optional_user_from_request(request) + if user is None: + # Not 401: an unauthenticated listing has always been an empty one. + return [] + return [ScheduledTaskResponse.from_domain(task) for task in await service.list_tasks(str(user.id))] + + +@router.post("/scheduled-tasks", response_model=ScheduledTaskResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def create_scheduled_task(request: Request, body: CreateScheduledTaskRequest, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + task = await service.create_scheduled_task(body.to_command(user_id), now=datetime.now(UTC)) + return ScheduledTaskResponse.from_domain(task) + + +@router.get("/scheduled-tasks/{task_id}", response_model=ScheduledTaskResponse) +@require_permission("threads", "read") +@_map_domain_errors +async def get_scheduled_task(task_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + return ScheduledTaskResponse.from_domain(await service.get_task(task_id, user_id=user_id)) + + +@router.patch("/scheduled-tasks/{task_id}", response_model=ScheduledTaskResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def update_scheduled_task( + task_id: str, + request: Request, + body: UpdateScheduledTaskRequest, + service: ScheduleServiceDep, +): + user_id = await _require_user_id(request) + # A schedule or context change needs the parts the client omitted, so the + # current task is read once and handed to `to_command`. That one extra + # fetch is what lets the command carry whole value objects instead of a + # patch of loose fields. + current = await service.get_task(task_id, user_id=user_id) if body.changes_schedule() or body.changes_context() else None + task = await service.update_scheduled_task(body.to_command(task_id, user_id, current), now=datetime.now(UTC)) + return ScheduledTaskResponse.from_domain(task) + + +@router.post("/scheduled-tasks/{task_id}/pause", response_model=ScheduledTaskResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def pause_scheduled_task(task_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + # No body, so no request model: the command is built right here (spec §2.1 ①). + return ScheduledTaskResponse.from_domain(await service.pause_task(PauseTask(task_id=task_id, user_id=user_id))) + + +@router.post("/scheduled-tasks/{task_id}/resume", response_model=ScheduledTaskResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def resume_scheduled_task(task_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + return ScheduledTaskResponse.from_domain(await service.resume_task(ResumeTask(task_id=task_id, user_id=user_id))) + + +@router.post("/scheduled-tasks/{task_id}/trigger", response_model=TriggerResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def trigger_scheduled_task(task_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + result = await service.trigger_task(TriggerTask(task_id=task_id, user_id=user_id), now=datetime.now(UTC)) + if result.outcome is DispatchOutcome.CONFLICT: + raise HTTPException(status_code=409, detail=result.error or "Scheduled task trigger conflicted with an active run") + if result.outcome is DispatchOutcome.FAILED: + # 502, not 500: the failure is downstream of this API -- the run could + # not be started -- and the task itself is intact. + raise HTTPException(status_code=502, detail=result.error or "Scheduled task trigger failed") + return TriggerResponse(id=task_id, triggered=True) + + +@router.delete("/scheduled-tasks/{task_id}", response_model=DeleteResponse) +@require_permission("threads", "write") +@_map_domain_errors +async def delete_scheduled_task(task_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + await service.delete_task(DeleteTask(task_id=task_id, user_id=user_id)) + return DeleteResponse(id=task_id, deleted=True) + + +@router.get("/scheduled-tasks/{task_id}/runs", response_model=list[ScheduledRunResponse]) +@require_permission("threads", "read") +@_map_domain_errors +async def list_scheduled_task_runs( + task_id: str, + request: Request, + service: ScheduleServiceDep, + limit: Annotated[int, Query(ge=1, le=200)] = 50, + offset: Annotated[int, Query(ge=0)] = 0, +): + user_id = await _require_user_id(request) + runs = await service.list_task_runs(task_id, user_id=user_id, limit=limit, offset=offset) + return [ScheduledRunResponse.from_domain(run) for run in runs] + + +@router.get("/threads/{thread_id}/scheduled-tasks", response_model=list[ScheduledTaskResponse]) +@require_permission("threads", "read", owner_check=True) +@_map_domain_errors +async def list_thread_scheduled_tasks(thread_id: str, request: Request, service: ScheduleServiceDep): + user_id = await _require_user_id(request) + tasks = await service.list_tasks_by_thread(user_id, thread_id) + return [ScheduledTaskResponse.from_domain(task) for task in tasks] diff --git a/backend/app/gateway/routers/scheduled_tasks.py b/backend/app/gateway/routers/scheduled_tasks.py deleted file mode 100644 index 35cd74bf1..000000000 --- a/backend/app/gateway/routers/scheduled_tasks.py +++ /dev/null @@ -1,310 +0,0 @@ -from __future__ import annotations - -import uuid -from datetime import UTC, datetime -from typing import Any - -from fastapi import APIRouter, HTTPException, Query, Request -from pydantic import BaseModel, Field - -from app.gateway.authz import require_permission -from app.gateway.deps import ( - get_config, - get_optional_user_from_request, - get_scheduled_task_repo, - get_scheduled_task_run_repo, - get_scheduled_task_service, - get_thread_store, -) -from deerflow.scheduler.schedules import ( - next_run_at as compute_next_run_at, -) -from deerflow.scheduler.schedules import ( - normalize_cron_expression, - validate_timezone, -) - -router = APIRouter(prefix="/api", tags=["scheduled-tasks"]) - - -def _ensure_task_mutable(task: dict[str, Any]) -> None: - if task.get("status") == "running": - raise HTTPException( - status_code=409, - detail="Scheduled task is currently running; retry after the active execution finishes", - ) - - -class ScheduledTaskCreateRequest(BaseModel): - thread_id: str | None = None - context_mode: str = "fresh_thread_per_run" - title: str = Field(min_length=1) - prompt: str = Field(min_length=1) - schedule_type: str - schedule_spec: dict[str, Any] - timezone: str - - -class ScheduledTaskUpdateRequest(BaseModel): - context_mode: str | None = None - thread_id: str | None = None - title: str | None = Field(default=None, min_length=1) - prompt: str | None = Field(default=None, min_length=1) - schedule_spec: dict[str, Any] | None = None - timezone: str | None = None - - -@router.get("/scheduled-tasks") -@require_permission("threads", "read") -async def list_scheduled_tasks(request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - return [] - return await repo.list_by_user(str(user.id)) - - -@router.post("/scheduled-tasks") -@require_permission("threads", "write") -async def create_scheduled_task(request: Request, body: ScheduledTaskCreateRequest): - config = get_config() - repo = get_scheduled_task_repo(request) - thread_store = get_thread_store(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - if body.context_mode not in {"fresh_thread_per_run", "reuse_thread"}: - raise HTTPException(status_code=422, detail="Unsupported context_mode") - if body.context_mode == "reuse_thread": - if not body.thread_id: - raise HTTPException(status_code=422, detail="reuse_thread requires thread_id") - if not await thread_store.check_access(body.thread_id, str(user.id), require_existing=True): - raise HTTPException(status_code=404, detail="Thread not found") - if body.schedule_type not in {"once", "cron"}: - raise HTTPException(status_code=422, detail="Unsupported schedule_type") - - schedule_spec = dict(body.schedule_spec) - try: - validate_timezone(body.timezone) - if body.schedule_type == "cron": - raw_cron = schedule_spec.get("cron") - if not isinstance(raw_cron, str): - raise HTTPException(status_code=422, detail="cron schedule requires schedule_spec.cron") - schedule_spec["cron"] = normalize_cron_expression(raw_cron) - next_run_at = compute_next_run_at( - body.schedule_type, - schedule_spec, - body.timezone, - now=datetime.now(UTC), - ) - except ValueError as exc: - raise HTTPException(status_code=422, detail=str(exc)) from exc - - if body.schedule_type == "once" and next_run_at is None: - raise HTTPException(status_code=422, detail="once schedule must be in the future") - if body.schedule_type == "once" and next_run_at is not None and (next_run_at - datetime.now(UTC)).total_seconds() < config.scheduler.min_once_delay_seconds: - raise HTTPException( - status_code=422, - detail=(f"once schedule must be at least {config.scheduler.min_once_delay_seconds} seconds in the future"), - ) - - return await repo.create( - task_id=f"task-{uuid.uuid4().hex}", - user_id=str(user.id), - thread_id=body.thread_id, - context_mode=body.context_mode, - assistant_id="lead_agent", - title=body.title, - prompt=body.prompt, - schedule_type=body.schedule_type, - schedule_spec=schedule_spec, - timezone=body.timezone, - next_run_at=next_run_at, - ) - - -@router.get("/scheduled-tasks/{task_id}") -@require_permission("threads", "read") -async def get_scheduled_task(task_id: str, request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - task = await repo.get(task_id, user_id=str(user.id)) - if task is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - return task - - -@router.patch("/scheduled-tasks/{task_id}") -@require_permission("threads", "write") -async def update_scheduled_task(task_id: str, request: Request, body: ScheduledTaskUpdateRequest): - config = get_config() - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - existing = await repo.get(task_id, user_id=str(user.id)) - if existing is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - _ensure_task_mutable(existing) - - updates = body.model_dump(exclude_none=True) - if "context_mode" in updates: - if updates["context_mode"] not in {"fresh_thread_per_run", "reuse_thread"}: - raise HTTPException(status_code=422, detail="Unsupported context_mode") - effective_context_mode = str(updates.get("context_mode", existing["context_mode"])) - effective_thread_id = updates.get("thread_id", existing.get("thread_id")) - if effective_context_mode == "reuse_thread": - if not effective_thread_id: - raise HTTPException(status_code=422, detail="reuse_thread requires thread_id") - thread_store = get_thread_store(request) - if not await thread_store.check_access(str(effective_thread_id), str(user.id), require_existing=True): - raise HTTPException(status_code=404, detail="Thread not found") - elif effective_context_mode == "fresh_thread_per_run": - updates["thread_id"] = None - if "timezone" in updates: - try: - validate_timezone(str(updates["timezone"])) - except ValueError as exc: - raise HTTPException(status_code=422, detail=str(exc)) from exc - - if "schedule_spec" in updates or "timezone" in updates: - schedule_spec = dict(existing["schedule_spec"]) - if "schedule_spec" in updates and isinstance(updates["schedule_spec"], dict): - schedule_spec = dict(updates["schedule_spec"]) - timezone = str(updates.get("timezone", existing["timezone"])) - try: - if existing["schedule_type"] == "cron": - raw_cron = schedule_spec.get("cron") - if not isinstance(raw_cron, str): - raise HTTPException( - status_code=422, - detail="cron schedule requires schedule_spec.cron", - ) - schedule_spec["cron"] = normalize_cron_expression(raw_cron) - next_run_at = compute_next_run_at( - existing["schedule_type"], - schedule_spec, - timezone, - now=datetime.now(UTC), - ) - except ValueError as exc: - raise HTTPException(status_code=422, detail=str(exc)) from exc - if existing["schedule_type"] == "once" and next_run_at is None: - raise HTTPException(status_code=422, detail="once schedule must be in the future") - if existing["schedule_type"] == "once" and next_run_at is not None and (next_run_at - datetime.now(UTC)).total_seconds() < config.scheduler.min_once_delay_seconds: - raise HTTPException( - status_code=422, - detail=(f"once schedule must be at least {config.scheduler.min_once_delay_seconds} seconds in the future"), - ) - updates["schedule_spec"] = schedule_spec - updates["next_run_at"] = next_run_at - # A terminal task (completed/failed/cancelled) whose schedule was just - # pushed into the future must be re-armed: claim_due_tasks only admits - # "enabled" rows, so leaving the terminal status would return 200 with - # a next_run_at that silently never fires. - if next_run_at is not None and existing["status"] in {"completed", "failed", "cancelled"}: - updates["status"] = "enabled" - - updated = await repo.update( - task_id, - user_id=str(user.id), - updates=updates, - ) - return updated - - -@router.post("/scheduled-tasks/{task_id}/pause") -@require_permission("threads", "write") -async def pause_scheduled_task(task_id: str, request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - existing = await repo.get(task_id, user_id=str(user.id)) - if existing is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - _ensure_task_mutable(existing) - updated = await repo.update(task_id, user_id=str(user.id), updates={"status": "paused"}) - if updated is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - return updated - - -@router.post("/scheduled-tasks/{task_id}/resume") -@require_permission("threads", "write") -async def resume_scheduled_task(task_id: str, request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - existing = await repo.get(task_id, user_id=str(user.id)) - if existing is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - _ensure_task_mutable(existing) - updated = await repo.update(task_id, user_id=str(user.id), updates={"status": "enabled"}) - if updated is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - return updated - - -@router.post("/scheduled-tasks/{task_id}/trigger") -@require_permission("threads", "write") -async def trigger_scheduled_task(task_id: str, request: Request): - repo = get_scheduled_task_repo(request) - service = get_scheduled_task_service(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - task = await repo.get(task_id, user_id=str(user.id)) - if task is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - result = await service.dispatch_task(task, now=datetime.now(UTC), trigger="manual") - if result["outcome"] == "conflict": - raise HTTPException(status_code=409, detail=result["error"] or "Scheduled task trigger conflicted with an active run") - if result["outcome"] == "failed": - raise HTTPException(status_code=502, detail=result["error"] or "Scheduled task trigger failed") - return {"id": task_id, "triggered": True} - - -@router.delete("/scheduled-tasks/{task_id}") -@require_permission("threads", "write") -async def delete_scheduled_task(task_id: str, request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - deleted = await repo.delete(task_id, user_id=str(user.id)) - if not deleted: - raise HTTPException(status_code=404, detail="Scheduled task not found") - return {"id": task_id, "deleted": deleted} - - -@router.get("/scheduled-tasks/{task_id}/runs") -@require_permission("threads", "read") -async def list_scheduled_task_runs( - task_id: str, - request: Request, - limit: int = Query(default=50, ge=1, le=200), - offset: int = Query(default=0, ge=0), -): - task_repo = get_scheduled_task_repo(request) - run_repo = get_scheduled_task_run_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - task = await task_repo.get(task_id, user_id=str(user.id)) - if task is None: - raise HTTPException(status_code=404, detail="Scheduled task not found") - return await run_repo.list_by_task(task_id, limit=limit, offset=offset) - - -@router.get("/threads/{thread_id}/scheduled-tasks") -@require_permission("threads", "read", owner_check=True) -async def list_thread_scheduled_tasks(thread_id: str, request: Request): - repo = get_scheduled_task_repo(request) - user = await get_optional_user_from_request(request) - if user is None: - raise HTTPException(status_code=401, detail="Authentication required") - return await repo.list_by_user_and_thread(str(user.id), thread_id) diff --git a/backend/app/scheduler/__init__.py b/backend/app/scheduler/__init__.py index a3c331940..2bec3007a 100644 --- a/backend/app/scheduler/__init__.py +++ b/backend/app/scheduler/__init__.py @@ -1,3 +1 @@ -from .service import ScheduledTaskService - -__all__ = ["ScheduledTaskService"] +"""Clock-driven entry point of the schedule context (the poller).""" diff --git a/backend/app/scheduler/poller.py b/backend/app/scheduler/poller.py new file mode 100644 index 000000000..0a1998029 --- /dev/null +++ b/backend/app/scheduler/poller.py @@ -0,0 +1,84 @@ +"""Driving adapter -- the clock that asks the schedule service to work. + +This is the only part of the scheduler that knows about time passing. It owns +*when* `ScheduleService.run_once` is called and what happens when a poll fails; +it owns nothing about what a poll means. Everything the old +`app/scheduler/service.py` mixed into its loop -- overlap policy, lease +semantics, budget accounting -- now lives in the domain and reaches this file +only as one awaited call. + +Two behaviours here are load-bearing: + + - **A failing poll must not end the loop.** A transient error (SQLite's + "database is locked" is the realistic one) would otherwise stop every + scheduled task for the rest of the process life, silently. + - **Startup reconciliation must not block startup.** The service lets + reconcile failures propagate on purpose -- whether they are fatal is the + caller's policy -- and this caller's policy is to log and keep scheduling. + A gateway refusing to start over leftover rows is worse than one running + with them. +""" + +from __future__ import annotations + +import asyncio +import logging +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from deerflow.domain.schedule.service import ScheduleService + +logger = logging.getLogger(__name__) + +RESTART_ERROR = "interrupted: gateway restarted before the run reached a terminal state" + + +class SchedulePoller: + """Runs `ScheduleService.run_once` on an interval until stopped.""" + + def __init__(self, service: ScheduleService, *, poll_interval_seconds: float) -> None: + self._service = service + self._poll_interval_seconds = poll_interval_seconds + self._task: asyncio.Task | None = None + self._stop = asyncio.Event() + + async def start(self) -> None: + """Reconcile what a crash left behind, then begin polling. + + Idempotent: a second call while running is a no-op, so a caller cannot + end up with two loops claiming the same tasks. + """ + if self._task is not None: + return + try: + stale_runs, stuck_tasks = await self._service.reconcile_on_startup(error=RESTART_ERROR) + if stale_runs: + logger.warning("Marked %d stale scheduled task run(s) as interrupted after restart", stale_runs) + if stuck_tasks: + logger.warning("Cancelled %d stuck once task(s) after restart", stuck_tasks) + except Exception: + logger.exception("Failed to reconcile scheduled tasks at startup; scheduling anyway") + self._stop.clear() + self._task = asyncio.create_task(self._run_loop()) + + async def stop(self) -> None: + """Signal the loop and wait for the in-flight poll to finish.""" + if self._task is None: + return + self._stop.set() + await self._task + self._task = None + + async def _run_loop(self) -> None: + while not self._stop.is_set(): + try: + await self._service.run_once(now=datetime.now(UTC)) + except Exception: + logger.exception("Scheduled task poll failed; retrying next interval") + try: + # Waiting on the stop event rather than sleeping keeps shutdown + # prompt at a production-sized interval. + await asyncio.wait_for(self._stop.wait(), timeout=self._poll_interval_seconds) + except TimeoutError: + continue diff --git a/backend/app/scheduler/service.py b/backend/app/scheduler/service.py deleted file mode 100644 index 9bcfc268a..000000000 --- a/backend/app/scheduler/service.py +++ /dev/null @@ -1,496 +0,0 @@ -from __future__ import annotations - -import asyncio -import logging -import socket -import uuid -from datetime import UTC, datetime -from typing import Any, Literal - -from fastapi import HTTPException - -from deerflow.persistence.scheduled_task_runs import ActiveScheduledRunConflict -from deerflow.runtime import ConflictError, RunRecord -from deerflow.scheduler.schedules import next_run_at - -logger = logging.getLogger(__name__) - -# Shared so the has_active_runs fast path and the unique-index race path return -# byte-identical outcomes for the same "task already has an active run" condition. -_ACTIVE_RUN_CONFLICT_ERROR = "task already has an active run" -_SKIP_ACTIVE_RUN_ERROR = "skipped: a previous run of this task is still active" - - -class ScheduledTaskService: - def __init__( - self, - *, - task_repo, - task_run_repo, - launch_run, - poll_interval_seconds: int, - lease_seconds: int, - max_concurrent_runs: int, - ) -> None: - self._task_repo = task_repo - self._task_run_repo = task_run_repo - self._launch_run = launch_run - self._poll_interval_seconds = poll_interval_seconds - self._lease_seconds = lease_seconds - self._max_concurrent_runs = max_concurrent_runs - self._lease_owner = f"{socket.gethostname()}:{uuid.uuid4().hex}" - self._task: asyncio.Task | None = None - self._stop = asyncio.Event() - - async def run_once(self, *, now: datetime) -> None: - # ``max_concurrent_runs`` is a global cap on active scheduled runs, not - # just a per-poll claim batch: long runs accumulate across poll cycles, - # so each cycle only claims into the remaining budget. - active = await self._task_run_repo.count_active_runs() - budget = self._max_concurrent_runs - active - if budget <= 0: - return - claimed = await self._task_repo.claim_due_tasks( - now=now, - lease_owner=self._lease_owner, - lease_seconds=self._lease_seconds, - limit=budget, - ) - for task in claimed: - await self.dispatch_task(task, now=now, trigger="scheduled") - - @staticmethod - def _is_overlap_conflict(exc: Exception) -> bool: - if isinstance(exc, ConflictError): - return True - return isinstance(exc, HTTPException) and exc.status_code == 409 - - @staticmethod - def _task_status_for_failure(task: dict[str, Any], *, trigger: str) -> str: - if trigger == "manual": - # A failed manual trigger must not consume the task's scheduled - # future: a `once` task with run_at still ahead would otherwise be - # flipped to "failed" and never claimed again. - return task.get("status") or "enabled" - if task["schedule_type"] == "once": - return "failed" - return "enabled" - - @staticmethod - def _task_status_for_launch(task: dict[str, Any], *, trigger: str) -> str: - # The task-level status to write once _launch_run has produced a live - # run. A `once` task stays "running" until handle_run_completion - # observes the real terminal outcome; declaring "completed" at launch - # would stick if the run fails or the process dies (startup - # reconciliation is cancel_stuck_once_tasks). - if task["schedule_type"] == "once": - return "running" - if trigger == "manual" and task.get("status") == "paused": - return "paused" - return "enabled" - - @staticmethod - def _task_status_for_skip(task: dict[str, Any]) -> str: - if task["schedule_type"] == "once": - # The single occurrence was lost to an overlapping run; "completed" - # would claim an execution that never happened. - return "failed" - return "enabled" - - async def dispatch_task( - self, - task: dict[str, Any], - *, - now: datetime, - trigger: str, - ) -> dict[str, Any]: - execution_thread_id = task.get("thread_id") - if task.get("context_mode") == "fresh_thread_per_run" or not execution_thread_id: - execution_thread_id = str(uuid.uuid4()) - # "skip" must hold for fresh-thread runs too, where every run gets a new - # thread and the same-thread multitask ConflictError below can never - # fire. Checked before creating this dispatch's own run row so the row - # does not count itself as the active run. A manual trigger against an - # active run is rejected outright (409 at the router) instead of being - # recorded as a skipped occurrence — nothing was scheduled to happen. - # - # This has_active_runs check is a non-atomic fast path: it runs in its - # own session and is separated from the create() below by await points, - # so two concurrent dispatches (double-click / client retry / a manual - # trigger racing the poller) can both observe no active run. The DB is - # the atomic arbiter — the partial unique index uq_scheduled_task_run_active - # rejects the second active insert, surfaced as ActiveScheduledRunConflict - # and collapsed to the SAME outcome as this fast path just below. - overlap_skip = task.get("overlap_policy", "skip") == "skip" - if overlap_skip and await self._task_run_repo.has_active_runs(task["id"]): - if trigger == "manual": - return self._active_run_conflict_result(execution_thread_id) - return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger) - - task_run_id = f"task-run-{uuid.uuid4().hex}" - try: - await self._task_run_repo.create( - run_record_id=task_run_id, - task_id=task["id"], - thread_id=execution_thread_id, - scheduled_for=now, - trigger=trigger, - status="queued", - ) - except ActiveScheduledRunConflict: - # Lost the create race for the task's single active slot: a - # concurrent dispatch passed the same fast-path check and inserted - # its active row first. Identical outcome to the fast path above. - if trigger == "manual": - return self._active_run_conflict_result(execution_thread_id) - return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger) - # Track whether _launch_run has produced a live run. A bookkeeping - # failure AFTER launch (the queued->running write, or the parent task - # update) must NOT be recorded as "failed": "failed" is outside the - # partial unique index uq_scheduled_task_run_active, so it would release - # the task's single active slot and the next dispatch would launch a - # duplicate run. Once launch succeeds we keep the row "running" and - # retain the launched run_id regardless of bookkeeping errors. - launched_run_id: str | None = None - launched_thread_id: str | None = None - # Flip immediately after _launch_run returns, before any further code - # that can raise (e.g. result["run_id"] on a malformed result). The - # retention branch keys off this flag, not `launched_run_id is not - # None`, so a launch that succeeded but whose result-unpacking raised - # still takes the retention path instead of the release-the-slot path. - launch_succeeded = False - try: - result = await self._launch_run( - thread_id=execution_thread_id, - assistant_id=task.get("assistant_id"), - prompt=task["prompt"], - owner_user_id=task.get("user_id"), - metadata={ - "scheduled_task_id": task["id"], - "scheduled_task_run_id": task_run_id, - "scheduled_trigger": trigger, - }, - ) - launch_succeeded = True - launched_run_id = result["run_id"] - launched_thread_id = result["thread_id"] - next_at = next_run_at( - task["schedule_type"], - task["schedule_spec"], - task["timezone"], - now=now, - ) - task_status = self._task_status_for_launch(task, trigger=trigger) - await self._task_run_repo.update_status( - task_run_id, - status="running", - run_id=launched_run_id, - started_at=now, - # A fast-failing run can reach handle_run_completion before this - # write resumes; never clobber its terminal status. - protect_terminal=True, - ) - await self._task_repo.update_after_launch( - task["id"], - status=task_status, - next_run_at=next_at, - last_run_at=now, - last_run_id=launched_run_id, - last_thread_id=launched_thread_id, - last_error=None, - increment_run_count=True, - # Same race as the run-row write above: a fast-failing run's - # completion hook may have already finalized a `once` task. - protect_terminal=True, - ) - return { - "outcome": "launched", - "task_run_id": task_run_id, - "run_id": launched_run_id, - "thread_id": launched_thread_id, - "error": None, - } - except Exception as exc: - if not launch_succeeded and self._is_overlap_conflict(exc) and trigger == "scheduled" and task.get("overlap_policy", "skip") == "skip": - # Pre-launch overlap conflict (e.g. same-thread multitask): no - # run was started, so recording a skip and releasing the slot is - # safe. Guarded by ``not launch_succeeded`` because a run that - # already launched must never be reclassified as a skip / failed. - return await self._finalize_skip( - task, - task_run_id=task_run_id, - thread_id=execution_thread_id, - now=now, - error=str(exc), - ) - - next_at = next_run_at( - task["schedule_type"], - task["schedule_spec"], - task["timezone"], - now=now, - ) - - if launch_succeeded: - # _launch_run succeeded, so a run is live even though - # post-launch bookkeeping raised. Keep the task-run row - # "running" so it keeps holding the task's single active slot - # (preventing a duplicate launch on the next dispatch) and - # persist the run_id on the parent task for recovery / - # reconciliation / cancellation. These writes are best-effort: - # if the DB is still down the row stays "queued" -- still - # active, still holding the slot -- so we log and still report - # the run as launched so callers know a run is in flight. - task_status = self._task_status_for_launch(task, trigger=trigger) - try: - await self._task_run_repo.update_status( - task_run_id, - status="running", - run_id=launched_run_id, - started_at=now, - protect_terminal=True, - ) - except Exception: - logger.exception( - "Scheduled task-run %s: post-launch bookkeeping failed; run %s is still live (task %s)", - task_run_id, - launched_run_id, - task["id"], - ) - try: - await self._task_repo.update_after_launch( - task["id"], - status=task_status, - next_run_at=next_at, - last_run_at=now, - last_run_id=launched_run_id, - last_thread_id=launched_thread_id, - # The bookkeeping exception is an infrastructure-level - # transient, not a run-level failure: the run launched - # and is still in flight. Clear last_error like the - # success path so the task list does not show an error - # on a task whose run is actively running; the real - # terminal outcome is written by handle_run_completion. - # The transient itself is logged above. - last_error=None, - increment_run_count=True, - protect_terminal=True, - ) - except Exception: - logger.exception( - "Scheduled task %s: post-launch update failed; run %s is still live", - task["id"], - launched_run_id, - ) - return { - "outcome": "launched", - "task_run_id": task_run_id, - "run_id": launched_run_id, - "thread_id": launched_thread_id, - "error": str(exc), - } - - # _launch_run itself failed (or a step before it did): no live run - # was created, so it is safe to release the active slot. - task_status = self._task_status_for_failure(task, trigger=trigger) - await self._task_run_repo.update_status( - task_run_id, - status="failed", - error=str(exc), - started_at=now, - finished_at=now, - ) - await self._task_repo.update_after_launch( - task["id"], - status=task_status, - next_run_at=next_at, - last_run_at=now, - last_run_id=None, - last_thread_id=execution_thread_id, - last_error=str(exc), - increment_run_count=False, - ) - return { - "outcome": "conflict" if self._is_overlap_conflict(exc) else "failed", - "task_run_id": task_run_id, - "run_id": None, - "thread_id": execution_thread_id, - "error": str(exc), - } - - def _active_run_conflict_result(self, thread_id: str) -> dict[str, Any]: - """Manual-trigger response when the task already has an active run. - - Nothing was scheduled to happen, so no run-history row is recorded; the - router maps this to a 409. - """ - return { - "outcome": "conflict", - "task_run_id": None, - "run_id": None, - "thread_id": thread_id, - "error": _ACTIVE_RUN_CONFLICT_ERROR, - } - - async def _record_scheduled_skip( - self, - task: dict[str, Any], - *, - thread_id: str, - now: datetime, - trigger: str, - ) -> dict[str, Any]: - """Record a skipped occurrence for a scheduled dispatch that overlapped an active run. - - The tombstone is created directly as terminal ``"skipped"`` rather than - the transient ``"queued"`` the launch path uses: a queued row is active - and would itself trip ``uq_scheduled_task_run_active`` against the - pre-existing run that is still holding the task's single active slot. - ``"skipped"`` is outside the index predicate, so it never conflicts. - """ - task_run_id = f"task-run-{uuid.uuid4().hex}" - await self._task_run_repo.create( - run_record_id=task_run_id, - task_id=task["id"], - thread_id=thread_id, - scheduled_for=now, - trigger=trigger, - status="skipped", - ) - return await self._finalize_skip(task, task_run_id=task_run_id, thread_id=thread_id, now=now, error=_SKIP_ACTIVE_RUN_ERROR) - - async def _finalize_skip( - self, - task: dict[str, Any], - *, - task_run_id: str, - thread_id: str, - now: datetime, - error: str, - ) -> dict[str, Any]: - next_at = next_run_at( - task["schedule_type"], - task["schedule_spec"], - task["timezone"], - now=now, - ) - await self._task_run_repo.update_status( - task_run_id, - status="skipped", - error=error, - started_at=now, - finished_at=now, - ) - await self._task_repo.update_after_launch( - task["id"], - status=self._task_status_for_skip(task), - next_run_at=next_at, - last_run_at=task.get("last_run_at"), - last_run_id=task.get("last_run_id"), - last_thread_id=task.get("last_thread_id"), - last_error=error if task["schedule_type"] == "once" else None, - increment_run_count=False, - ) - return { - "outcome": "skipped", - "task_run_id": task_run_id, - "run_id": None, - "thread_id": thread_id, - "error": error, - } - - async def handle_run_completion(self, record: RunRecord) -> None: - metadata = record.metadata or {} - task_id = metadata.get("scheduled_task_id") - task_run_id = metadata.get("scheduled_task_run_id") - user_id = record.user_id - if not isinstance(task_id, str) or not isinstance(task_run_id, str) or not user_id: - return - - terminal_status: Literal["success", "failed", "interrupted"] | None - if record.status.value == "success": - terminal_status = "success" - error = None - elif record.status.value == "interrupted": - # Distinct from "failed": an interrupt (user cancel, same-thread - # takeover) carries no error and is not an execution failure. - terminal_status = "interrupted" - error = record.error or "run was interrupted before completion" - elif record.status.value in {"error", "timeout"}: - terminal_status = "failed" - error = record.error - else: - terminal_status = None - error = record.error - if terminal_status is None: - return - - await self._task_run_repo.update_status( - task_run_id, - status=terminal_status, - run_id=record.run_id, - error=error, - finished_at=datetime.now(UTC), - ) - - task = await self._task_repo.get(task_id, user_id=user_id) - if task is None: - return - - updates: dict[str, Any] = {"last_error": error} - if task["schedule_type"] == "once": - # The single occurrence is consumed either way (the run did launch, - # so re-arming risks duplicate side effects), but an interrupt ends - # as "cancelled", not "failed". - if terminal_status == "success": - updates["status"] = "completed" - elif terminal_status == "interrupted": - updates["status"] = "cancelled" - else: - updates["status"] = "failed" - await self._task_repo.update(task_id, user_id=user_id, updates=updates) - - async def start(self) -> None: - if self._task is not None: - return - restart_error = "interrupted: gateway restarted before the run reached a terminal state" - try: - stale = await self._task_run_repo.mark_stale_active_runs(error=restart_error) - if stale: - logger.warning("Marked %d stale scheduled task run(s) as interrupted after restart", stale) - except Exception: - logger.exception("Failed to sweep stale scheduled task runs at startup") - try: - # The run rows above are only half the story: a launched `once` - # task is parked in "running" until the (now dead) completion hook - # would have finalized it, so reconcile the parent rows too. - stuck = await self._task_repo.cancel_stuck_once_tasks(error=restart_error) - if stuck: - logger.warning("Cancelled %d stuck once task(s) after restart", stuck) - except Exception: - logger.exception("Failed to reconcile stuck once tasks at startup") - self._stop.clear() - self._task = asyncio.create_task(self._run_loop()) - - async def stop(self) -> None: - if self._task is None: - return - self._stop.set() - await self._task - self._task = None - - async def _run_loop(self) -> None: - while not self._stop.is_set(): - try: - await self.run_once(now=datetime.now(UTC)) - except Exception: - # A transient DB error (e.g. SQLite "database is locked") must - # not kill the poller task for the rest of the process life. - logger.exception("Scheduled task poll failed; retrying next interval") - try: - await asyncio.wait_for( - self._stop.wait(), - timeout=self._poll_interval_seconds, - ) - except TimeoutError: - continue diff --git a/backend/docs/GITHUB_AGENTS.md b/backend/docs/GITHUB_AGENTS.md index d46004cec..f32977048 100644 --- a/backend/docs/GITHUB_AGENTS.md +++ b/backend/docs/GITHUB_AGENTS.md @@ -259,7 +259,7 @@ Key properties: - **Drain-conflict edge case**: if the drain's own `runs.create()` also hits `ConflictError` — e.g. a manual Web UI turn or a scheduled run raced onto the same thread — the popped batch is requeued (not lost, not retried in a tight loop). It is picked up again the next time this manager successfully creates *and watches* a run on that thread, which is guaranteed to eventually happen once whatever is occupying the thread finishes and any subsequent trigger succeeds. - **Reactions are out of scope for this slice.** GitHub's reaction API (`eyes`/`confused` acknowledgment) has no existing integration in this codebase and needs its own design pass; buffered comments are coalesced silently, with no per-comment acknowledgment. - **Config**: gated behind `ChannelRunPolicy.buffer_followups_on_busy` (default `False`); GitHub's own registration in `app/gateway/github/run_policy.py` opts in, since it is exactly the fire_and_forget + log-only-send channel this was designed for. Any other channel that adopts `fire_and_forget=True` in the future keeps the old silent-drop-with-log behavior unless it opts in too. -- **Plumbing**: the watcher subscribes to the Gateway's `StreamBridge` singleton (`app.state.stream_bridge`), which `ChannelManager` previously had no way to reach — every other consumer gets it via `get_stream_bridge(request)`, which needs an HTTP `Request` the bus-consumer loop doesn't have. It is threaded as a zero-arg closure from `app.py`'s lifespan (`get_stream_bridge=lambda: getattr(app.state, "stream_bridge", None)`) through `start_channel_service()` → `ChannelService.__init__` → `ChannelManager.__init__`, mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` closure already used for `ScheduledTaskService` in the same lifespan function. +- **Plumbing**: the watcher subscribes to the Gateway's `StreamBridge` singleton (`app.state.stream_bridge`), which `ChannelManager` previously had no way to reach — every other consumer gets it via `get_stream_bridge(request)`, which needs an HTTP `Request` the bus-consumer loop doesn't have. It is threaded as a zero-arg closure from `app.py`'s lifespan (`get_stream_bridge=lambda: getattr(app.state, "stream_bridge", None)`) through `start_channel_service()` → `ChannelService.__init__` → `ChannelManager.__init__`, mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` closure already used for the schedule composition root in the same lifespan function. - **Single-process scope (known limitation)**: the buffer and watcher tasks live in one `ChannelManager` instance's process memory. Under `GATEWAY_WORKERS>1` or a multi-pod deployment, a follow-up comment that a load balancer or webhook fan-out routes to a *different* worker than the one that created the busy run will not see that worker's buffer. This mirrors the cross-pod gap already documented for issue #4120 (IM leader election or a shared buffer store would close it) and is deliberately deferred — single-process/single-pod deployments, the safe default, are unaffected. ## Outbound is Log-only diff --git a/backend/packages/harness/deerflow/config/reload_boundary.py b/backend/packages/harness/deerflow/config/reload_boundary.py index 6f6f87d62..52e4a04f6 100644 --- a/backend/packages/harness/deerflow/config/reload_boundary.py +++ b/backend/packages/harness/deerflow/config/reload_boundary.py @@ -66,8 +66,8 @@ STARTUP_ONLY_FIELDS: dict[str, str] = { "start_channel_service() wires the connection repository and channel workers once at startup, and the channel-connections router caches the merged provider config on app.state; channel_connections.* edits need a restart." ), "scheduler": ( - "ScheduledTaskService is constructed and started once during Gateway lifespan startup; enabled, poll_interval_seconds, lease_seconds, " - "and max_concurrent_runs are captured into the service instance and the background poller task is not rebuilt on config.yaml edits." + "ScheduleService and SchedulePoller are constructed and started once during Gateway lifespan startup; enabled, poll_interval_seconds, lease_seconds, " + "and max_concurrent_runs are captured into the assembled service/poller and the background poller task is not rebuilt on config.yaml edits." ), "run_ownership": ( "RunOwnershipConfig is captured once into RunManager at langgraph_runtime() startup; the lease heartbeat background task is created and " diff --git a/backend/packages/harness/deerflow/domain/schedule/__init__.py b/backend/packages/harness/deerflow/domain/schedule/__init__.py new file mode 100644 index 000000000..f673e972e --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/__init__.py @@ -0,0 +1,83 @@ +"""Schedule bounded context: standing instructions to run a prompt on time. + +Public API of the context. Import domain objects, commands, errors, and the +service from here; import ports from `deerflow.domain.schedule.ports` -- they +are contracts consumed by adapters and tests, not everyday call-site symbols. +""" + +from deerflow.domain.schedule.commands import ( + UNSET, + ContextChange, + CreateScheduledTask, + DeleteTask, + PauseTask, + ResumeTask, + TriggerTask, + UnsetType, + UpdateScheduledTask, +) +from deerflow.domain.schedule.exceptions import ( + ActiveRunConflictError, + CorruptStoredScheduleError, + InvalidContextModeError, + InvalidScheduleError, + LaunchFailedError, + ScheduleError, + TaskNotFoundError, + TaskNotMutableError, + ThreadBusyError, + ThreadNotFoundError, +) +from deerflow.domain.schedule.model import ( + ACTIVE_RUN_STATUSES, + TERMINAL_RUN_STATUSES, + TERMINAL_TASK_STATUSES, + ContextMode, + DispatchOutcome, + RunStatus, + ScheduledRun, + ScheduledTask, + SchedulePolicy, + ScheduleSpec, + ScheduleType, + TaskStatus, + TriggerKind, +) +from deerflow.domain.schedule.service import DispatchResult, ScheduleService + +__all__ = [ + "ACTIVE_RUN_STATUSES", + "TERMINAL_RUN_STATUSES", + "TERMINAL_TASK_STATUSES", + "UNSET", + "ActiveRunConflictError", + "ContextChange", + "ContextMode", + "CorruptStoredScheduleError", + "CreateScheduledTask", + "DeleteTask", + "DispatchOutcome", + "DispatchResult", + "InvalidContextModeError", + "InvalidScheduleError", + "LaunchFailedError", + "PauseTask", + "ResumeTask", + "RunStatus", + "ScheduleError", + "SchedulePolicy", + "ScheduleService", + "ScheduleSpec", + "ScheduleType", + "ScheduledRun", + "ScheduledTask", + "TaskNotFoundError", + "TaskNotMutableError", + "TaskStatus", + "ThreadBusyError", + "ThreadNotFoundError", + "TriggerKind", + "TriggerTask", + "UnsetType", + "UpdateScheduledTask", +] diff --git a/backend/packages/harness/deerflow/domain/schedule/commands.py b/backend/packages/harness/deerflow/domain/schedule/commands.py new file mode 100644 index 000000000..75cf4f09b --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/commands.py @@ -0,0 +1,130 @@ +"""Commands of the schedule context. + +One frozen dataclass per HTTP-driven state-changing use case -- the named +carrier of "the information required to perform an operation on the domain" +(AWS hexagonal guidance). Commands are dumb data on purpose: business +validation stays on the aggregate (``ScheduledTask``'s factory and +transitions), and structural validation stays on the primary adapter's api +model, so error attribution (a malformed schedule reported before an unknown +thread) is owned by the handler's construction order. + +Two groups of use cases are deliberately NOT commands: + +- **Queries** (``list_tasks``, ``get_task``, ``list_task_runs``, ...) keep + plain parameters -- a command expresses an intent to change state, and + wrapping reads would be pure boilerplate. +- **Clock- and callback-driven writes** (``run_once``, ``dispatch_task``, + ``handle_run_completion``, ``reconcile_on_startup``). Those drivers have no + wire shape to translate (spec §5.3): the poller hands the service a claimed + aggregate and a clock reading, and the completion hook hands it an already + domain-typed ``RunOutcome``. A command would re-wrap domain vocabulary in + more domain vocabulary. + +``now`` is not a command field: it is the server's clock reading, passed +explicitly to the handler (``now=``) like every other rule input, not part of +the client's intent. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, Final, final + +if TYPE_CHECKING: + from deerflow.domain.schedule.model import ContextMode, ScheduleSpec + + +@final +class UnsetType: + """The type of ``UNSET`` -- "the client did not supply this field". + + Partial updates need three states (absent, null, value). ``None`` cannot + carry two of them, so absence gets its own singleton; a field that is + ``UNSET`` is left untouched by the handler. + """ + + _instance: UnsetType | None = None + + def __new__(cls) -> UnsetType: + if cls._instance is None: + cls._instance = super().__new__(cls) + return cls._instance + + def __repr__(self) -> str: + return "UNSET" + + def __bool__(self) -> bool: + return False + + +UNSET: Final[UnsetType] = UnsetType() + + +@dataclass(frozen=True) +class ContextChange: + """A requested change of execution context. + + The mode and the thread always move together -- ``with_context`` takes + both, and clearing the thread is what switching to a fresh-thread mode + means. Packaging them keeps ``None`` unambiguous everywhere else: the one + field for which ``None`` is a meaningful value travels inside this object. + """ + + context_mode: str | ContextMode + thread_id: str | None = None + + +@dataclass(frozen=True) +class CreateScheduledTask: + """Register a new standing instruction to run a prompt on time.""" + + user_id: str + title: str + prompt: str + schedule: ScheduleSpec + context_mode: str | ContextMode + thread_id: str | None = None + + +@dataclass(frozen=True) +class UpdateScheduledTask: + """Partially update a task; ``UNSET`` means "not supplied".""" + + task_id: str + user_id: str + title: str | UnsetType = UNSET + prompt: str | UnsetType = UNSET + schedule: ScheduleSpec | UnsetType = UNSET + context: ContextChange | UnsetType = UNSET + + +@dataclass(frozen=True) +class PauseTask: + """Stop claiming this task until it is resumed.""" + + task_id: str + user_id: str + + +@dataclass(frozen=True) +class ResumeTask: + """Re-admit this task to claiming.""" + + task_id: str + user_id: str + + +@dataclass(frozen=True) +class DeleteTask: + """Remove the task; its execution history rows go with it.""" + + task_id: str + user_id: str + + +@dataclass(frozen=True) +class TriggerTask: + """Dispatch a task on demand, even while it is paused.""" + + task_id: str + user_id: str diff --git a/backend/packages/harness/deerflow/domain/schedule/exceptions.py b/backend/packages/harness/deerflow/domain/schedule/exceptions.py new file mode 100644 index 000000000..d07c3eb36 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/exceptions.py @@ -0,0 +1,67 @@ +"""The known errors of the schedule context. + +One family under one base class, so the primary adapter can map the whole +family onto protocol codes in a single table. Class names keep the PEP 8 +``Error`` suffix; the module is named ``exceptions`` after the AWS +hexagonal guidance's domain folder of the same name. +""" + +from __future__ import annotations + + +class ScheduleError(Exception): + """Base error for the schedule domain.""" + + +class InvalidScheduleError(ScheduleError): + """Timezone, cron expression, or run_at is not usable.""" + + +class InvalidContextModeError(ScheduleError): + """context_mode is unknown, or reuse_thread is missing its thread_id.""" + + +class TaskNotFoundError(ScheduleError): + """The task does not exist or does not belong to the user.""" + + +class TaskNotMutableError(ScheduleError): + """The task is currently running and cannot be edited.""" + + +class ThreadNotFoundError(ScheduleError): + """reuse_thread points at a thread the user cannot access.""" + + +class CorruptStoredScheduleError(ScheduleError): + """A stored task row can no longer be rebuilt into a valid aggregate. + + Raised by the persistence adapter, never by the aggregate: it means the + *storage* is damaged, not that a client submitted something invalid -- + which is why it is deliberately absent from the router's status table and + falls through to the unclassified-500 branch instead of riding + ``InvalidScheduleError``'s 422. + """ + + +class ActiveRunConflictError(ScheduleError): + """The task already holds its single active run slot. + + Raised by the run repository when the partial unique index + ``uq_scheduled_task_run_active`` rejects a second active row. Moved here + from ``persistence/scheduled_task_runs/sql.py`` (was + ``ActiveScheduledRunConflict``) so the domain owns its own vocabulary. + """ + + +class ThreadBusyError(ScheduleError): + """The execution thread already has an in-flight run. + + Translated by the RunLauncher adapter from ConflictError / HTTP 409. + This is what removes `from fastapi import HTTPException` from the + orchestration layer. + """ + + +class LaunchFailedError(ScheduleError): + """The run could not be launched for any non-conflict reason.""" diff --git a/backend/packages/harness/deerflow/domain/schedule/model/__init__.py b/backend/packages/harness/deerflow/domain/schedule/model/__init__.py new file mode 100644 index 000000000..c8f3e2d44 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/model/__init__.py @@ -0,0 +1,51 @@ +"""Domain model of the schedule context. + +A package rather than a single module because this context has two +aggregates plus a value object, which a single `model.py` could not hold +without becoming the file nobody reads. Re-exports below make the split +invisible to callers: `deerflow.domain.schedule.model` exposes exactly +what a single `model.py` would have. + +The re-exports below are ordered alphabetically because ruff's isort rule +owns that block; alphabetical happens to satisfy the real dependency order +too (enums depend on nothing, run depends on enums, spec depends on enums, +task depends on both), so it needs no override. The domain errors are not +re-exported here: they live one level up in +`deerflow.domain.schedule.exceptions`, a sibling of the model per the AWS +domain layout, and are imported from there. + +What actually keeps the package acyclic is a rule isort cannot express: +**a submodule imports its siblings directly, never this package.** Reaching +back through `deerflow.domain.schedule.model` makes a submodule depend on a +partially initialized package — it only works while the symbol happens to +sit above that submodule's own line here, and reordering this block silently +breaks it. Keep it that way. +""" + +from deerflow.domain.schedule.model.enums import ( + ContextMode, + DispatchOutcome, + RunStatus, + ScheduleType, + TaskStatus, + TriggerKind, +) +from deerflow.domain.schedule.model.run import ACTIVE_RUN_STATUSES, TERMINAL_RUN_STATUSES, ScheduledRun +from deerflow.domain.schedule.model.spec import SchedulePolicy, ScheduleSpec +from deerflow.domain.schedule.model.task import TERMINAL_TASK_STATUSES, ScheduledTask + +__all__ = [ + "ACTIVE_RUN_STATUSES", + "TERMINAL_RUN_STATUSES", + "TERMINAL_TASK_STATUSES", + "ContextMode", + "DispatchOutcome", + "RunStatus", + "SchedulePolicy", + "ScheduleSpec", + "ScheduleType", + "ScheduledRun", + "ScheduledTask", + "TaskStatus", + "TriggerKind", +] diff --git a/backend/packages/harness/deerflow/domain/schedule/model/enums.py b/backend/packages/harness/deerflow/domain/schedule/model/enums.py new file mode 100644 index 000000000..d54846fd2 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/model/enums.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +from enum import StrEnum + + +class TaskStatus(StrEnum): + """Lifecycle of a standing instruction. + + RUNNING does not mean "the agent is executing" -- it means this dispatch + round's scheduling ownership is held. That is why `ensure_mutable` refuses + edits in that state, and why a crash can strand a task there. + + The last three are terminal for the *current* schedule, not forever: moving + the schedule into the future re-arms them (see TERMINAL_TASK_STATUSES). + """ + + ENABLED = "enabled" + PAUSED = "paused" + RUNNING = "running" + COMPLETED = "completed" + FAILED = "failed" + CANCELLED = "cancelled" + + +class ContextMode(StrEnum): + """Which thread a dispatch executes in. + + REUSE_THREAD lets consecutive executions see each other's history and + requires a thread up front; the default gives each one a fresh thread and + therefore no shared context. + """ + + FRESH_THREAD_PER_RUN = "fresh_thread_per_run" + REUSE_THREAD = "reuse_thread" + + +class ScheduleType(StrEnum): + """How the next fire time is derived. + + The two branch almost every rule in this context: CRON always has a next + occurrence, ONCE is consumed by its first one. + """ + + ONCE = "once" + CRON = "cron" + + +class RunStatus(StrEnum): + """Lifecycle of one execution record. + + QUEUED and RUNNING are the two that occupy a task's single active slot; + everything else is terminal. + + SKIPPED is created directly terminal and never passes through QUEUED -- a + queued tombstone would collide with the very run it is recording the + overlap with (see ScheduledRun.skipped_tombstone). INTERRUPTED is kept + distinct from FAILED because a cancel or takeover is not an execution + failure, and the two lead a `once` task to different terminal states. + """ + + QUEUED = "queued" + RUNNING = "running" + SUCCESS = "success" + FAILED = "failed" + INTERRUPTED = "interrupted" + SKIPPED = "skipped" + + +class DispatchOutcome(StrEnum): + """What one dispatch attempt produced. + + A domain vocabulary rather than a set of strings: the caller branches on + all four, and the distinction between SKIPPED and CONFLICT is itself a + business rule -- a dropped scheduled occurrence is accounted for, while a + rejected on-demand trigger is reported and leaves no trace. + """ + + LAUNCHED = "launched" + SKIPPED = "skipped" + CONFLICT = "conflict" + FAILED = "failed" + + +class TriggerKind(StrEnum): + """What caused a dispatch. + + The two kinds diverge in almost every decision the domain makes — how an + overlap is handled, what status survives a failed launch, whether a paused + task stays paused — so this is a first-class concept rather than the raw + string the old service compared in four places. + """ + + SCHEDULED = "scheduled" + MANUAL = "manual" diff --git a/backend/packages/harness/deerflow/domain/schedule/model/run.py b/backend/packages/harness/deerflow/domain/schedule/model/run.py new file mode 100644 index 000000000..373387f6e --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/model/run.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import uuid +from dataclasses import dataclass, field +from datetime import UTC, datetime + +from deerflow.domain.schedule.model.enums import RunStatus + +ACTIVE_RUN_STATUSES: tuple[RunStatus, ...] = (RunStatus.QUEUED, RunStatus.RUNNING) +"""The statuses that occupy a task's single active-run slot. + +Must stay in lockstep with the predicate of the partial unique index +``uq_scheduled_task_run_active`` (``status IN ('queued','running')``, declared +in ``persistence/scheduled_task_runs/model.py``). Drift here silently +decouples the overlap fast path from its atomic arbiter — the consistency +assertion lives in a separate test module rather than the domain tests, which +stay dependency-free. +""" + +TERMINAL_RUN_STATUSES: frozenset[RunStatus] = frozenset( + { + RunStatus.SUCCESS, + RunStatus.FAILED, + RunStatus.SKIPPED, + RunStatus.INTERRUPTED, + } +) +"""Statuses a run row can no longer leave. + +Used by the repository's ``protect_terminal`` compare-and-set: a fast-failing +run can reach the completion hook before the launch path's own write lands. +""" + + +@dataclass(frozen=True) +class ScheduledRun: + """One execution record of a scheduled task — the history row. + + A separate aggregate from ``ScheduledTask``: the two are written in + independent transactions and reference each other only by ``task_id``. + """ + + record_id: str + task_id: str + thread_id: str + scheduled_for: datetime + trigger: str + status: RunStatus + run_id: str | None = None + error: str | None = None + started_at: datetime | None = None + finished_at: datetime | None = None + created_at: datetime = field(default_factory=lambda: datetime.now(UTC)) + + @classmethod + def queued(cls, *, task_id: str, thread_id: str, scheduled_for: datetime, trigger: str) -> ScheduledRun: + """The active row of a normal dispatch. + + Inserting this is what the unique index arbitrates: the loser of a + concurrent insert surfaces as ``ActiveRunConflictError``. + + The ``task-run-{hex}`` id shape is depended on by existing rows and by + the run metadata that links a Gateway run back to this record — do not + change it. + """ + return cls( + record_id=f"task-run-{uuid.uuid4().hex}", + task_id=task_id, + thread_id=thread_id, + scheduled_for=scheduled_for, + trigger=trigger, + status=RunStatus.QUEUED, + ) + + @classmethod + def skipped_tombstone(cls, *, task_id: str, thread_id: str, scheduled_for: datetime, trigger: str) -> ScheduledRun: + """A dropped occurrence, created directly as terminal ``SKIPPED``. + + Deliberately a second factory rather than ``queued()`` followed by a + status change: ``QUEUED`` falls inside ``uq_scheduled_task_run_active``'s + predicate and would collide with the pre-existing run that still holds + the task's single active slot. ``SKIPPED`` is outside the predicate and + can never conflict (service.py:249-256). + """ + return cls( + record_id=f"task-run-{uuid.uuid4().hex}", + task_id=task_id, + thread_id=thread_id, + scheduled_for=scheduled_for, + trigger=trigger, + status=RunStatus.SKIPPED, + ) + + @property + def is_active(self) -> bool: + """Whether this row occupies the task's single active-run slot.""" + return self.status in ACTIVE_RUN_STATUSES diff --git a/backend/packages/harness/deerflow/domain/schedule/model/spec.py b/backend/packages/harness/deerflow/domain/schedule/model/spec.py new file mode 100644 index 000000000..4fe54ff93 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/model/spec.py @@ -0,0 +1,191 @@ +from __future__ import annotations + +from dataclasses import dataclass +from datetime import UTC, datetime +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from croniter import croniter + +from deerflow.domain.schedule.exceptions import InvalidScheduleError +from deerflow.domain.schedule.model.enums import ScheduleType + +CRON_FIELD_COUNT = 5 + + +@dataclass(frozen=True) +class SchedulePolicy: + """Operator-tunable thresholds the domain needs but must not read itself. + + Built by the composition root from the scheduler configuration and passed + in. Deliberately not held by any aggregate: a task whose meaning changes + with deployment config is not a domain object. + + The defaults are the permissive ones on purpose -- "nobody configured a + policy" must not invent a business constraint. Real values only ever + arrive from the outer ring. + """ + + min_once_delay_seconds: int = 0 + """How far ahead a one-shot schedule must be at submission time. Read by + `ScheduleSpec.ensure_launchable`; a cron schedule is never subject to it.""" + + max_concurrent_runs: int = 1 + """Ceiling on active scheduled executions across ALL tasks. Long runs + accumulate across polls, so each poll may only claim into what is left.""" + + lease_seconds: int = 60 + """How long a claim on a task stays valid. Bounds how quickly a task + orphaned between claim and dispatch becomes reachable again.""" + + +@dataclass(frozen=True) +class ScheduleSpec: + """Parsed, validated view of (schedule_type, schedule_spec, timezone). + + The stored JSON spec is mapped in and out by the adapter layer, never here: + a `Mapping[str, Any]` in a domain signature would mean the domain is + handling a persistence/transport format. The two halves of that parsing + split cleanly — structural checks (is the key present? is it a str?) belong + to the boundary, value rules (5-field cron, resolvable timezone, run_at + present) belong to __post_init__ below. Storage keeps the same raw JSON, so + this needs no migration. + + Normalization happens in __post_init__ rather than in the factories below, + so direct construction cannot bypass it: a frozen dataclass is still + constructible field-by-field, and "valid on construction" has to hold for + that path too. + """ + + schedule_type: ScheduleType + timezone: str + cron: str | None = None + run_at: datetime | None = None + + def __post_init__(self) -> None: + # The timezone is checked first because normalizing a naive run_at + # below needs it to already be known-good. + try: + zone = ZoneInfo(self.timezone) + except ZoneInfoNotFoundError as exc: + raise InvalidScheduleError(f"Unknown timezone: {self.timezone}") from exc + + if self.schedule_type is ScheduleType.CRON: + if not self.cron: + raise InvalidScheduleError("cron schedule requires schedule_spec.cron") + fields = [part for part in self.cron.split() if part] + if len(fields) != CRON_FIELD_COUNT: + raise InvalidScheduleError(f"Cron expression must contain exactly {CRON_FIELD_COUNT} fields") + object.__setattr__(self, "cron", " ".join(fields)) + + if self.schedule_type is ScheduleType.ONCE: + if self.run_at is None: + raise InvalidScheduleError("once schedule requires run_at") + if self.run_at.tzinfo is None: + # A naive run_at means wall-clock time in this schedule's own + # timezone (schedules.py:40-43). Localizing here rather than at + # every read site keeps the rest of this class tz-aware only. + object.__setattr__(self, "run_at", self.run_at.replace(tzinfo=zone)) + + @classmethod + def cron_schedule(cls, expr: str, timezone: str) -> ScheduleSpec: + """Readability sugar — all validation lives in __post_init__.""" + return cls(ScheduleType.CRON, timezone, cron=expr) + + @classmethod + def once_at(cls, run_at: datetime, timezone: str) -> ScheduleSpec: + """Readability sugar — all validation lives in __post_init__.""" + return cls(ScheduleType.ONCE, timezone, run_at=run_at) + + @classmethod + def from_primitives(cls, schedule_type: str, *, cron: str | None, run_at: str | None, timezone: str) -> ScheduleSpec: + """Build from the loose strings both boundaries arrive as. + + The HTTP body and the stored JSON column both carry a schedule type + plus a mapping, so the rule for turning that into a value object lives + here instead of being written out once per adapter. Each adapter keeps + only what is genuinely its own — which keys its own format uses — and + the errors below stay one family the router maps uniformly. + + Four strings rather than a `Mapping[str, Any]` on purpose: a mapping in + this signature would mean the domain is handling a transport or storage + format, which is the thing this class exists to avoid. + + `cron` and `run_at` are annotated as the contract expects them but + checked rather than trusted — both callers read from data a client can + influence, so a non-string must be rejected here instead of reaching + `fromisoformat` or the cron parser. The field the given type does not + use is ignored: a boundary that carries both keys is not an error. + + Raises: + InvalidScheduleError: unknown schedule type, the type's field + missing or not a string, or an unparseable `run_at`. Value + rules — 5-field cron, resolvable timezone — are left to + __post_init__ and surface as the same error. + """ + try: + kind = ScheduleType(schedule_type) + except ValueError as exc: + raise InvalidScheduleError(f"Unsupported schedule_type: {schedule_type}") from exc + + if kind is ScheduleType.CRON: + if not isinstance(cron, str): + raise InvalidScheduleError("cron schedule requires schedule_spec.cron") + return cls.cron_schedule(cron, timezone) + + if not isinstance(run_at, str): + raise InvalidScheduleError("once schedule requires run_at") + try: + parsed = datetime.fromisoformat(run_at) + except ValueError as exc: + raise InvalidScheduleError(f"once schedule has an unparseable run_at: {run_at!r}") from exc + return cls.once_at(parsed, timezone) + + def next_after(self, now: datetime) -> datetime | None: + """Next fire time in UTC, or None when there is no future occurrence. + + The dispatch-path calculation (was `next_run_at` in schedules.py:24-55). + It applies no submission-time policy — see ensure_launchable for that, + and do not swap the two: re-arming a cron task through the stricter one + would reject it right after a perfectly normal launch. + + ONCE returns run_at while it is still ahead of `now`, else None (the + single occurrence is in the past). CRON is evaluated in this schedule's + timezone and returned as UTC. A naive `now` is read as UTC + (schedules.py:32-33). + """ + if now.tzinfo is None: + now = now.replace(tzinfo=UTC) + + if self.schedule_type is ScheduleType.ONCE: + return self.run_at if self.run_at > now else None + + zone = ZoneInfo(self.timezone) + next_local = croniter(self.cron, now.astimezone(zone)).get_next(datetime) + if next_local.tzinfo is None: + # Unreachable with an aware input on today's croniter, and kept + # verbatim from schedules.py:51-52 rather than dropped: it costs + # one branch and guards a library detail we do not control. + next_local = next_local.replace(tzinfo=zone) + return next_local.astimezone(UTC) + + def ensure_launchable(self, now: datetime, policy: SchedulePolicy) -> datetime | None: + """Next fire time, with the constraints that only apply at submission. + + Used by create/update; the dispatch path must use next_after instead. + + Raises: + InvalidScheduleError: a ONCE schedule with no future occurrence, or + one closer than policy.min_once_delay_seconds. CRON is never + subject to the delay floor (router:105, router:196). + """ + if now.tzinfo is None: + now = now.replace(tzinfo=UTC) + + next_at = self.next_after(now) + if self.schedule_type is not ScheduleType.ONCE: + return next_at + if next_at is None: + raise InvalidScheduleError("once schedule must be in the future") + if (next_at - now).total_seconds() < policy.min_once_delay_seconds: + raise InvalidScheduleError(f"once schedule must be at least {policy.min_once_delay_seconds} seconds in the future") + return next_at diff --git a/backend/packages/harness/deerflow/domain/schedule/model/task.py b/backend/packages/harness/deerflow/domain/schedule/model/task.py new file mode 100644 index 000000000..bf5b6e848 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/model/task.py @@ -0,0 +1,289 @@ +from __future__ import annotations + +import uuid +from dataclasses import dataclass, field, replace +from datetime import UTC, datetime + +from deerflow.domain.schedule.exceptions import InvalidContextModeError, TaskNotMutableError +from deerflow.domain.schedule.model.enums import ContextMode, RunStatus, ScheduleType, TaskStatus, TriggerKind +from deerflow.domain.schedule.model.spec import SchedulePolicy, ScheduleSpec + +TERMINAL_TASK_STATUSES = frozenset({TaskStatus.COMPLETED, TaskStatus.FAILED, TaskStatus.CANCELLED}) +"""The statuses in which a task's current schedule is done. + +Lives here rather than in enums.py: it is a business subset of TaskStatus, so +it belongs beside the aggregate that reasons about it. + +Two rules read it, and they share one constant rather than drifting into two +same-valued ones: + +- `with_schedule` re-arms them to ENABLED once the schedule moves into the + future — claiming only admits ENABLED rows, so leaving a terminal status + there would hand back a next_run_at that silently never fires. +- The repository's `protect_terminal` compare-and-set refuses to overwrite + them, because a fast-failing run's completion hook can land before the launch + path's own write. + +The two coincide because terminal *is* the definition of re-armable. The name +states what the statuses are; what each rule does with them belongs to that +rule. +""" + + +def _parse_context_mode(value: str | ContextMode) -> ContextMode: + """Coerce a wire-level context_mode into the enum. + + Reported as InvalidContextModeError rather than letting ValueError + escape, so the router can map the whole ScheduleError family uniformly + — an unknown mode was a 422 at router:76-77. + """ + try: + return ContextMode(value) + except ValueError as exc: + raise InvalidContextModeError(f"Unsupported context_mode: {value}") from exc + + +@dataclass(frozen=True) +class ScheduledTask: + """A user's standing instruction to run one prompt on a schedule. + + Aggregate root of the schedule context. Holds every invariant the old + router/service pair scattered across three files; nothing here knows about + HTTP, SQL, or the run runtime. + + The two bookkeeping timestamps have one owner each. `created_at` is the + aggregate's construction instant -- `create` stamps it from its explicit + `now` input and the adapter persists it verbatim, never minting its own. + `updated_at` belongs to the *storage* write path: the CAS methods + (`record_launch` / `record_completion`) update the row without ever + holding an aggregate, so only the adapter can stamp "last written" -- the + `with_*` / `paused` / `resumed` transitions therefore deliberately leave + `updated_at` alone, and the field's value on a read-back is the adapter's + stamp, seeded at construction time. Neither timestamp is a rule input; + the clock the rules see is always an explicit `now=` parameter. + """ + + task_id: str + user_id: str + title: str + prompt: str + schedule: ScheduleSpec + context_mode: ContextMode = ContextMode.FRESH_THREAD_PER_RUN + thread_id: str | None = None + assistant_id: str | None = "lead_agent" + status: TaskStatus = TaskStatus.ENABLED + # The MVP fixes this to "skip"; a single-valued enum would be noise. The + # comparison is confined to `skips_on_overlap` so a second policy only has + # to change one place. + overlap_policy: str = "skip" + next_run_at: datetime | None = None + last_run_at: datetime | None = None + last_run_id: str | None = None + last_thread_id: str | None = None + last_error: str | None = None + run_count: int = 0 + created_at: datetime = field(default_factory=lambda: datetime.now(UTC)) + updated_at: datetime = field(default_factory=lambda: datetime.now(UTC)) + + def __post_init__(self) -> None: + if self.context_mode is ContextMode.REUSE_THREAD and not self.thread_id: + raise InvalidContextModeError("reuse_thread requires thread_id") + + @classmethod + def create( + cls, + *, + user_id: str, + title: str, + prompt: str, + schedule: ScheduleSpec, + context_mode: str | ContextMode, + thread_id: str | None, + now: datetime, + policy: SchedulePolicy, + ) -> ScheduledTask: + """Factory: generate identity, normalize context, validate invariants. + + FRESH_THREAD_PER_RUN drops any supplied thread_id (router:164-165 does + the same on update) — carrying a thread the task will never use would + make `resolve_execution_thread` ambiguous. + + Thread existence/ownership is NOT checked here: that needs the + ThreadLookup port and belongs to the service. + """ + mode = _parse_context_mode(context_mode) + effective_thread = thread_id if mode is ContextMode.REUSE_THREAD else None + return cls( + task_id=f"task-{uuid.uuid4().hex}", + user_id=user_id, + title=title, + prompt=prompt, + schedule=schedule, + context_mode=mode, + thread_id=effective_thread, + next_run_at=schedule.ensure_launchable(now, policy), + # The clock is already an explicit rule input here; reading it a + # second time through the field defaults would give one + # construction two instants. + created_at=now, + updated_at=now, + ) + + @property + def skips_on_overlap(self) -> bool: + """Whether an overlapping dispatch is dropped rather than queued.""" + return self.overlap_policy == "skip" + + def resolve_execution_thread(self) -> str: + """Pick the thread this dispatch executes in (service.py:94-96). + + NOT idempotent: FRESH_THREAD_PER_RUN mints a new uuid4 on every call, + so a dispatch must call this exactly once and reuse the value for the + run row, the launch, and the result. + + The empty-thread_id fallback is kept even though __post_init__ makes it + unreachable for new aggregates — rows predating that invariant can + still carry REUSE_THREAD with no thread. + """ + if self.context_mode is ContextMode.FRESH_THREAD_PER_RUN or not self.thread_id: + return str(uuid.uuid4()) + return self.thread_id + + def ensure_mutable(self) -> None: + """Reject edits while this dispatch round's lease is held. + + Was `_ensure_task_mutable` in the router (409). Called by every + state-changing method here rather than by the caller, so the rule + cannot be forgotten at a new call site. The router historically applied + it to update/pause/resume but not trigger/delete; that asymmetry is + intentional and is preserved by keeping those two off this path. + + Raises: + TaskNotMutableError: the task is currently RUNNING. + """ + if self.status is TaskStatus.RUNNING: + raise TaskNotMutableError("Scheduled task is currently running; retry after the active execution finishes") + + def status_after_launch(self, *, trigger: TriggerKind) -> TaskStatus: + """Status to persist after a successful launch (service.py:152-161). + + Precondition on `self.status`: for a SCHEDULED trigger it is already + RUNNING (written by claim_due); for a MANUAL trigger it is the + user-facing status. The PAUSED rule below depends on that. + + Decision order is load-bearing — ONCE is tested first, so manually + triggering a paused ONCE task yields RUNNING, not PAUSED: + + ONCE -> RUNNING (await the completion + hook; declaring COMPLETED at + launch would stick if the run + fails or the process dies) + MANUAL and self.status PAUSED -> PAUSED (a manual run must not + silently resume the schedule) + otherwise -> ENABLED + """ + if self.schedule.schedule_type is ScheduleType.ONCE: + return TaskStatus.RUNNING + if trigger is TriggerKind.MANUAL and self.status is TaskStatus.PAUSED: + return TaskStatus.PAUSED + return TaskStatus.ENABLED + + def status_after_failure(self, *, trigger: TriggerKind) -> TaskStatus: + """Status to persist after a failed launch (was _task_status_for_failure). + + Decision order is load-bearing — MANUAL is tested first, so a failed + manual trigger of a ONCE task keeps its status instead of burning the + occurrence: + + MANUAL -> self.status (unchanged; the old dict-based code + needed an `or "enabled"` fallback, which the + field default makes unnecessary here) + SCHEDULED, ONCE -> FAILED + SCHEDULED, CRON -> ENABLED (the next occurrence still stands) + """ + if trigger is TriggerKind.MANUAL: + return self.status + if self.schedule.schedule_type is ScheduleType.ONCE: + return TaskStatus.FAILED + return TaskStatus.ENABLED + + def status_after_skip(self) -> TaskStatus: + """Status to persist after an overlap skip (was _task_status_for_skip). + + No `trigger` parameter: a skip only happens on the SCHEDULED path — a + manual trigger that overlaps is rejected outright and records no run + row at all, so a parameter here would imply a branch that cannot exist. + + ONCE -> FAILED (the single occurrence was lost; COMPLETED would + claim an execution that never happened) + CRON -> ENABLED + """ + if self.schedule.schedule_type is ScheduleType.ONCE: + return TaskStatus.FAILED + return TaskStatus.ENABLED + + def status_after_completion(self, outcome: RunStatus) -> TaskStatus | None: + """Status to persist when a launched run reaches a terminal state. + + Returns None when the status must not change — every CRON task, whose + schedule outlives any single run. + + CRON -> None + ONCE, SUCCESS -> COMPLETED + ONCE, INTERRUPTED -> CANCELLED (a cancel or same-thread takeover is + not an execution failure) + ONCE, otherwise -> FAILED + + The occurrence is consumed either way: the run did launch, so re-arming + would risk duplicate side effects. Note the caller writes `last_error` + unconditionally, whether or not this returns None (service.py:346). + """ + if self.schedule.schedule_type is not ScheduleType.ONCE: + return None + if outcome is RunStatus.SUCCESS: + return TaskStatus.COMPLETED + if outcome is RunStatus.INTERRUPTED: + return TaskStatus.CANCELLED + return TaskStatus.FAILED + + def with_schedule(self, schedule: ScheduleSpec, *, now: datetime, policy: SchedulePolicy) -> ScheduledTask: + """Replace the schedule and recompute next_run_at (router:172-208). + + A terminal task whose schedule just moved into the future must be + re-armed: claim_due only admits ENABLED rows, so leaving the terminal + status would return 200 with a next_run_at that silently never fires. + + Raises: + TaskNotMutableError: via ensure_mutable. + InvalidScheduleError: via ensure_launchable. + """ + self.ensure_mutable() + next_at = schedule.ensure_launchable(now, policy) + status = TaskStatus.ENABLED if next_at is not None and self.status in TERMINAL_TASK_STATUSES else self.status + return replace(self, schedule=schedule, next_run_at=next_at, status=status) + + def with_context(self, context_mode: str | ContextMode, thread_id: str | None) -> ScheduledTask: + """Replace execution context (router:153-165). + + FRESH_THREAD_PER_RUN forces thread_id to None. Thread existence is the + service's job (ThreadLookup port), not this aggregate's. + + Raises: + TaskNotMutableError: via ensure_mutable. + InvalidContextModeError: unknown mode, or REUSE_THREAD with no + thread (raised by __post_init__ on the replacement). + """ + self.ensure_mutable() + mode = _parse_context_mode(context_mode) + effective_thread = thread_id if mode is ContextMode.REUSE_THREAD else None + return replace(self, context_mode=mode, thread_id=effective_thread) + + def paused(self) -> ScheduledTask: + """Stop claiming this task until resumed.""" + self.ensure_mutable() + return replace(self, status=TaskStatus.PAUSED) + + def resumed(self) -> ScheduledTask: + """Re-admit this task to claim_due.""" + self.ensure_mutable() + return replace(self, status=TaskStatus.ENABLED) diff --git a/backend/packages/harness/deerflow/domain/schedule/ports.py b/backend/packages/harness/deerflow/domain/schedule/ports.py new file mode 100644 index 000000000..931485144 --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/ports.py @@ -0,0 +1,332 @@ +"""Output ports of the schedule context. + +Contracts the domain declares and the outer ring implements. Signatures are +technology-neutral: no SQL, table names, HTTP status codes, or run-runtime +types appear here, and every method exchanges domain objects. + +The docstrings are longer than the code on purpose -- they are the semantic +contract the adapter must honour and the contract tests are written from. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Protocol, runtime_checkable + +from deerflow.domain.schedule.model import ( + RunStatus, + ScheduledRun, + ScheduledTask, + TaskStatus, +) + + +@dataclass(frozen=True) +class LaunchedRun: + """What the launcher reports back once a run is admitted. + + `thread_id` is echoed rather than assumed: the launcher is free to return a + different thread than the one requested, and the task's bookkeeping records + what actually ran. + """ + + run_id: str + thread_id: str + + +@dataclass(frozen=True) +class RunOutcome: + """A launched run reaching a terminal state, in domain vocabulary. + + The app layer converts its own run record into this before calling the + service, so the domain never imports the run runtime. That converter also + owns the filtering the completion hook used to do inline -- a run that + carries no scheduled-task metadata, or has not reached a terminal state, + simply produces no RunOutcome and the service is never called. + + `status` is narrowed to the three terminal outcomes a scheduled run can + report: SUCCESS, FAILED, INTERRUPTED. INTERRUPTED is deliberately distinct + from FAILED -- a cancel or same-thread takeover is not an execution + failure, and the task ends CANCELLED rather than FAILED. + """ + + task_id: str + record_id: str + run_id: str + user_id: str + status: RunStatus + error: str | None + + +@runtime_checkable +class ScheduledTaskRepository(Protocol): + """Persistence port for the task aggregate. + + Every read is scoped by `user_id`: a task belonging to someone else is + reported as absent (None / False / omitted from a list), never as a + permission error -- the caller must not be able to distinguish "not yours" + from "does not exist". + """ + + async def add(self, task: ScheduledTask) -> ScheduledTask: + """Insert a new task and return the stored state.""" + ... + + async def get(self, task_id: str, *, user_id: str) -> ScheduledTask | None: + """Return the task, or None when it is absent or owned by someone else.""" + ... + + async def list_by_user(self, user_id: str) -> list[ScheduledTask]: + """Every task owned by the user, newest first.""" + ... + + async def list_by_user_and_thread(self, user_id: str, thread_id: str) -> list[ScheduledTask]: + """The user's tasks bound to one thread, newest first. + + Only `reuse_thread` tasks can match: a `fresh_thread_per_run` task + carries no thread. + """ + ... + + async def save(self, task: ScheduledTask) -> ScheduledTask | None: + """Persist a whole aggregate, keyed by its own id and owner. + + Whole-aggregate replacement rather than a field patch: the aggregate is + immutable, so a caller that changed anything is holding a complete new + value. Returns None when the row is absent or owned by someone else. + + Not to be used for the post-dispatch write -- see `record_launch`. + """ + ... + + async def delete(self, task_id: str, *, user_id: str) -> bool: + """Remove the task. False when it was absent or owned by someone else.""" + ... + + async def claim_due(self, *, now: datetime, lease_seconds: int, limit: int) -> list[ScheduledTask]: + """Atomically take ownership of up to `limit` tasks that are due. + + A task is due when its next fire time has passed AND either it is + claimable (enabled, with no live claim), or it is stuck mid-dispatch + with an expired claim -- the process that took it died between claiming + and dispatching, and it must not stay unreachable forever. + + Claiming marks the tasks as running and stamps the claim, so a + concurrent claimer cannot take the same ones. Returns them in the state + they were left in *after* the claim. + + Which process is claiming is deliberately absent: it is an identity, + not a rule. The implementation may record one for diagnostics, but + nothing reads it back -- expiry alone decides whether a claim can be + taken over -- so the domain has no reason to carry it. + + Atomicity is the implementation's responsibility. An in-memory double + can satisfy every rule above under single-threaded use while providing + no concurrency guarantee at all; that difference is out of scope for + the contract tests and is covered separately against a real database. + """ + ... + + async def record_launch( + self, + task_id: str, + *, + status: TaskStatus, + next_run_at: datetime | None, + last_run_at: datetime | None, + last_run_id: str | None, + last_thread_id: str | None, + last_error: str | None, + increment_run_count: bool, + protect_terminal: bool = False, + ) -> None: + """Write the outcome of a dispatch and release the claim. + + Deliberately NOT expressed as `save(task)`: `protect_terminal` makes + this a compare-and-set against a run that may be finalizing + concurrently. When it is set and the stored task has already reached a + terminal status, the status and error are left alone and only the + scheduling bookkeeping is written -- a read-modify-write through the + aggregate would reintroduce the very race the flag exists to close. + + Every field is assigned unconditionally, so a caller preserving a value + must pass the current one back. The claim is always released. + """ + ... + + async def record_completion( + self, + task_id: str, + *, + user_id: str, + status: TaskStatus | None, + error: str | None, + ) -> None: + """Write a finished run's verdict onto the task. + + Deliberately NOT expressed as `save(task)`, for the same reason + `record_launch` is not: the two race. A run that fails fast reaches its + completion hook while the dispatch path is still writing its + bookkeeping, so a read-modify-write through the aggregate would replay + a snapshot taken before that write and roll it back -- restoring an + elapsed `next_run_at` and an out-of-date `run_count`, and leaving the + task in `running` with no live claim, which neither `claim_due` branch + nor `cancel_stuck_once_tasks` can reach. + + The two writes therefore own disjoint fields: the launch owns the + schedule, this owns the verdict. Nothing here touches `next_run_at`, + `last_run_at`, `last_run_id`, `last_thread_id`, `run_count` or the + claim. + + `status` is `None` when the verdict must not move the status -- every + cron task, whose schedule outlives any single run. `error` is written + either way, so a cron task still reports what went wrong last time. + + Scoped by `user_id` like every other read: a task belonging to someone + else is left alone rather than reported. An unknown task is ignored -- + one deleted mid-flight simply has nothing to update. + """ + ... + + async def cancel_stuck_once_tasks(self, *, error: str) -> int: + """Reconcile `once` tasks orphaned mid-flight by a process crash. + + A launched `once` task waits in running for its completion hook, and + its claim was released at launch -- so the expired-claim branch of + `claim_due` can never see it, and after a crash the hook is gone. This + marks those tasks cancelled and returns how many were reconciled. + + Tasks still holding a claim are left alone: they were claimed but not + launched, and expired-claim reclaim recovers them safely. + """ + ... + + +@runtime_checkable +class ScheduledRunRepository(Protocol): + """Persistence port for execution records. + + Unlike tasks, runs are not read by owner: they are only ever reached + through a task the caller already proved it owns. + """ + + async def add(self, run: ScheduledRun) -> ScheduledRun: + """Insert an execution record. + + Raises: + ActiveRunConflictError: the task already holds its single active + slot. Translating the storage-level rejection into this domain + error is the adapter's job and is load-bearing -- the service + collapses it to exactly the same outcome as the `has_active` + fast path, and those two paths must stay indistinguishable. + + A terminal record (a skip tombstone) is outside the active-slot rule + and must never raise it. + """ + ... + + async def list_by_task(self, task_id: str, *, limit: int, offset: int) -> list[ScheduledRun]: + """One task's execution history, newest first.""" + ... + + async def count_active(self) -> int: + """How many executions are active across ALL tasks. + + Global on purpose: it bounds total scheduled concurrency, not per-task + overlap. Long runs accumulate across polls, so each poll may only claim + into whatever budget is left. + """ + ... + + async def has_active(self, task_id: str) -> bool: + """Whether this task currently holds its active slot. + + A non-atomic fast path by nature -- it cannot be relied on to exclude a + concurrent dispatch. `add` is the arbiter; this exists to avoid the + common case of doing pointless work. + """ + ... + + async def update_status( + self, + record_id: str, + *, + status: RunStatus, + run_id: str | None = None, + error: str | None = None, + started_at: datetime | None = None, + finished_at: datetime | None = None, + protect_terminal: bool = False, + ) -> None: + """Advance an execution record. + + With `protect_terminal`, a record that already reached a terminal state + keeps its status and error; only bookkeeping the terminal write could + not have known (the run id, the start time) is backfilled. A run that + fails fast can reach its completion hook before the launch path's own + write lands, and the completion is the authoritative one. + + Unknown record ids are ignored rather than raising: the caller is + writing bookkeeping, not asserting existence. + """ + ... + + async def mark_stale_active(self, *, error: str) -> int: + """Terminalize executions orphaned by a process crash, returning the count. + + Runs execute in-process, so any record still active at startup belongs + to a process that is gone. This is only sound while a single scheduler + instance owns the table. + """ + ... + + +class RunLauncher(Protocol): + """Output port for actually starting the work. + + The contract the adapter MUST honour, because it is what keeps the run + runtime and the web framework out of the inner ring: + + - the execution thread is already busy -> ThreadBusyError + - anything else goes wrong -> LaunchFailedError + + Nothing else may escape. The domain distinguishes those two because they + lead to different outcomes -- a busy thread on a scheduled dispatch is a + skipped occurrence, while a genuine failure is recorded as one. + """ + + async def launch( + self, + *, + thread_id: str, + assistant_id: str | None, + prompt: str, + owner_user_id: str | None, + metadata: dict[str, str], + ) -> LaunchedRun: + """Start one execution and return its identity. + + `metadata` is opaque correlation data the domain attaches so the + eventual outcome can be traced back to this task and record; the + adapter must carry it through untouched. + """ + ... + + +class ThreadLookup(Protocol): + """Narrow port: the only question the schedule context asks about threads. + + Deliberately not the full thread store -- depending on this one method + keeps the context decoupled from the wider conversation model. + """ + + async def exists_for_user(self, thread_id: str, user_id: str) -> bool: + """Whether this thread exists AND the user may use it. + + Both halves matter: binding a task to a thread that does not exist yet + is as invalid as binding it to someone else's. The two are deliberately + not distinguished in the result -- reporting them differently would let + a caller probe for the existence of threads they cannot see. + """ + ... diff --git a/backend/packages/harness/deerflow/domain/schedule/service.py b/backend/packages/harness/deerflow/domain/schedule/service.py new file mode 100644 index 000000000..7f31d37be --- /dev/null +++ b/backend/packages/harness/deerflow/domain/schedule/service.py @@ -0,0 +1,462 @@ +"""Application service of the schedule context (its input port). + +Orchestrates use cases only: fetch, apply domain rules, persist through output +ports. It holds no business rules itself -- every decision below is delegated +to the aggregate -- and knows nothing about HTTP, SQL, or the run runtime. +`user_id` is always passed in explicitly; resolving the current user is the +primary adapter's job. + +The dispatch path deliberately mirrors the structure of the pre-migration +`app/scheduler/service.py` (since deleted), including its ordering and its +comments, because its concurrency and idempotency semantics are load-bearing +and are pinned by tests that must keep passing unchanged. +""" + +from __future__ import annotations + +from dataclasses import dataclass, replace +from datetime import datetime + +from deerflow.domain.schedule.commands import ( + UNSET, + CreateScheduledTask, + DeleteTask, + PauseTask, + ResumeTask, + TriggerTask, + UpdateScheduledTask, +) +from deerflow.domain.schedule.exceptions import ( + ActiveRunConflictError, + LaunchFailedError, + TaskNotFoundError, + ThreadBusyError, + ThreadNotFoundError, +) +from deerflow.domain.schedule.model import ( + ContextMode, + DispatchOutcome, + RunStatus, + ScheduledRun, + ScheduledTask, + SchedulePolicy, + ScheduleType, + TriggerKind, +) +from deerflow.domain.schedule.ports import ( + RunLauncher, + RunOutcome, + ScheduledRunRepository, + ScheduledTaskRepository, + ThreadLookup, +) + +# Shared so the has_active fast path and the active-slot race path produce +# byte-identical outcomes for the same "task already has an active run" +# condition. Two callers must not be able to tell which one rejected them. +_ACTIVE_RUN_CONFLICT_ERROR = "task already has an active run" +_SKIP_ACTIVE_RUN_ERROR = "skipped: a previous run of this task is still active" + + +@dataclass(frozen=True) +class DispatchResult: + """What one dispatch attempt produced. + + `outcome` drives the caller's protocol mapping -- a manual trigger turns + CONFLICT into 409 and FAILED into 502 -- so it is domain vocabulary + rather than a bare string. + """ + + outcome: DispatchOutcome + record_id: str | None + run_id: str | None + thread_id: str + error: str | None + + +class ScheduleService: + """Every scheduled-task use case, orchestrated over the four output ports. + + The input port of this context: primary adapters (an HTTP router, the + poller, the run-completion hook) call these methods and translate what + comes back. Domain errors are allowed to propagate -- mapping them onto a + protocol is the adapter's job, not this class's. + """ + + def __init__( + self, + *, + tasks: ScheduledTaskRepository, + runs: ScheduledRunRepository, + launcher: RunLauncher, + threads: ThreadLookup, + policy: SchedulePolicy, + ) -> None: + self._tasks = tasks + self._runs = runs + self._launcher = launcher + self._threads = threads + self._policy = policy + + # ------------------------------------------------------------------ reads + + async def list_tasks(self, user_id: str) -> list[ScheduledTask]: + """Every task the user owns, newest first.""" + return await self._tasks.list_by_user(user_id) + + async def list_tasks_by_thread(self, user_id: str, thread_id: str) -> list[ScheduledTask]: + """The user's tasks bound to one thread. + + Only reuse_thread tasks can appear: a fresh-thread task carries no + binding to show. + """ + return await self._tasks.list_by_user_and_thread(user_id, thread_id) + + async def get_task(self, task_id: str, *, user_id: str) -> ScheduledTask: + """Raises TaskNotFoundError when absent or owned by someone else -- the + caller must not be able to tell those apart.""" + task = await self._tasks.get(task_id, user_id=user_id) + if task is None: + raise TaskNotFoundError("Scheduled task not found") + return task + + async def list_task_runs(self, task_id: str, *, user_id: str, limit: int = 50, offset: int = 0) -> list[ScheduledRun]: + """Execution history, gated on ownership of the parent task.""" + await self.get_task(task_id, user_id=user_id) + return await self._runs.list_by_task(task_id, limit=limit, offset=offset) + + # ------------------------------------------------------------------ writes + + async def create_scheduled_task(self, cmd: CreateScheduledTask, *, now: datetime) -> ScheduledTask: + """Register a new standing instruction. + + The aggregate is built first so a malformed schedule or context mode is + reported as such before any IO happens; only then is the thread + binding verified. + + Raises: + InvalidScheduleError / InvalidContextModeError: from the aggregate. + ThreadNotFoundError: reuse_thread pointing at a thread the user + cannot use. + """ + task = ScheduledTask.create( + user_id=cmd.user_id, + title=cmd.title, + prompt=cmd.prompt, + schedule=cmd.schedule, + context_mode=cmd.context_mode, + thread_id=cmd.thread_id, + now=now, + policy=self._policy, + ) + await self._require_thread(task) + return await self._tasks.add(task) + + async def update_scheduled_task(self, cmd: UpdateScheduledTask, *, now: datetime) -> ScheduledTask: + """Partially update a task; ``UNSET`` means "not supplied". + + The one field for which ``None`` is a meaningful value, `thread_id`, + travels inside `ContextChange` where it is unambiguous. + + Context and schedule are applied through the aggregate's own + transitions, so the re-arm rule and the running-task gate cannot be + bypassed by patching fields directly. + """ + task = await self.get_task(cmd.task_id, user_id=cmd.user_id) + task.ensure_mutable() + + if cmd.context is not UNSET: + task = task.with_context(cmd.context.context_mode, cmd.context.thread_id) + await self._require_thread(task) + if cmd.schedule is not UNSET: + task = task.with_schedule(cmd.schedule, now=now, policy=self._policy) + if cmd.title is not UNSET: + task = replace(task, title=cmd.title) + if cmd.prompt is not UNSET: + task = replace(task, prompt=cmd.prompt) + + return await self._save(task) + + async def pause_task(self, cmd: PauseTask) -> ScheduledTask: + """Stop claiming this task until it is resumed. + + Refused while the task is being dispatched -- see `ensure_mutable`. + """ + task = await self.get_task(cmd.task_id, user_id=cmd.user_id) + return await self._save(task.paused()) + + async def resume_task(self, cmd: ResumeTask) -> ScheduledTask: + """Re-admit this task to claiming. Same gate as `pause_task`.""" + task = await self.get_task(cmd.task_id, user_id=cmd.user_id) + return await self._save(task.resumed()) + + async def delete_task(self, cmd: DeleteTask) -> None: + """Deleting is deliberately not gated on the task being idle: the + pre-migration router applied that gate to update/pause/resume only.""" + if not await self._tasks.delete(cmd.task_id, user_id=cmd.user_id): + raise TaskNotFoundError("Scheduled task not found") + + # ------------------------------------------------------------------ dispatch + + async def trigger_task(self, cmd: TriggerTask, *, now: datetime) -> DispatchResult: + """Dispatch a task on demand. Unlike the scheduled path this is allowed + while the task is paused, and leaves it paused.""" + task = await self.get_task(cmd.task_id, user_id=cmd.user_id) + return await self.dispatch_task(task, now=now, trigger=TriggerKind.MANUAL) + + async def run_once(self, *, now: datetime) -> list[DispatchResult]: + """Claim whatever is due and dispatch it. + + `max_concurrent_runs` is a global cap on active scheduled runs, not a + per-poll batch size: long runs accumulate across poll cycles, so each + cycle only claims into the remaining budget. + """ + active = await self._runs.count_active() + budget = self._policy.max_concurrent_runs - active + if budget <= 0: + return [] + claimed = await self._tasks.claim_due(now=now, lease_seconds=self._policy.lease_seconds, limit=budget) + return [await self.dispatch_task(task, now=now, trigger=TriggerKind.SCHEDULED) for task in claimed] + + async def dispatch_task(self, task: ScheduledTask, *, now: datetime, trigger: TriggerKind) -> DispatchResult: + """Turn one due task into one execution. + + Called once per dispatch, so `resolve_execution_thread` is called once + and its value reused for the record, the launch, and the result. + """ + execution_thread_id = task.resolve_execution_thread() + + # "skip" must hold for fresh-thread runs too, where every run gets a + # new thread and the same-thread busy signal below can never fire. + # Checked before creating this dispatch's own record so the record does + # not count itself as the active run. A manual trigger against an + # active run is rejected outright instead of being recorded as a + # skipped occurrence -- nothing was scheduled to happen. + # + # This check is a NON-ATOMIC fast path: two concurrent dispatches (a + # manual trigger racing the poller, a double-click, a client retry) can + # both observe no active run. The repository is the atomic arbiter -- + # it rejects the second active record with ActiveRunConflictError, + # which collapses to the SAME outcome as this fast path just below. + if task.skips_on_overlap and await self._runs.has_active(task.task_id): + if trigger is TriggerKind.MANUAL: + return self._conflict(execution_thread_id) + return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger) + + record = ScheduledRun.queued( + task_id=task.task_id, + thread_id=execution_thread_id, + scheduled_for=now, + trigger=trigger, + ) + try: + await self._runs.add(record) + except ActiveRunConflictError: + # Lost the race for the task's single active slot: a concurrent + # dispatch passed the same fast-path check and inserted first. + # Identical outcome to the fast path above -- that equality is what + # the dispatch-race regression tests pin. + if trigger is TriggerKind.MANUAL: + return self._conflict(execution_thread_id) + return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger) + + # Only the launch is guarded. The port contract admits exactly two + # escapes, so a failure of the bookkeeping writes below is a genuine + # fault and propagates instead of being recorded as a failed launch -- + # which would mark an already-running execution as failed. + try: + launched = await self._launcher.launch( + thread_id=execution_thread_id, + assistant_id=task.assistant_id, + prompt=task.prompt, + owner_user_id=task.user_id, + metadata={ + "scheduled_task_id": task.task_id, + "scheduled_task_run_id": record.record_id, + "scheduled_trigger": str(trigger), + }, + ) + except ThreadBusyError as exc: + # The execution thread is already busy. On the scheduled path under + # a skip policy this is an overlap like any other; anything else is + # reported as a conflict the caller has to deal with. + if trigger is TriggerKind.SCHEDULED and task.skips_on_overlap: + return await self._finalize_skip(task, record_id=record.record_id, thread_id=execution_thread_id, now=now, error=str(exc)) + return await self._fail(task, record_id=record.record_id, thread_id=execution_thread_id, now=now, trigger=trigger, error=str(exc), outcome=DispatchOutcome.CONFLICT) + except LaunchFailedError as exc: + return await self._fail(task, record_id=record.record_id, thread_id=execution_thread_id, now=now, trigger=trigger, error=str(exc), outcome=DispatchOutcome.FAILED) + + await self._runs.update_status( + record.record_id, + status=RunStatus.RUNNING, + run_id=launched.run_id, + started_at=now, + # A fast-failing run can reach handle_run_completion before this + # write lands; never clobber its terminal verdict. + protect_terminal=True, + ) + await self._tasks.record_launch( + task.task_id, + status=task.status_after_launch(trigger=trigger), + next_run_at=task.schedule.next_after(now), + last_run_at=now, + last_run_id=launched.run_id, + last_thread_id=launched.thread_id, + last_error=None, + increment_run_count=True, + # Same race as the record write above. + protect_terminal=True, + ) + return DispatchResult(DispatchOutcome.LAUNCHED, record.record_id, launched.run_id, launched.thread_id, None) + + # ------------------------------------------------------------------ lifecycle + + async def handle_run_completion(self, outcome: RunOutcome, *, now: datetime) -> None: + """Write back a launched run's terminal verdict. + + A task deleted while its run was in flight simply has nothing to + update, and that is not an error. + """ + await self._runs.update_status( + outcome.record_id, + status=outcome.status, + run_id=outcome.run_id, + error=outcome.error, + finished_at=now, + ) + # Read to ask the aggregate what this outcome means, then write only + # that. `status_after_completion` reads nothing but the schedule type, + # which no concurrent write can change, so this read carries no + # time-of-check risk -- and `record_completion` deliberately does not + # write the fields that a concurrent `record_launch` owns. + task = await self._tasks.get(outcome.task_id, user_id=outcome.user_id) + if task is None: + return + # The error is recorded whether or not the status moves: a cron task + # keeps its schedule but still reports what went wrong last time. + await self._tasks.record_completion( + outcome.task_id, + user_id=outcome.user_id, + status=task.status_after_completion(outcome.status), + error=outcome.error, + ) + + async def reconcile_on_startup(self, *, error: str) -> tuple[int, int]: + """Clean up what a process crash left behind, returning what was fixed. + + Two sweeps, because a crash strands two different things: execution + records that can never finish, and `once` tasks parked waiting for a + completion hook that died with the process. The second is not covered + by expired-claim reclaim -- a launched task released its claim, so the + claim query can never see it again. + + Failures propagate: whether a partial reconcile should block startup is + the caller's policy, not the domain's. + """ + stale_runs = await self._runs.mark_stale_active(error=error) + stuck_tasks = await self._tasks.cancel_stuck_once_tasks(error=error) + return stale_runs, stuck_tasks + + # ------------------------------------------------------------------ internals + + async def _require_thread(self, task: ScheduledTask) -> None: + if task.context_mode is not ContextMode.REUSE_THREAD: + return + # The aggregate guarantees a thread here; the check keeps that + # guarantee from being silently dropped under `python -O`. + if not task.thread_id or not await self._threads.exists_for_user(task.thread_id, task.user_id): + raise ThreadNotFoundError("Thread not found") + + async def _save(self, task: ScheduledTask) -> ScheduledTask: + """Persist an already-validated aggregate. + + Read-modify-write rather than a conditional UPDATE: pushing the + "not while running" rule into a storage predicate would put it beyond + the reach of a zero-IO test and give it a second home. The pre-existing + code had the same shape, and closing the window properly needs + optimistic locking, which needs a schema change. + """ + saved = await self._tasks.save(task) + if saved is None: + raise TaskNotFoundError("Scheduled task not found") + return saved + + def _conflict(self, thread_id: str) -> DispatchResult: + """A manual trigger against an active run. + + No history record is written: nothing was scheduled to happen, so there + is no occurrence to account for. + """ + return DispatchResult(DispatchOutcome.CONFLICT, None, None, thread_id, _ACTIVE_RUN_CONFLICT_ERROR) + + async def _record_scheduled_skip(self, task: ScheduledTask, *, thread_id: str, now: datetime, trigger: TriggerKind) -> DispatchResult: + """Account for a scheduled occurrence dropped because of an overlap. + + The tombstone is created directly terminal rather than as the transient + queued record the launch path uses: a queued record is active and would + itself be refused against the pre-existing run that is still holding + the task's single active slot. + """ + record = ScheduledRun.skipped_tombstone( + task_id=task.task_id, + thread_id=thread_id, + scheduled_for=now, + trigger=trigger, + ) + await self._runs.add(record) + return await self._finalize_skip(task, record_id=record.record_id, thread_id=thread_id, now=now, error=_SKIP_ACTIVE_RUN_ERROR) + + async def _finalize_skip(self, task: ScheduledTask, *, record_id: str, thread_id: str, now: datetime, error: str) -> DispatchResult: + await self._runs.update_status( + record_id, + status=RunStatus.SKIPPED, + error=error, + started_at=now, + finished_at=now, + ) + await self._tasks.record_launch( + task.task_id, + status=task.status_after_skip(), + next_run_at=task.schedule.next_after(now), + # A skip is not an execution, so the launch bookkeeping carries + # over unchanged; record_launch assigns unconditionally, so + # "unchanged" has to be spelled out. + last_run_at=task.last_run_at, + last_run_id=task.last_run_id, + last_thread_id=task.last_thread_id, + # Only a lost one-shot occurrence is worth surfacing; a cron task + # simply waits for its next turn. + last_error=error if task.schedule.schedule_type is ScheduleType.ONCE else None, + increment_run_count=False, + ) + return DispatchResult(DispatchOutcome.SKIPPED, record_id, None, thread_id, error) + + async def _fail( + self, + task: ScheduledTask, + *, + record_id: str, + thread_id: str, + now: datetime, + trigger: TriggerKind, + error: str, + outcome: DispatchOutcome, + ) -> DispatchResult: + await self._runs.update_status( + record_id, + status=RunStatus.FAILED, + error=error, + started_at=now, + finished_at=now, + ) + await self._tasks.record_launch( + task.task_id, + status=task.status_after_failure(trigger=trigger), + next_run_at=task.schedule.next_after(now), + last_run_at=now, + last_run_id=None, + last_thread_id=thread_id, + last_error=error, + increment_run_count=False, + ) + return DispatchResult(outcome, record_id, None, thread_id, error) diff --git a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/__init__.py b/backend/packages/harness/deerflow/persistence/scheduled_task_runs/__init__.py index 8c014cf90..6669b4054 100644 --- a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/__init__.py +++ b/backend/packages/harness/deerflow/persistence/scheduled_task_runs/__init__.py @@ -1,4 +1,12 @@ -from .model import ScheduledTaskRunRow -from .sql import ActiveScheduledRunConflict, ScheduledTaskRunRepository +"""ORM row of the scheduled_task_runs table. -__all__ = ["ActiveScheduledRunConflict", "ScheduledTaskRunRow", "ScheduledTaskRunRepository"] +The table definition -- including the partial unique index +``uq_scheduled_task_run_active`` that arbitrates the single active slot -- +lives here with the shared engine/alembic infrastructure; its only reader and +writer is the schedule context's secondary adapter +(``app/adapters/schedule/scheduled_run_repository.py``). +""" + +from .model import ScheduledTaskRunRow + +__all__ = ["ScheduledTaskRunRow"] diff --git a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py b/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py deleted file mode 100644 index afedb4b4f..000000000 --- a/backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py +++ /dev/null @@ -1,171 +0,0 @@ -from __future__ import annotations - -from datetime import UTC, datetime -from typing import Any - -from sqlalchemy import func, select -from sqlalchemy.exc import IntegrityError -from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker - -from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow -from deerflow.utils.time import coerce_iso - -TERMINAL_RUN_STATUSES: frozenset[str] = frozenset({"success", "failed", "skipped", "interrupted"}) -ACTIVE_RUN_STATUSES: tuple[str, ...] = ("queued", "running") - - -class ActiveScheduledRunConflict(Exception): - """A concurrent dispatch already holds the task's single active-run slot. - - Raised by :meth:`ScheduledTaskRunRepository.create` when inserting an - active (queued/running) run row would violate the partial unique index - ``uq_scheduled_task_run_active`` (at most one active run per ``task_id``). - This is the atomic counterpart to the non-atomic ``has_active_runs`` check - in ``ScheduledTaskService.dispatch_task``: two dispatches can both pass that - check, but only one can insert the active row — the loser lands here. - - Translating the SQLAlchemy ``IntegrityError`` into a domain exception at - the repository boundary keeps the service layer free of ``sqlalchemy.exc`` - coupling (mirrors ``deerflow.runtime.ConflictError`` for the runs table). - """ - - def __init__(self, task_id: str) -> None: - self.task_id = task_id - super().__init__(f"scheduled task {task_id!r} already has an active run") - - -class ScheduledTaskRunRepository: - def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None: - self._sf = session_factory - - @staticmethod - def _row_to_dict(row: ScheduledTaskRunRow) -> dict[str, Any]: - data = row.to_dict() - for key in ("scheduled_for", "started_at", "finished_at", "created_at"): - if data.get(key) is not None: - data[key] = coerce_iso(data[key]) - return data - - async def create( - self, - *, - run_record_id: str, - task_id: str, - thread_id: str, - scheduled_for: datetime, - trigger: str, - status: str, - ) -> dict[str, Any]: - row = ScheduledTaskRunRow( - id=run_record_id, - task_id=task_id, - thread_id=thread_id, - scheduled_for=scheduled_for, - trigger=trigger, - status=status, - created_at=datetime.now(UTC), - ) - async with self._sf() as session: - session.add(row) - try: - await session.commit() - except IntegrityError: - await session.rollback() - # Only active-status inserts can trip the partial unique index - # ``uq_scheduled_task_run_active``; a terminal-status row (e.g. - # a "skipped" tombstone) is outside its predicate and cannot - # conflict, so any IntegrityError there is a genuine fault and - # is re-raised untranslated. - if status in ACTIVE_RUN_STATUSES: - raise ActiveScheduledRunConflict(task_id) from None - raise - await session.refresh(row) - return self._row_to_dict(row) - - async def list_by_task(self, task_id: str, *, limit: int = 50, offset: int = 0) -> list[dict[str, Any]]: - stmt = ( - select(ScheduledTaskRunRow) - .where(ScheduledTaskRunRow.task_id == task_id) - .order_by( - ScheduledTaskRunRow.created_at.desc(), - ScheduledTaskRunRow.id.desc(), - ) - .limit(limit) - .offset(offset) - ) - async with self._sf() as session: - result = await session.execute(stmt) - return [self._row_to_dict(row) for row in result.scalars()] - - async def count_active_runs(self) -> int: - """Global count of queued/running rows, used to bound cross-task concurrency.""" - stmt = select(func.count()).select_from(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(ACTIVE_RUN_STATUSES)) - async with self._sf() as session: - result = await session.execute(stmt) - return int(result.scalar() or 0) - - async def update_status( - self, - run_record_id: str, - *, - status: str, - run_id: str | None = None, - error: str | None = None, - started_at: datetime | None = None, - finished_at: datetime | None = None, - protect_terminal: bool = False, - ) -> None: - async with self._sf() as session: - row = await session.get(ScheduledTaskRunRow, run_record_id) - if row is None: - return - if protect_terminal and row.status in TERMINAL_RUN_STATUSES: - # The launch-path "running" write lost the race against the - # completion hook; keep the terminal status/error and only - # backfill bookkeeping the completion write could not know. - if row.run_id is None and run_id is not None: - row.run_id = run_id - if row.started_at is None and started_at is not None: - row.started_at = started_at - await session.commit() - return - row.status = status - row.run_id = run_id - row.error = error - if started_at is not None: - row.started_at = started_at - if finished_at is not None: - row.finished_at = finished_at - await session.commit() - - async def has_active_runs(self, task_id: str) -> bool: - stmt = ( - select(ScheduledTaskRunRow.id) - .where( - ScheduledTaskRunRow.task_id == task_id, - ScheduledTaskRunRow.status.in_(ACTIVE_RUN_STATUSES), - ) - .limit(1) - ) - async with self._sf() as session: - result = await session.execute(stmt) - return result.scalars().first() is not None - - async def mark_stale_active_runs(self, *, error: str) -> int: - """Fail-fast bookkeeping for runs orphaned by a process crash. - - Agent runs execute in-process, so any ``queued``/``running`` row found - at scheduler startup belongs to a run whose process is gone. Only valid - under the MVP's single-scheduler-instance assumption. - """ - stmt = select(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(ACTIVE_RUN_STATUSES)) - now = datetime.now(UTC) - async with self._sf() as session: - result = await session.execute(stmt) - rows = list(result.scalars()) - for row in rows: - row.status = "interrupted" - row.error = error - row.finished_at = now - await session.commit() - return len(rows) diff --git a/backend/packages/harness/deerflow/persistence/scheduled_tasks/__init__.py b/backend/packages/harness/deerflow/persistence/scheduled_tasks/__init__.py index 18b6ae7fe..8cd1e94f6 100644 --- a/backend/packages/harness/deerflow/persistence/scheduled_tasks/__init__.py +++ b/backend/packages/harness/deerflow/persistence/scheduled_tasks/__init__.py @@ -1,4 +1,10 @@ -from .model import ScheduledTaskRow -from .sql import ScheduledTaskRepository +"""ORM row of the scheduled_tasks table. -__all__ = ["ScheduledTaskRow", "ScheduledTaskRepository"] +The table definition lives here with the shared engine/alembic +infrastructure; its only reader and writer is the schedule context's +secondary adapter (``app/adapters/schedule/scheduled_task_repository.py``). +""" + +from .model import ScheduledTaskRow + +__all__ = ["ScheduledTaskRow"] diff --git a/backend/packages/harness/deerflow/persistence/scheduled_tasks/sql.py b/backend/packages/harness/deerflow/persistence/scheduled_tasks/sql.py deleted file mode 100644 index 7f05f12bd..000000000 --- a/backend/packages/harness/deerflow/persistence/scheduled_tasks/sql.py +++ /dev/null @@ -1,232 +0,0 @@ -from __future__ import annotations - -from datetime import UTC, datetime, timedelta -from typing import Any - -from sqlalchemy import and_, or_, select -from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker - -from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow -from deerflow.utils.time import coerce_iso - -TERMINAL_TASK_STATUSES: frozenset[str] = frozenset({"completed", "failed", "cancelled"}) - - -class ScheduledTaskRepository: - def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None: - self._sf = session_factory - - @staticmethod - def _row_to_dict(row: ScheduledTaskRow) -> dict[str, Any]: - data = row.to_dict() - for key in ( - "created_at", - "updated_at", - "next_run_at", - "last_run_at", - "lease_expires_at", - ): - if data.get(key) is not None: - data[key] = coerce_iso(data[key]) - return data - - async def create( - self, - *, - task_id: str, - user_id: str, - thread_id: str | None, - context_mode: str, - assistant_id: str | None, - title: str, - prompt: str, - schedule_type: str, - schedule_spec: dict[str, Any], - timezone: str, - next_run_at: datetime | None, - ) -> dict[str, Any]: - now = datetime.now(UTC) - row = ScheduledTaskRow( - id=task_id, - user_id=user_id, - thread_id=thread_id, - context_mode=context_mode, - assistant_id=assistant_id, - title=title, - prompt=prompt, - schedule_type=schedule_type, - schedule_spec=schedule_spec, - timezone=timezone, - next_run_at=next_run_at, - created_at=now, - updated_at=now, - ) - async with self._sf() as session: - session.add(row) - await session.commit() - await session.refresh(row) - return self._row_to_dict(row) - - async def get(self, task_id: str, *, user_id: str) -> dict[str, Any] | None: - async with self._sf() as session: - row = await session.get(ScheduledTaskRow, task_id) - if row is None or row.user_id != user_id: - return None - return self._row_to_dict(row) - - async def list_by_user(self, user_id: str) -> list[dict[str, Any]]: - stmt = select(ScheduledTaskRow).where(ScheduledTaskRow.user_id == user_id).order_by(ScheduledTaskRow.created_at.desc(), ScheduledTaskRow.id.desc()) - async with self._sf() as session: - result = await session.execute(stmt) - return [self._row_to_dict(row) for row in result.scalars()] - - async def update( - self, - task_id: str, - *, - user_id: str, - updates: dict[str, Any], - ) -> dict[str, Any] | None: - async with self._sf() as session: - row = await session.get(ScheduledTaskRow, task_id) - if row is None or row.user_id != user_id: - return None - for key, value in updates.items(): - if hasattr(row, key): - setattr(row, key, value) - row.updated_at = datetime.now(UTC) - await session.commit() - await session.refresh(row) - return self._row_to_dict(row) - - async def delete(self, task_id: str, *, user_id: str) -> bool: - async with self._sf() as session: - row = await session.get(ScheduledTaskRow, task_id) - if row is None or row.user_id != user_id: - return False - await session.delete(row) - await session.commit() - return True - - async def claim_due_tasks( - self, - *, - now: datetime, - lease_owner: str, - lease_seconds: int, - limit: int, - ) -> list[dict[str, Any]]: - lease_expires_at = now + timedelta(seconds=lease_seconds) - stmt = ( - select(ScheduledTaskRow) - .where( - ScheduledTaskRow.next_run_at.is_not(None), - ScheduledTaskRow.next_run_at <= now, - or_( - and_( - ScheduledTaskRow.status == "enabled", - or_( - ScheduledTaskRow.lease_expires_at.is_(None), - ScheduledTaskRow.lease_expires_at < now, - ), - ), - # A task stuck in "running" with an expired lease means the - # claiming process died between claim and dispatch; it must - # stay reclaimable or the task is dead forever. - and_( - ScheduledTaskRow.status == "running", - ScheduledTaskRow.lease_expires_at.is_not(None), - ScheduledTaskRow.lease_expires_at < now, - ), - ), - ) - .order_by(ScheduledTaskRow.next_run_at.asc(), ScheduledTaskRow.id.asc()) - .limit(limit) - .with_for_update(skip_locked=True) - ) - async with self._sf() as session: - result = await session.execute(stmt) - rows = list(result.scalars()) - for row in rows: - row.lease_owner = lease_owner - row.lease_expires_at = lease_expires_at - row.status = "running" - row.updated_at = datetime.now(UTC) - await session.commit() - return [self._row_to_dict(row) for row in rows] - - async def update_after_launch( - self, - task_id: str, - *, - status: str, - next_run_at: datetime | None, - last_run_at: datetime | None, - last_run_id: str | None, - last_thread_id: str | None, - last_error: str | None, - increment_run_count: bool, - protect_terminal: bool = False, - ) -> None: - async with self._sf() as session: - row = await session.get(ScheduledTaskRow, task_id) - if row is None: - return - if protect_terminal and row.status in TERMINAL_TASK_STATUSES: - # A fast-failing run can reach handle_run_completion (which - # finalizes a `once` task) before this launch-path write - # commits; keep the hook's status/error and only record the - # launch bookkeeping. - pass - else: - row.status = status - row.last_error = last_error - row.next_run_at = next_run_at - row.last_run_at = last_run_at - row.last_run_id = last_run_id - row.last_thread_id = last_thread_id - if increment_run_count: - row.run_count += 1 - row.lease_owner = None - row.lease_expires_at = None - row.updated_at = datetime.now(UTC) - await session.commit() - - async def list_by_user_and_thread(self, user_id: str, thread_id: str) -> list[dict[str, Any]]: - stmt = ( - select(ScheduledTaskRow) - .where( - ScheduledTaskRow.user_id == user_id, - ScheduledTaskRow.thread_id == thread_id, - ) - .order_by(ScheduledTaskRow.created_at.desc(), ScheduledTaskRow.id.desc()) - ) - async with self._sf() as session: - result = await session.execute(stmt) - return [self._row_to_dict(row) for row in result.scalars()] - - async def cancel_stuck_once_tasks(self, *, error: str) -> int: - """Reconcile ``once`` tasks orphaned in ``running`` by a process crash. - - A launched ``once`` task stays ``running`` until the in-process - completion hook moves it to a terminal status; its lease was cleared at - launch, so the claim query's expired-lease reclaim branch never sees - it. After a crash the hook is gone and the task would be stuck forever. - Tasks still holding a lease are left alone — they were claimed but not - launched, and expired-lease reclaim recovers them safely. - """ - stmt = select(ScheduledTaskRow).where( - ScheduledTaskRow.schedule_type == "once", - ScheduledTaskRow.status == "running", - ScheduledTaskRow.lease_expires_at.is_(None), - ) - async with self._sf() as session: - result = await session.execute(stmt) - rows = list(result.scalars()) - now = datetime.now(UTC) - for row in rows: - row.status = "cancelled" - row.last_error = error - row.updated_at = now - await session.commit() - return len(rows) diff --git a/backend/packages/harness/deerflow/scheduler/__init__.py b/backend/packages/harness/deerflow/scheduler/__init__.py deleted file mode 100644 index b2262f3e8..000000000 --- a/backend/packages/harness/deerflow/scheduler/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -from .schedules import next_run_at, normalize_cron_expression, validate_timezone - -__all__ = ["next_run_at", "normalize_cron_expression", "validate_timezone"] diff --git a/backend/packages/harness/deerflow/scheduler/schedules.py b/backend/packages/harness/deerflow/scheduler/schedules.py deleted file mode 100644 index 8509c3523..000000000 --- a/backend/packages/harness/deerflow/scheduler/schedules.py +++ /dev/null @@ -1,55 +0,0 @@ -from __future__ import annotations - -from datetime import UTC, datetime -from zoneinfo import ZoneInfo, ZoneInfoNotFoundError - -from croniter import croniter - - -def validate_timezone(timezone_name: str) -> str: - try: - ZoneInfo(timezone_name) - except ZoneInfoNotFoundError as exc: - raise ValueError(f"Unknown timezone: {timezone_name}") from exc - return timezone_name - - -def normalize_cron_expression(expr: str) -> str: - parts = [part for part in expr.split() if part] - if len(parts) != 5: - raise ValueError("Cron expression must contain exactly 5 fields") - return " ".join(parts) - - -def next_run_at( - schedule_type: str, - schedule_spec: dict[str, object], - timezone_name: str, - *, - now: datetime, -) -> datetime | None: - validate_timezone(timezone_name) - if now.tzinfo is None: - now = now.replace(tzinfo=UTC) - - if schedule_type == "once": - run_at_raw = schedule_spec.get("run_at") - if not isinstance(run_at_raw, str): - raise ValueError("once schedule requires run_at") - run_at = datetime.fromisoformat(run_at_raw) - if run_at.tzinfo is None: - # A naive run_at means "wall-clock time in the task's declared - # timezone", matching how cron schedules interpret it. - run_at = run_at.replace(tzinfo=ZoneInfo(timezone_name)) - return run_at if run_at > now else None - - if schedule_type == "cron": - cron_expr = normalize_cron_expression(str(schedule_spec.get("cron", ""))) - zone = ZoneInfo(timezone_name) - local_now = now.astimezone(zone) - next_local = croniter(cron_expr, local_now).get_next(datetime) - if next_local.tzinfo is None: - next_local = next_local.replace(tzinfo=zone) - return next_local.astimezone(UTC) - - raise ValueError(f"Unsupported schedule_type: {schedule_type}") diff --git a/backend/tests/schedule_fakes.py b/backend/tests/schedule_fakes.py new file mode 100644 index 000000000..4e7fae5f3 --- /dev/null +++ b/backend/tests/schedule_fakes.py @@ -0,0 +1,293 @@ +"""Shared zero-IO doubles for the schedule context. + +Lives in its own module rather than inside a test file so the port contract +suite and the service tests can both import it as an ordinary module, without +relying on the tests directory happening to be on ``sys.path``. + +**Concurrency is explicitly out of scope.** These doubles model the +single-threaded semantics of each port -- which rows `claim_due` selects, that +`add` refuses a second active run -- and provide no atomicity whatsoever. A +green run here says nothing about two dispatchers racing; that is covered +against a real database in ``test_schedule_dispatch_race.py``. Do not +read a passing contract suite as licence to run more than one scheduler. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field, replace +from datetime import datetime, timedelta + +from deerflow.domain.schedule.exceptions import ActiveRunConflictError +from deerflow.domain.schedule.model import ( + ACTIVE_RUN_STATUSES, + TERMINAL_RUN_STATUSES, + TERMINAL_TASK_STATUSES, + RunStatus, + ScheduledRun, + ScheduledTask, + ScheduleType, + TaskStatus, +) +from deerflow.domain.schedule.ports import LaunchedRun + + +@dataclass +class _TaskRow: + """One stored task: the aggregate plus the columns the domain has no + business knowing about. + + The lease lives here rather than on ``ScheduledTask`` for the same reason + it lives in the table and not in the domain -- it is a storage-level + concurrency control, not a business fact about the task. + """ + + task: ScheduledTask + lease_owner: str | None = None + lease_expires_at: datetime | None = None + + +class InMemoryScheduledTaskRepository: + """ScheduledTaskRepository double backed by a dict.""" + + def __init__(self) -> None: + self._rows: dict[str, _TaskRow] = {} + + # -- inspection helpers (tests only, not part of the port) ----------- + + def lease_of(self, task_id: str) -> tuple[str | None, datetime | None]: + row = self._rows[task_id] + return row.lease_owner, row.lease_expires_at + + def seed(self, task: ScheduledTask, *, lease_owner: str | None = None, lease_expires_at: datetime | None = None) -> ScheduledTask: + """Install a task directly, optionally already claimed.""" + self._rows[task.task_id] = _TaskRow(task=task, lease_owner=lease_owner, lease_expires_at=lease_expires_at) + return task + + # -- port ------------------------------------------------------------ + + async def add(self, task: ScheduledTask) -> ScheduledTask: + self._rows[task.task_id] = _TaskRow(task=task) + return task + + async def get(self, task_id: str, *, user_id: str) -> ScheduledTask | None: + row = self._rows.get(task_id) + if row is None or row.task.user_id != user_id: + return None + return row.task + + async def list_by_user(self, user_id: str) -> list[ScheduledTask]: + rows = [row.task for row in self._rows.values() if row.task.user_id == user_id] + return sorted(rows, key=lambda t: (t.created_at, t.task_id), reverse=True) + + async def list_by_user_and_thread(self, user_id: str, thread_id: str) -> list[ScheduledTask]: + tasks = await self.list_by_user(user_id) + return [task for task in tasks if task.thread_id == thread_id] + + async def save(self, task: ScheduledTask) -> ScheduledTask | None: + row = self._rows.get(task.task_id) + if row is None or row.task.user_id != task.user_id: + return None + row.task = task + return task + + async def delete(self, task_id: str, *, user_id: str) -> bool: + row = self._rows.get(task_id) + if row is None or row.task.user_id != user_id: + return False + del self._rows[task_id] + return True + + async def claim_due(self, *, now: datetime, lease_seconds: int, limit: int) -> list[ScheduledTask]: + def claimable(row: _TaskRow) -> bool: + task = row.task + if task.next_run_at is None or task.next_run_at > now: + return False + expired = row.lease_expires_at is not None and row.lease_expires_at < now + if task.status is TaskStatus.ENABLED: + return row.lease_expires_at is None or expired + # Stuck mid-dispatch: the claimer died between claim and launch. + return task.status is TaskStatus.RUNNING and expired + + due = sorted( + (row for row in self._rows.values() if claimable(row)), + key=lambda row: (row.task.next_run_at, row.task.task_id), + )[:limit] + + claimed = [] + for row in due: + # Stands in for whatever identity a real adapter records. + row.lease_owner = "fake-worker" + row.lease_expires_at = now + timedelta(seconds=lease_seconds) + row.task = replace(row.task, status=TaskStatus.RUNNING) + claimed.append(row.task) + return claimed + + async def record_launch( + self, + task_id: str, + *, + status: TaskStatus, + next_run_at: datetime | None, + last_run_at: datetime | None, + last_run_id: str | None, + last_thread_id: str | None, + last_error: str | None, + increment_run_count: bool, + protect_terminal: bool = False, + ) -> None: + row = self._rows.get(task_id) + if row is None: + return + task = row.task + if not (protect_terminal and task.status in TERMINAL_TASK_STATUSES): + task = replace(task, status=status, last_error=last_error) + task = replace( + task, + next_run_at=next_run_at, + last_run_at=last_run_at, + last_run_id=last_run_id, + last_thread_id=last_thread_id, + ) + if increment_run_count: + task = replace(task, run_count=task.run_count + 1) + row.task = task + row.lease_owner = None + row.lease_expires_at = None + + async def record_completion( + self, + task_id: str, + *, + user_id: str, + status: TaskStatus | None, + error: str | None, + ) -> None: + row = self._rows.get(task_id) + if row is None or row.task.user_id != user_id: + return + # Only the verdict; every scheduling field belongs to record_launch. + row.task = replace(row.task, last_error=error) + if status is not None: + row.task = replace(row.task, status=status) + + async def cancel_stuck_once_tasks(self, *, error: str) -> int: + cancelled = 0 + for row in self._rows.values(): + stuck = row.task.status is TaskStatus.RUNNING and row.task.schedule.schedule_type is ScheduleType.ONCE and row.lease_expires_at is None + if stuck: + row.task = replace(row.task, status=TaskStatus.CANCELLED, last_error=error) + cancelled += 1 + return cancelled + + +class InMemoryScheduledRunRepository: + """ScheduledRunRepository double that also models the active-slot rule. + + ``add`` refusing a second active record for one task is not an + implementation detail to be skipped here: the service collapses that + rejection to the same outcome as its own fast path, and a double that never + raises would let that collapse go untested. + """ + + def __init__(self) -> None: + self._rows: dict[str, ScheduledRun] = {} + + def all_runs(self) -> list[ScheduledRun]: + """Inspection helper (tests only, not part of the port).""" + return list(self._rows.values()) + + async def add(self, run: ScheduledRun) -> ScheduledRun: + if run.is_active and any(other.task_id == run.task_id and other.is_active for other in self._rows.values()): + raise ActiveRunConflictError(f"scheduled task {run.task_id!r} already has an active run") + self._rows[run.record_id] = run + return run + + async def list_by_task(self, task_id: str, *, limit: int = 50, offset: int = 0) -> list[ScheduledRun]: + rows = [run for run in self._rows.values() if run.task_id == task_id] + rows.sort(key=lambda run: (run.created_at, run.record_id), reverse=True) + return rows[offset : offset + limit] + + async def count_active(self) -> int: + return sum(1 for run in self._rows.values() if run.is_active) + + async def has_active(self, task_id: str) -> bool: + return any(run.task_id == task_id and run.is_active for run in self._rows.values()) + + async def update_status( + self, + record_id: str, + *, + status: RunStatus, + run_id: str | None = None, + error: str | None = None, + started_at: datetime | None = None, + finished_at: datetime | None = None, + protect_terminal: bool = False, + ) -> None: + run = self._rows.get(record_id) + if run is None: + return + if protect_terminal and run.status in TERMINAL_RUN_STATUSES: + # Keep the terminal verdict; only backfill what it could not know. + if run.run_id is None and run_id is not None: + run = replace(run, run_id=run_id) + if run.started_at is None and started_at is not None: + run = replace(run, started_at=started_at) + self._rows[record_id] = run + return + run = replace(run, status=status, run_id=run_id, error=error) + if started_at is not None: + run = replace(run, started_at=started_at) + if finished_at is not None: + run = replace(run, finished_at=finished_at) + self._rows[record_id] = run + + async def mark_stale_active(self, *, error: str) -> int: + stale = [run for run in self._rows.values() if run.status in ACTIVE_RUN_STATUSES] + for run in stale: + self._rows[run.record_id] = replace(run, status=RunStatus.INTERRUPTED, error=error) + return len(stale) + + +@dataclass +class FakeRunLauncher: + """RunLauncher double. + + `fail_with` makes every launch raise instead -- set it to a ThreadBusyError + or a LaunchFailedError to drive the two branches the domain distinguishes. + """ + + fail_with: Exception | None = None + calls: list[dict] = field(default_factory=list) + + async def launch( + self, + *, + thread_id: str, + assistant_id: str | None, + prompt: str, + owner_user_id: str | None, + metadata: dict[str, str], + ) -> LaunchedRun: + self.calls.append( + { + "thread_id": thread_id, + "assistant_id": assistant_id, + "prompt": prompt, + "owner_user_id": owner_user_id, + "metadata": metadata, + } + ) + if self.fail_with is not None: + raise self.fail_with + return LaunchedRun(run_id=f"run-{len(self.calls)}", thread_id=thread_id) + + +class FakeThreadLookup: + """ThreadLookup double backed by a thread_id -> owner mapping.""" + + def __init__(self, threads: dict[str, str] | None = None) -> None: + self._threads = dict(threads or {}) + + async def exists_for_user(self, thread_id: str, user_id: str) -> bool: + return self._threads.get(thread_id) == user_id diff --git a/backend/tests/test_composition.py b/backend/tests/test_composition.py new file mode 100644 index 000000000..f26df2f11 --- /dev/null +++ b/backend/tests/test_composition.py @@ -0,0 +1,146 @@ +"""Tests for the composition root. + +The point of extracting `build_domain_services` from the lifespan is that the +rules below become assertions. Before, "a memory backend means no service, +and the routes answer 503" was a comment inside a 180-line startup function +that no test could reach without booting the whole application. +""" + +from __future__ import annotations + +import pytest + +from app.composition import DomainServices, build_domain_services, build_run_completion_hook, build_schedule_policy +from deerflow.config.scheduler_config import SchedulerConfig +from deerflow.domain.schedule.model import SchedulePolicy + + +class _StubThreadStore: + async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool: + return True + + +async def _launch_run(**kwargs): + return {"run_id": "run-1", "thread_id": kwargs.get("thread_id", "thread-1")} + + +def _build(session_factory, scheduler_config=None) -> DomainServices: + return build_domain_services( + session_factory=session_factory, + thread_store=_StubThreadStore(), + launch_run=_launch_run, + scheduler_config=scheduler_config or SchedulerConfig(), + ) + + +class TestMemoryBackend: + """`session_factory is None` is how `database.backend: memory` presents.""" + + def test_no_session_factory_yields_no_services(self): + assert _build(None).schedule is None + + def test_it_does_not_degrade_to_an_in_memory_implementation(self): + """Refusing is the intended behaviour: a scheduled task that silently + vanishes on restart is worse than one the API declines to accept.""" + assert _build(None).schedule is None + + +class TestSqlBackend: + def test_a_session_factory_yields_the_service(self): + """A fake sessionmaker is enough -- wiring must not touch the database, + which is what makes this assertable without a live engine.""" + assert _build(object()).schedule is not None + + +class TestSchedulePolicy: + def test_every_threshold_comes_from_the_operator_config(self): + policy = build_schedule_policy( + SchedulerConfig( + min_once_delay_seconds=30, + max_concurrent_runs=7, + lease_seconds=90, + ) + ) + assert policy == SchedulePolicy( + min_once_delay_seconds=30, + max_concurrent_runs=7, + lease_seconds=90, + ) + + def test_the_domain_defaults_are_not_what_production_gets(self): + """The domain's defaults are permissive so that "nobody configured a + policy" invents no business constraint. Production must not inherit + them by accident -- the config's own defaults are the real values.""" + from_config = build_schedule_policy(SchedulerConfig()) + assert from_config != SchedulePolicy() + assert from_config.min_once_delay_seconds == SchedulerConfig().min_once_delay_seconds + + @pytest.mark.parametrize( + ("field", "value"), + [ + ("min_once_delay_seconds", 45), + ("max_concurrent_runs", 5), + ("lease_seconds", 300), + ], + ) + def test_each_field_is_mapped_from_its_own_config_key(self, field, value): + """Guards against two thresholds being wired from one key -- a + transposition the type checker cannot see, since all three are ints.""" + policy = build_schedule_policy(SchedulerConfig(**{field: value})) + assert getattr(policy, field) == value + + def test_poll_interval_is_not_part_of_the_policy(self): + """How often to look is the poller's business, not a rule any task is + subject to.""" + assert not hasattr(SchedulePolicy(), "poll_interval_seconds") + + +class TestEveryPortGetsTheRightAdapter: + """Deliberately white-box: verifying which adapter landed in which slot is + the entire job of a composition root, and all four are keyword arguments + of compatible shape, so a transposition type-checks cleanly and would only + surface as a production error. + """ + + def test_each_schedule_port_is_filled_with_its_own_adapter(self): + from app.adapters.schedule.run_launcher import GatewayRunLauncher + from app.adapters.schedule.scheduled_run_repository import SqlScheduledRunRepository + from app.adapters.schedule.scheduled_task_repository import SqlScheduledTaskRepository + from app.adapters.schedule.thread_lookup import ThreadStoreThreadLookup + + service = _build(object()).schedule + assert isinstance(service._tasks, SqlScheduledTaskRepository) + assert isinstance(service._runs, SqlScheduledRunRepository) + assert isinstance(service._launcher, GatewayRunLauncher) + assert isinstance(service._threads, ThreadStoreThreadLookup) + + def test_the_policy_reaches_the_service(self): + service = _build(object(), SchedulerConfig(max_concurrent_runs=9)).schedule + assert service._policy.max_concurrent_runs == 9 + + +class TestRunCompletionHook: + """The inbound half of the wiring. + + What the hook *does* with a run is asserted in + `test_schedule_run_completion.py`; all that is left here is the assembly + decision, which is this module's whole job. + """ + + def test_no_service_installs_no_hook(self): + """`None` rather than a hook that always declines: with no service + there is nothing for the runtime to call, and saying so lets it skip + the callback entirely.""" + assert build_run_completion_hook(None) is None + + def test_a_service_is_wrapped_in_the_inbound_adapter(self): + from app.adapters.schedule.run_completion import ScheduleRunCompletionListener + + hook = build_run_completion_hook(_build(object()).schedule) + assert isinstance(hook, ScheduleRunCompletionListener) + + def test_the_hook_is_bound_to_the_service_it_was_given(self): + """Deliberately white-box, for the same reason as the port assertions + above: a hook wired to the wrong service type-checks cleanly.""" + service = _build(object()).schedule + assert build_run_completion_hook(service)._service is service diff --git a/backend/tests/test_gateway_run_drain_shutdown.py b/backend/tests/test_gateway_run_drain_shutdown.py index 528997069..f4b2f8959 100644 --- a/backend/tests/test_gateway_run_drain_shutdown.py +++ b/backend/tests/test_gateway_run_drain_shutdown.py @@ -30,6 +30,7 @@ from typing import Annotated, TypedDict import pytest from langgraph.checkpoint.memory import InMemorySaver +from deerflow.config.scheduler_config import SchedulerConfig from deerflow.runtime import RunManager, RunStatus @@ -196,7 +197,7 @@ async def test_langgraph_runtime_drains_runs_before_closing_checkpointer(monkeyp monkeypatch.setattr(RunManager, "shutdown", spy_shutdown, raising=False) app = FastAPI() - startup_config = SimpleNamespace(database=SimpleNamespace(backend="memory", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=None) + startup_config = SimpleNamespace(database=SimpleNamespace(backend="memory", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=None, scheduler=SchedulerConfig()) async with langgraph_runtime(app, startup_config): pass diff --git a/backend/tests/test_gateway_run_recovery.py b/backend/tests/test_gateway_run_recovery.py index 9f6b05c7e..6374b69c4 100644 --- a/backend/tests/test_gateway_run_recovery.py +++ b/backend/tests/test_gateway_run_recovery.py @@ -14,6 +14,7 @@ from fastapi import FastAPI import deerflow.runtime as runtime_module from app.gateway import deps as gateway_deps from deerflow.config.run_ownership_config import RunOwnershipConfig +from deerflow.config.scheduler_config import SchedulerConfig from deerflow.persistence import engine as engine_module from deerflow.persistence import thread_meta as thread_meta_module from deerflow.runtime import END_SENTINEL, MemoryStreamBridge, RunManager @@ -224,6 +225,7 @@ async def test_sqlite_runtime_reconciles_orphaned_runs_on_startup(monkeypatch): database=SimpleNamespace(backend="sqlite", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=SimpleNamespace(backend="memory"), stream_bridge=SimpleNamespace(recovered_stream_cleanup_delay_seconds=60.0), + scheduler=SchedulerConfig(), ) thread_store = _FakeThreadStore() stream_bridge = _FakeStreamBridge(existing_streams={"run-1"}) @@ -269,6 +271,7 @@ async def test_sqlite_runtime_does_not_mark_thread_error_when_newer_run_is_succe database=SimpleNamespace(backend="sqlite", checkpoint_channel_mode="full", checkpoint_delta=SimpleNamespace(snapshot_frequency=10)), run_events=SimpleNamespace(backend="memory"), stream_bridge=SimpleNamespace(recovered_stream_cleanup_delay_seconds=60.0), + scheduler=SchedulerConfig(), ) thread_store = _FakeThreadStore() stream_bridge = _FakeStreamBridge(existing_streams={"old-running"}) diff --git a/backend/tests/test_gateway_services.py b/backend/tests/test_gateway_services.py index 842546991..bcca4bf73 100644 --- a/backend/tests/test_gateway_services.py +++ b/backend/tests/test_gateway_services.py @@ -1407,7 +1407,7 @@ def _make_start_run_persistence_context(): run_events_config=None, thread_store=thread_store, checkpoint_channel_mode="full", - scheduled_task_service=None, + run_completion_hook=None, ) request = SimpleNamespace( headers={}, diff --git a/backend/tests/test_schedule_corrupt_rows.py b/backend/tests/test_schedule_corrupt_rows.py new file mode 100644 index 000000000..d3335982f --- /dev/null +++ b/backend/tests/test_schedule_corrupt_rows.py @@ -0,0 +1,126 @@ +"""A corrupt stored schedule is an operator problem, not a client error. + +``SqlScheduledTaskRepository._to_domain`` rebuilds the aggregate on every +read, so a row whose stored schedule no longer parses surfaces at read time. +It used to surface as ``InvalidScheduleError`` -- the same error the aggregate +raises for a *client-submitted* schedule, which the router maps to 422. That +double duty told the client "your request is wrong" about a request that was +perfectly fine, and made the row unrepairable over HTTP: PATCH reads the task +before writing, so the fix path 422'd too. + +``CorruptStoredScheduleError`` splits the vocabulary: it is raised only by the +persistence adapter, and it is deliberately absent from the router's status +table so it falls through to the unclassified-500 branch -- a server-side +fault reported as one. + +SQL-only: the in-memory double stores whole aggregates and cannot hold a +corrupt row, which is exactly why this file exists beside the contract suite +rather than inside it. +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from datetime import UTC, datetime + +import pytest +import pytest_asyncio + +from app.adapters.schedule.scheduled_task_repository import SqlScheduledTaskRepository +from app.gateway.routers.schedule.router import _STATUS_BY_ERROR +from deerflow.config.database_config import DatabaseConfig +from deerflow.domain.schedule.exceptions import CorruptStoredScheduleError, InvalidScheduleError +from deerflow.domain.schedule.model import ScheduledTask, SchedulePolicy, ScheduleSpec +from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config +from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow + +pytestmark = pytest.mark.asyncio + +NOW = datetime(2026, 7, 27, 12, 0, tzinfo=UTC) + + +@pytest_asyncio.fixture +async def repo(tmp_path) -> AsyncIterator[SqlScheduledTaskRepository]: + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + yield SqlScheduledTaskRepository(sf) + finally: + await close_engine() + + +async def _add_task(repo: SqlScheduledTaskRepository, *, title: str = "healthy") -> ScheduledTask: + return await repo.add( + ScheduledTask.create( + user_id="user-1", + title=title, + prompt="p", + schedule=ScheduleSpec.cron_schedule("0 9 * * *", "UTC"), + context_mode="fresh_thread_per_run", + thread_id=None, + now=NOW, + policy=SchedulePolicy(), + ) + ) + + +async def _corrupt(task_id: str, **columns) -> None: + """Damage a stored row directly -- the exact shape a bug or a manual edit + would leave behind, unreachable through the port.""" + sf = get_session_factory() + assert sf is not None + async with sf() as session: + row = await session.get(ScheduledTaskRow, task_id) + assert row is not None + for name, value in columns.items(): + setattr(row, name, value) + await session.commit() + + +class TestSingleRowReads: + async def test_a_corrupt_schedule_raises_the_dedicated_error(self, repo): + task = await _add_task(repo) + await _corrupt(task.task_id, schedule_spec={}) + + with pytest.raises(CorruptStoredScheduleError): + await repo.get(task.task_id, user_id="user-1") + + async def test_the_dedicated_error_is_not_the_client_facing_one(self, repo): + """The router maps InvalidScheduleError to 422; a corrupt row must not + ride that mapping.""" + task = await _add_task(repo) + await _corrupt(task.task_id, schedule_spec={}) + + with pytest.raises(CorruptStoredScheduleError) as exc_info: + await repo.get(task.task_id, user_id="user-1") + assert not isinstance(exc_info.value, InvalidScheduleError) + + async def test_a_corrupt_context_mode_is_reported_the_same_way(self, repo): + """Enum rebuild failures are the same fault as spec parse failures: a + raw ValueError out of the adapter would be a technical exception + crossing the boundary untranslated.""" + task = await _add_task(repo) + await _corrupt(task.task_id, context_mode="not-a-mode") + + with pytest.raises(CorruptStoredScheduleError): + await repo.get(task.task_id, user_id="user-1") + + +class TestListReads: + async def test_a_corrupt_row_does_not_take_down_the_listing(self, repo): + healthy = await _add_task(repo, title="healthy") + broken = await _add_task(repo, title="broken") + await _corrupt(broken.task_id, schedule_spec={}) + + listed = await repo.list_by_user("user-1") + + assert [t.task_id for t in listed] == [healthy.task_id] + + +class TestRouterMapping: + def test_the_corrupt_row_error_is_unclassified_on_purpose(self): + """Absent from the status table means the router's fallthrough turns + it into a 500 -- a new protocol decision would have to add it here + deliberately.""" + assert CorruptStoredScheduleError not in _STATUS_BY_ERROR diff --git a/backend/tests/test_schedule_dispatch_race.py b/backend/tests/test_schedule_dispatch_race.py new file mode 100644 index 000000000..d502b3978 --- /dev/null +++ b/backend/tests/test_schedule_dispatch_race.py @@ -0,0 +1,223 @@ +"""Concurrency regression tests for the scheduled-task dispatch TOCTOU. + +``ScheduleService.dispatch_task`` guards "at most one active run per task +when overlap_policy=skip" with a non-atomic ``has_active`` fast path followed +by a separate queued-record insert. Two concurrent dispatches (double-click, +client retry, or a manual trigger racing the poller) can both pass the check +and both launch. The database is the atomic arbiter via the partial unique +index ``uq_scheduled_task_run_active`` (``task_id WHERE status IN +('queued','running')``); the losing insert is translated to +``ActiveRunConflictError`` and collapsed to the same outcome as the fast +path. + +These tests drive the REAL ``SqlScheduledRunRepository`` + +``SqlScheduledTaskRepository`` + ``ScheduleService`` against a real +file-backed sqlite database (so the index is actually enforced), with a fake +launcher that only records launches. The contract suite deliberately does not +own this: its doubles provide no atomicity, so a green contract run says +nothing about two dispatchers racing -- this file is where that is proven. +""" + +from __future__ import annotations + +import asyncio +from datetime import UTC, datetime + +import pytest +from schedule_fakes import FakeThreadLookup + +from app.adapters.schedule.scheduled_run_repository import SqlScheduledRunRepository +from app.adapters.schedule.scheduled_task_repository import SqlScheduledTaskRepository +from deerflow.config.database_config import DatabaseConfig +from deerflow.domain.schedule.exceptions import ActiveRunConflictError +from deerflow.domain.schedule.model import DispatchOutcome, RunStatus, ScheduledRun, ScheduledTask, SchedulePolicy, ScheduleSpec, TriggerKind +from deerflow.domain.schedule.ports import LaunchedRun +from deerflow.domain.schedule.service import ScheduleService +from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config + +pytestmark = pytest.mark.asyncio + +NOW = datetime(2026, 7, 27, 12, 0, tzinfo=UTC) + + +class _BarrierRunRepo(SqlScheduledRunRepository): + """Real repository that only releases both dispatchers past ``has_active`` + once both have read it, so their ``add()`` calls genuinely race for the + task's single active slot -- a deterministic reproduction of the + check-then-insert TOCTOU.""" + + def __init__(self, session_factory, barrier: asyncio.Barrier | None) -> None: + super().__init__(session_factory) + self._barrier = barrier + + async def has_active(self, task_id: str) -> bool: + result = await super().has_active(task_id) + if self._barrier is not None: + await self._barrier.wait() + return result + + +class _RecordingLauncher: + """Launch double that yields first, so a truly-concurrent sibling can + interleave before the launch is recorded.""" + + def __init__(self) -> None: + self.calls: list[dict] = [] + + async def launch(self, *, thread_id, assistant_id, prompt, owner_user_id, metadata) -> LaunchedRun: + await asyncio.sleep(0) + self.calls.append({"thread_id": thread_id, "metadata": metadata}) + return LaunchedRun(run_id=f"run-{len(self.calls)}", thread_id=thread_id) + + +def _make_service(tasks, runs, launcher) -> ScheduleService: + return ScheduleService( + tasks=tasks, + runs=runs, + launcher=launcher, + threads=FakeThreadLookup(), + policy=SchedulePolicy(min_once_delay_seconds=60, max_concurrent_runs=10, lease_seconds=120), + ) + + +async def _seed_task(tasks: SqlScheduledTaskRepository, title: str) -> ScheduledTask: + # fresh_thread_per_run: every dispatch gets a NEW thread_id, so #4003's + # per-thread uq_runs_thread_active can never fire for two dispatches of the + # same task -- this is precisely the gap the per-task index closes. + return await tasks.add( + ScheduledTask.create( + user_id="user-1", + title=title, + prompt="do the thing", + schedule=ScheduleSpec.cron_schedule("*/5 * * * *", "UTC"), + context_mode="fresh_thread_per_run", + thread_id=None, + now=NOW, + policy=SchedulePolicy(), + ) + ) + + +async def _active_run_count(runs: SqlScheduledRunRepository, task_id: str) -> int: + rows = await runs.list_by_task(task_id, limit=100) + return sum(1 for row in rows if row.is_active) + + +async def test_two_concurrent_manual_dispatches_launch_exactly_once(tmp_path): + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + tasks = SqlScheduledTaskRepository(sf) + runs = _BarrierRunRepo(sf, asyncio.Barrier(2)) + launcher = _RecordingLauncher() + service = _make_service(tasks, runs, launcher) + task = await _seed_task(tasks, "task-race-manual") + + results = await asyncio.gather( + service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL), + service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL), + ) + + outcomes = sorted((result.outcome for result in results), key=str) + # Exactly one wins the active slot; the loser is a 409-style conflict. + assert outcomes == [DispatchOutcome.CONFLICT, DispatchOutcome.LAUNCHED], outcomes + assert len(launcher.calls) == 1, launcher.calls + assert await _active_run_count(runs, task.task_id) == 1 + # The manual loser records no run-history row (nothing was scheduled). + conflict = next(r for r in results if r.outcome is DispatchOutcome.CONFLICT) + assert conflict.record_id is None + finally: + await close_engine() + + +async def test_scheduled_and_manual_dispatch_launch_exactly_once(tmp_path): + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + tasks = SqlScheduledTaskRepository(sf) + runs = _BarrierRunRepo(sf, asyncio.Barrier(2)) + launcher = _RecordingLauncher() + service = _make_service(tasks, runs, launcher) + task = await _seed_task(tasks, "task-race-mixed") + + results = await asyncio.gather( + service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED), + service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL), + ) + + outcomes = [result.outcome for result in results] + # Whichever won launched; the loser is conflict (manual) or skipped + # (scheduled). Which one wins is timing-dependent, but exactly one runs. + assert outcomes.count(DispatchOutcome.LAUNCHED) == 1, outcomes + assert set(outcomes) <= {DispatchOutcome.LAUNCHED, DispatchOutcome.CONFLICT, DispatchOutcome.SKIPPED}, outcomes + assert len(launcher.calls) == 1, launcher.calls + assert await _active_run_count(runs, task.task_id) == 1 + finally: + await close_engine() + + +async def test_natural_timing_concurrent_dispatch_launches_exactly_once(tmp_path): + # No barrier: exercise the fix under the same natural interleaving that + # reproduced the bug (5/5 both-launch before the index). The fix must hold + # whether the second dispatch is caught by the has_active fast path or by + # the index-violation path. + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + tasks = SqlScheduledTaskRepository(sf) + runs = SqlScheduledRunRepository(sf) + for i in range(5): + launcher = _RecordingLauncher() + service = _make_service(tasks, runs, launcher) + task = await _seed_task(tasks, f"task-natural-{i}") + + results = await asyncio.gather( + service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL), + service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL), + ) + + outcomes = [result.outcome for result in results] + assert outcomes.count(DispatchOutcome.LAUNCHED) == 1, (i, outcomes) + assert len(launcher.calls) == 1, (i, launcher.calls) + assert await _active_run_count(runs, task.task_id) == 1, i + finally: + await close_engine() + + +async def test_partial_unique_index_enforces_one_active_run_per_task(tmp_path): + # Focused repository-level test of the index semantics + the typed conflict. + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + runs = SqlScheduledRunRepository(sf) + now = datetime(2026, 7, 2, 1, 0, tzinfo=UTC) + + first = ScheduledRun.queued(task_id="t1", thread_id="th1", scheduled_for=now, trigger=TriggerKind.SCHEDULED) + await runs.add(first) + + # queued -> running is a same-row UPDATE: keeps the one active slot, no + # violation (this is the normal launch transition). + await runs.update_status(first.record_id, status=RunStatus.RUNNING, run_id="run-1", started_at=now) + assert await runs.has_active("t1") is True + + # A second active insert for the same task is a domain conflict. + with pytest.raises(ActiveRunConflictError): + await runs.add(ScheduledRun.queued(task_id="t1", thread_id="th2", scheduled_for=now, trigger=TriggerKind.MANUAL)) + + # Terminal-status rows for the same task are outside the index predicate. + await runs.add(ScheduledRun.skipped_tombstone(task_id="t1", thread_id="th3", scheduled_for=now, trigger=TriggerKind.SCHEDULED)) + + # A different task's active row is independent. + await runs.add(ScheduledRun.queued(task_id="t2", thread_id="th4", scheduled_for=now, trigger=TriggerKind.SCHEDULED)) + + # Finishing the active run frees the slot; a fresh active row is allowed. + await runs.update_status(first.record_id, status=RunStatus.SUCCESS, run_id="run-1", finished_at=now) + assert await runs.has_active("t1") is False + await runs.add(ScheduledRun.queued(task_id="t1", thread_id="th5", scheduled_for=now, trigger=TriggerKind.SCHEDULED)) + assert await runs.has_active("t1") is True + finally: + await close_engine() diff --git a/backend/tests/test_schedule_domain.py b/backend/tests/test_schedule_domain.py new file mode 100644 index 000000000..302b6eed3 --- /dev/null +++ b/backend/tests/test_schedule_domain.py @@ -0,0 +1,627 @@ +"""Domain tests for the schedule bounded context. + +Deliberately synchronous and dependency-free: no pytest-asyncio, no fakes, no +IO. Everything under `deerflow.domain.schedule.model` is pure, and this file is +the proof — if a rule here ever needs a stub, the rule has leaked out of the +inner ring. + +The three `status_after_*` truth tables (TestStatusAfter*) are the reason this +commit exists: those rules used to be static methods on +`app/scheduler/service.py` with no direct coverage at all, reachable only +through the service's integration tests. +""" + +from __future__ import annotations + +import uuid +from dataclasses import replace +from datetime import UTC, datetime, timedelta + +import pytest + +from deerflow.domain.schedule.exceptions import InvalidContextModeError, InvalidScheduleError, TaskNotMutableError +from deerflow.domain.schedule.model import ( + ContextMode, + RunStatus, + ScheduledRun, + ScheduledTask, + SchedulePolicy, + ScheduleSpec, + ScheduleType, + TaskStatus, + TriggerKind, +) + +NOW = datetime(2026, 7, 27, 12, 0, tzinfo=UTC) +NO_DELAY = SchedulePolicy(min_once_delay_seconds=0) +MIN_60 = SchedulePolicy(min_once_delay_seconds=60) + + +def cron_spec(expr: str = "0 9 * * *", timezone: str = "UTC") -> ScheduleSpec: + return ScheduleSpec.cron_schedule(expr, timezone) + + +def once_spec(*, after_seconds: int = 3600, timezone: str = "UTC") -> ScheduleSpec: + return ScheduleSpec.once_at(NOW + timedelta(seconds=after_seconds), timezone) + + +def make_task( + *, + schedule: ScheduleSpec | None = None, + status: TaskStatus = TaskStatus.ENABLED, + context_mode: ContextMode = ContextMode.FRESH_THREAD_PER_RUN, + thread_id: str | None = None, +) -> ScheduledTask: + return ScheduledTask( + task_id="task-1", + user_id="user-1", + title="Daily summary", + prompt="Summarize", + schedule=schedule if schedule is not None else cron_spec(), + status=status, + context_mode=context_mode, + thread_id=thread_id, + ) + + +# ---------------------------------------------------------------- A. ScheduleSpec invariants + + +class TestScheduleSpecInvariants: + def test_unknown_timezone_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="Unknown timezone"): + ScheduleSpec.cron_schedule("0 9 * * *", "Mars/Olympus_Mons") + + def test_cron_without_expression_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="requires schedule_spec"): + ScheduleSpec(ScheduleType.CRON, "UTC") + + @pytest.mark.parametrize("expr", ["0 9 * *", "0 9 * * * *", "0"]) + def test_cron_must_have_exactly_five_fields(self, expr): + with pytest.raises(InvalidScheduleError, match="exactly 5 fields"): + ScheduleSpec.cron_schedule(expr, "UTC") + + def test_once_without_run_at_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="requires run_at"): + ScheduleSpec(ScheduleType.ONCE, "UTC") + + def test_direct_construction_cannot_bypass_validation(self): + """A frozen dataclass is still constructible field-by-field, so the + rules have to live in __post_init__ rather than in the factories.""" + with pytest.raises(InvalidScheduleError, match="exactly 5 fields"): + ScheduleSpec(ScheduleType.CRON, "UTC", cron="not-a-cron") + + def test_cron_whitespace_is_normalized_on_construction(self): + assert ScheduleSpec(ScheduleType.CRON, "UTC", cron=" 0 9 * * * ").cron == "0 9 * * *" + + def test_naive_run_at_is_localized_to_the_schedule_timezone(self): + spec = ScheduleSpec.once_at(datetime(2026, 8, 1, 9, 0), "Asia/Shanghai") # noqa: DTZ001 -- naive input is the subject + assert spec.run_at.utcoffset() == timedelta(hours=8) + assert spec.run_at.astimezone(UTC) == datetime(2026, 8, 1, 1, 0, tzinfo=UTC) + + def test_aware_run_at_is_left_alone(self): + """Covers the shape the frontend submits (`zonedLocalToUtcIso`, trailing + Z): already aware, so localization must leave it alone rather than + reinterpreting it in the task's timezone. Parsing that spelling is + `from_primitives`' job and is tested there.""" + run_at = datetime(2026, 8, 1, 9, 0, tzinfo=UTC) + assert ScheduleSpec.once_at(run_at, "Asia/Shanghai").run_at == run_at + + +# ---------------------------------------------------------------- A2. from_primitives + + +class TestFromPrimitives: + """The one construction path that starts from untrusted strings. + + Both boundaries -- the HTTP body and the stored JSON column -- arrive as a + schedule type plus a loose mapping, so the rule for turning that into a + value object lives here rather than being written out once per adapter. + Each adapter is left with the part that is genuinely its own: which keys + its format uses. + + Values are type-checked rather than assumed. The signature says `str | + None` because that is the contract, but both callers read from data a + client can influence, so a non-string has to be rejected here rather than + reaching `datetime.fromisoformat` or the cron parser. + """ + + def test_builds_a_cron_schedule(self): + spec = ScheduleSpec.from_primitives("cron", cron="0 9 * * *", run_at=None, timezone="Asia/Shanghai") + assert spec.schedule_type is ScheduleType.CRON + assert spec.cron == "0 9 * * *" + assert spec.timezone == "Asia/Shanghai" + + def test_builds_a_once_schedule_from_an_iso_string(self): + spec = ScheduleSpec.from_primitives("once", cron=None, run_at="2026-08-01T09:00:00", timezone="Asia/Shanghai") + assert spec.schedule_type is ScheduleType.ONCE + assert spec.run_at == datetime(2026, 8, 1, 1, 0, tzinfo=UTC) + + def test_a_trailing_z_run_at_is_the_same_instant(self): + """The shape the frontend submits (`zonedLocalToUtcIso`).""" + spec = ScheduleSpec.from_primitives("once", cron=None, run_at="2026-08-01T01:00:00Z", timezone="Asia/Shanghai") + assert spec.run_at == datetime(2026, 8, 1, 1, 0, tzinfo=UTC) + + def test_unknown_schedule_type_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="Unsupported schedule_type"): + ScheduleSpec.from_primitives("teleport", cron="0 9 * * *", run_at=None, timezone="UTC") + + @pytest.mark.parametrize("cron", [None, 5, ""]) + def test_cron_without_a_usable_expression_is_rejected(self, cron): + with pytest.raises(InvalidScheduleError, match="requires schedule_spec"): + ScheduleSpec.from_primitives("cron", cron=cron, run_at=None, timezone="UTC") + + @pytest.mark.parametrize("run_at", [None, 5]) + def test_once_without_a_string_run_at_is_rejected(self, run_at): + with pytest.raises(InvalidScheduleError, match="requires run_at"): + ScheduleSpec.from_primitives("once", cron=None, run_at=run_at, timezone="UTC") + + def test_an_unparseable_run_at_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="unparseable run_at"): + ScheduleSpec.from_primitives("once", cron=None, run_at="next tuesday", timezone="UTC") + + def test_the_irrelevant_field_is_ignored(self): + """A boundary that carries both keys must not be rejected for it.""" + spec = ScheduleSpec.from_primitives("cron", cron="0 9 * * *", run_at="2026-08-01T09:00:00", timezone="UTC") + assert spec.run_at is None + + def test_value_rules_still_come_from_post_init(self): + """Not re-implemented here -- this is why each adapter stays thin.""" + with pytest.raises(InvalidScheduleError, match="exactly 5 fields"): + ScheduleSpec.from_primitives("cron", cron="0 9 * *", run_at=None, timezone="UTC") + with pytest.raises(InvalidScheduleError, match="Unknown timezone"): + ScheduleSpec.from_primitives("cron", cron="0 9 * * *", run_at=None, timezone="Mars/Olympus_Mons") + + def test_whitespace_in_a_cron_expression_is_normalized(self): + spec = ScheduleSpec.from_primitives("cron", cron=" 0 9 * * * ", run_at=None, timezone="UTC") + assert spec.cron == "0 9 * * *" + + +# ---------------------------------------------------------------- B. next_after + + +class TestNextAfter: + def test_once_in_the_future_returns_run_at(self): + spec = once_spec(after_seconds=3600) + assert spec.next_after(NOW) == NOW + timedelta(seconds=3600) + + def test_once_in_the_past_returns_none(self): + spec = ScheduleSpec.once_at(NOW - timedelta(seconds=1), "UTC") + assert spec.next_after(NOW) is None + + def test_once_exactly_now_returns_none(self): + """`run_at > now`, not `>=` (schedules.py:44).""" + assert ScheduleSpec.once_at(NOW, "UTC").next_after(NOW) is None + + def test_cron_returns_the_next_occurrence_in_utc(self): + # 09:00 UTC daily; NOW is 12:00 UTC, so the next one is tomorrow. + assert cron_spec("0 9 * * *", "UTC").next_after(NOW) == datetime(2026, 7, 28, 9, 0, tzinfo=UTC) + + def test_cron_is_evaluated_in_the_schedule_timezone(self): + # 09:00 Shanghai == 01:00 UTC. NOW (12:00 UTC) is past today's, so the + # answer is tomorrow 01:00 UTC -- not 09:00 UTC. + assert cron_spec("0 9 * * *", "Asia/Shanghai").next_after(NOW) == datetime(2026, 7, 28, 1, 0, tzinfo=UTC) + + def test_cron_absorbs_a_dst_transition(self): + """US DST starts 2026-03-08, so the same wall-clock cron maps to a + different UTC instant on either side of it. This is what evaluating in + the schedule's timezone buys.""" + spec = cron_spec("0 9 * * *", "America/New_York") + before = spec.next_after(datetime(2026, 3, 7, 0, 0, tzinfo=UTC)) + after = spec.next_after(datetime(2026, 3, 9, 0, 0, tzinfo=UTC)) + assert before == datetime(2026, 3, 7, 14, 0, tzinfo=UTC) # EST, UTC-5 + assert after == datetime(2026, 3, 9, 13, 0, tzinfo=UTC) # EDT, UTC-4 + + def test_naive_now_is_read_as_utc(self): + spec = cron_spec("0 9 * * *", "UTC") + assert spec.next_after(NOW.replace(tzinfo=None)) == spec.next_after(NOW) + + +# ---------------------------------------------------------------- C. ensure_launchable + + +class TestEnsureLaunchable: + def test_once_in_the_past_is_rejected(self): + spec = ScheduleSpec.once_at(NOW - timedelta(hours=1), "UTC") + with pytest.raises(InvalidScheduleError, match="must be in the future"): + spec.ensure_launchable(NOW, NO_DELAY) + + def test_once_closer_than_the_minimum_delay_is_rejected(self): + with pytest.raises(InvalidScheduleError, match="at least 60 seconds"): + once_spec(after_seconds=59).ensure_launchable(NOW, MIN_60) + + def test_once_exactly_at_the_minimum_delay_is_accepted(self): + assert once_spec(after_seconds=60).ensure_launchable(NOW, MIN_60) == NOW + timedelta(seconds=60) + + def test_cron_is_never_subject_to_the_delay_floor(self): + """router:105 / router:196 apply the floor only to `once`; a cron + schedule whose next fire is seconds away is perfectly legal.""" + spec = cron_spec("* * * * *", "UTC") + assert spec.ensure_launchable(NOW, MIN_60) is not None + + def test_naive_now_is_read_as_utc(self): + """Same tolerance as next_after: a caller handing over a naive clock + reading must not silently shift the delay floor by the local offset.""" + spec = once_spec(after_seconds=3600) + assert spec.ensure_launchable(NOW.replace(tzinfo=None), MIN_60) == spec.ensure_launchable(NOW, MIN_60) + + +class TestSchedulePolicyDefaults: + """ "Nobody configured a policy" must not invent a business constraint. + + Real values only ever arrive from the composition root (spec.py:22-25). The + defaults being the permissive ones is what keeps an unconfigured deployment + from silently rejecting schedules a configured one would accept -- so they + are pinned here rather than left to whatever the dataclass happens to say. + """ + + def test_defaults_are_permissive(self): + policy = SchedulePolicy() + assert policy.min_once_delay_seconds == 0 + assert policy.max_concurrent_runs == 1 + assert policy.lease_seconds == 60 + + def test_the_default_delay_floor_admits_an_imminent_once_schedule(self): + """The consequence of the constant above, stated as behaviour: with no + configured policy a one-second-away one-shot is legal.""" + assert once_spec(after_seconds=1).ensure_launchable(NOW, SchedulePolicy()) == NOW + timedelta(seconds=1) + + +# ---------------------------------------------------------------- D. value semantics + + +class TestScheduleSpecEquality: + """A value object is its values -- two specs built from equivalent inputs + must compare equal, because the repository relies on that to decide whether + a schedule actually changed. + + Serialization shape (the JSON written to `scheduled_tasks.schedule_spec`) + is deliberately NOT tested here: that mapping lives in the adapter layer, + and so do its tests. + """ + + def test_equivalent_cron_inputs_converge(self): + assert ScheduleSpec.cron_schedule(" 0 9 * * * ", "UTC") == ScheduleSpec.cron_schedule("0 9 * * *", "UTC") + + def test_equivalent_once_inputs_converge(self): + naive = ScheduleSpec.once_at(datetime(2026, 8, 1, 9, 0), "Asia/Shanghai") # noqa: DTZ001 -- naive input is the subject + aware = ScheduleSpec.once_at(datetime.fromisoformat("2026-08-01T01:00:00Z"), "Asia/Shanghai") + assert naive == aware + + def test_timezone_is_part_of_identity(self): + at = datetime(2026, 8, 1, 9, 0, tzinfo=UTC) + assert ScheduleSpec.once_at(at, "UTC") != ScheduleSpec.once_at(at, "Asia/Shanghai") + + +# ---------------------------------------------------------------- E. ScheduledTask invariants + + +class TestScheduledTaskInvariants: + def test_reuse_thread_requires_a_thread(self): + with pytest.raises(InvalidContextModeError, match="reuse_thread requires thread_id"): + make_task(context_mode=ContextMode.REUSE_THREAD, thread_id=None) + + def test_unknown_context_mode_is_a_domain_error(self): + """Not a bare ValueError -- the router maps the ScheduleError family + uniformly (router:76-77 was a 422).""" + with pytest.raises(InvalidContextModeError, match="Unsupported context_mode"): + ScheduledTask.create( + user_id="user-1", + title="t", + prompt="p", + schedule=cron_spec(), + context_mode="teleport", + thread_id=None, + now=NOW, + policy=NO_DELAY, + ) + + def test_create_drops_thread_id_for_fresh_thread_mode(self): + task = ScheduledTask.create( + user_id="user-1", + title="t", + prompt="p", + schedule=cron_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id="thread-9", + now=NOW, + policy=NO_DELAY, + ) + assert task.thread_id is None + + def test_create_computes_next_run_at_and_defaults(self): + task = ScheduledTask.create( + user_id="user-1", + title="t", + prompt="p", + schedule=cron_spec("0 9 * * *", "UTC"), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + now=NOW, + policy=NO_DELAY, + ) + assert task.task_id.startswith("task-") + assert task.status is TaskStatus.ENABLED + assert task.run_count == 0 + assert task.assistant_id == "lead_agent" + assert task.next_run_at == datetime(2026, 7, 28, 9, 0, tzinfo=UTC) + + def test_create_stamps_the_bookkeeping_timestamps_from_now(self): + """The factory already receives the clock as a rule input; reading it + a second time through the field defaults would give the aggregate two + slightly different construction instants -- and the stored row a third + if the adapter minted its own. One explicit `now`, one truth.""" + task = ScheduledTask.create( + user_id="user-1", + title="t", + prompt="p", + schedule=cron_spec("0 9 * * *", "UTC"), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + now=NOW, + policy=NO_DELAY, + ) + assert task.created_at == NOW + assert task.updated_at == NOW + + def test_create_rejects_a_once_schedule_inside_the_delay_floor(self): + with pytest.raises(InvalidScheduleError, match="at least 60 seconds"): + ScheduledTask.create( + user_id="user-1", + title="t", + prompt="p", + schedule=once_spec(after_seconds=10), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + now=NOW, + policy=MIN_60, + ) + + def test_skips_on_overlap_reflects_the_policy(self): + assert make_task().skips_on_overlap is True + + +# ---------------------------------------------------------------- F. the three truth tables + + +class TestStatusAfterLaunch: + @pytest.mark.parametrize( + ("schedule", "trigger", "status", "expected"), + [ + (once_spec(), TriggerKind.SCHEDULED, TaskStatus.RUNNING, TaskStatus.RUNNING), + (once_spec(), TriggerKind.MANUAL, TaskStatus.ENABLED, TaskStatus.RUNNING), + (cron_spec(), TriggerKind.MANUAL, TaskStatus.PAUSED, TaskStatus.PAUSED), + (cron_spec(), TriggerKind.MANUAL, TaskStatus.ENABLED, TaskStatus.ENABLED), + (cron_spec(), TriggerKind.SCHEDULED, TaskStatus.RUNNING, TaskStatus.ENABLED), + ], + ) + def test_truth_table(self, schedule, trigger, status, expected): + assert make_task(schedule=schedule, status=status).status_after_launch(trigger=trigger) is expected + + def test_manual_trigger_of_paused_once_task_yields_running(self): + """Decision-order trap: ONCE is tested before the MANUAL+PAUSED rule, + so the paused branch never sees a once task.""" + task = make_task(schedule=once_spec(), status=TaskStatus.PAUSED) + assert task.status_after_launch(trigger=TriggerKind.MANUAL) is TaskStatus.RUNNING + + +class TestStatusAfterFailure: + @pytest.mark.parametrize( + ("schedule", "trigger", "status", "expected"), + [ + (once_spec(), TriggerKind.SCHEDULED, TaskStatus.RUNNING, TaskStatus.FAILED), + (cron_spec(), TriggerKind.SCHEDULED, TaskStatus.RUNNING, TaskStatus.ENABLED), + (cron_spec(), TriggerKind.MANUAL, TaskStatus.ENABLED, TaskStatus.ENABLED), + (cron_spec(), TriggerKind.MANUAL, TaskStatus.PAUSED, TaskStatus.PAUSED), + ], + ) + def test_truth_table(self, schedule, trigger, status, expected): + assert make_task(schedule=schedule, status=status).status_after_failure(trigger=trigger) is expected + + @pytest.mark.parametrize("status", [TaskStatus.ENABLED, TaskStatus.PAUSED]) + def test_failed_manual_trigger_of_once_task_keeps_status(self, status): + """Decision-order trap: MANUAL is tested before ONCE, so a failed + manual trigger cannot burn the task's single scheduled occurrence.""" + task = make_task(schedule=once_spec(), status=status) + assert task.status_after_failure(trigger=TriggerKind.MANUAL) is status + + +class TestStatusAfterSkip: + def test_once_is_failed_not_completed(self): + assert make_task(schedule=once_spec()).status_after_skip() is TaskStatus.FAILED + + def test_cron_stays_enabled(self): + assert make_task(schedule=cron_spec()).status_after_skip() is TaskStatus.ENABLED + + +class TestStatusAfterCompletion: + @pytest.mark.parametrize("outcome", list(RunStatus)) + def test_cron_never_changes_status(self, outcome): + assert make_task(schedule=cron_spec()).status_after_completion(outcome) is None + + @pytest.mark.parametrize( + ("outcome", "expected"), + [ + (RunStatus.SUCCESS, TaskStatus.COMPLETED), + # INTERRUPTED is CANCELLED rather than FAILED: a user cancel or a + # same-thread takeover carries no execution failure. + (RunStatus.INTERRUPTED, TaskStatus.CANCELLED), + (RunStatus.FAILED, TaskStatus.FAILED), + ], + ) + def test_once_maps_each_terminal_outcome(self, outcome, expected): + assert make_task(schedule=once_spec()).status_after_completion(outcome) is expected + + +# ---------------------------------------------------------------- G. mutability gate + + +class TestMutabilityGate: + def test_ensure_mutable_rejects_a_running_task(self): + with pytest.raises(TaskNotMutableError, match="currently running"): + make_task(status=TaskStatus.RUNNING).ensure_mutable() + + @pytest.mark.parametrize("status", [s for s in TaskStatus if s is not TaskStatus.RUNNING]) + def test_ensure_mutable_allows_every_other_status(self, status): + make_task(status=status).ensure_mutable() + + def test_pause_and_resume_are_gated(self): + running = make_task(status=TaskStatus.RUNNING) + with pytest.raises(TaskNotMutableError): + running.paused() + with pytest.raises(TaskNotMutableError): + running.resumed() + + def test_pause_then_resume_round_trips(self): + task = make_task(status=TaskStatus.ENABLED) + assert task.paused().status is TaskStatus.PAUSED + assert task.paused().resumed().status is TaskStatus.ENABLED + + def test_transitions_do_not_mutate_the_original(self): + task = make_task(status=TaskStatus.ENABLED) + task.paused() + assert task.status is TaskStatus.ENABLED + + def test_transitions_leave_updated_at_to_the_repository(self): + """The repository stamps `updated_at` on write (task.py:54-57). A second + clock read here would make the domain impure and a competing source of + truth for the same column.""" + task = make_task(status=TaskStatus.ENABLED) + assert task.paused().updated_at == task.updated_at + assert task.paused().resumed().updated_at == task.updated_at + assert task.with_schedule(cron_spec("0 10 * * *"), now=NOW, policy=NO_DELAY).updated_at == task.updated_at + assert task.with_context(ContextMode.FRESH_THREAD_PER_RUN, None).updated_at == task.updated_at + + +# ---------------------------------------------------------------- H. with_schedule + + +class TestWithSchedule: + @pytest.mark.parametrize("status", [TaskStatus.COMPLETED, TaskStatus.FAILED, TaskStatus.CANCELLED]) + def test_terminal_task_is_rearmed_by_a_future_schedule(self, status): + """claim_due only admits ENABLED rows, so leaving the terminal status + would return a next_run_at that silently never fires (router:203-208).""" + task = make_task(schedule=once_spec(), status=status) + updated = task.with_schedule(once_spec(after_seconds=7200), now=NOW, policy=NO_DELAY) + assert updated.status is TaskStatus.ENABLED + assert updated.next_run_at == NOW + timedelta(seconds=7200) + + @pytest.mark.parametrize("status", [TaskStatus.ENABLED, TaskStatus.PAUSED]) + def test_non_terminal_status_is_preserved(self, status): + updated = make_task(status=status).with_schedule(cron_spec("0 10 * * *"), now=NOW, policy=NO_DELAY) + assert updated.status is status + + def test_running_task_cannot_be_rescheduled(self): + task = make_task(status=TaskStatus.RUNNING) + with pytest.raises(TaskNotMutableError): + task.with_schedule(cron_spec("0 10 * * *"), now=NOW, policy=NO_DELAY) + + def test_invalid_once_schedule_propagates(self): + task = make_task(schedule=once_spec(), status=TaskStatus.COMPLETED) + with pytest.raises(InvalidScheduleError, match="must be in the future"): + task.with_schedule(ScheduleSpec.once_at(NOW - timedelta(hours=1), "UTC"), now=NOW, policy=NO_DELAY) + + +# ---------------------------------------------------------------- I. with_context + + +class TestWithContext: + def test_switching_to_fresh_thread_clears_the_thread(self): + task = make_task(context_mode=ContextMode.REUSE_THREAD, thread_id="thread-1") + updated = task.with_context(ContextMode.FRESH_THREAD_PER_RUN, "thread-1") + assert updated.context_mode is ContextMode.FRESH_THREAD_PER_RUN + assert updated.thread_id is None + + def test_switching_to_reuse_thread_without_a_thread_is_rejected(self): + with pytest.raises(InvalidContextModeError, match="reuse_thread requires thread_id"): + make_task().with_context(ContextMode.REUSE_THREAD, None) + + def test_switching_to_reuse_thread_keeps_the_thread(self): + updated = make_task().with_context("reuse_thread", "thread-7") + assert updated.context_mode is ContextMode.REUSE_THREAD + assert updated.thread_id == "thread-7" + + def test_running_task_cannot_change_context(self): + with pytest.raises(TaskNotMutableError): + make_task(status=TaskStatus.RUNNING).with_context(ContextMode.FRESH_THREAD_PER_RUN, None) + + +# ---------------------------------------------------------------- J. execution thread + + +class TestResolveExecutionThread: + def test_fresh_thread_mode_mints_a_new_thread_every_call(self): + """Non-idempotent by design -- a dispatch must call this once and reuse + the value for the run row, the launch, and the result.""" + task = make_task(context_mode=ContextMode.FRESH_THREAD_PER_RUN) + assert task.resolve_execution_thread() != task.resolve_execution_thread() + + def test_reuse_thread_mode_returns_the_bound_thread(self): + task = make_task(context_mode=ContextMode.REUSE_THREAD, thread_id="thread-1") + assert task.resolve_execution_thread() == "thread-1" + assert task.resolve_execution_thread() == "thread-1" + + def test_reuse_thread_with_an_empty_thread_falls_back_to_a_fresh_one(self): + """Unreachable for new aggregates (__post_init__ forbids it) but rows + predating that invariant can still carry this shape. + + Asserting against the fresh-thread semantics rather than against `None`: + the fallback has to mint a real, distinct thread per call, which is what + the fresh-thread branch means. + """ + legacy = ScheduledTask.__new__(ScheduledTask) + object.__setattr__(legacy, "context_mode", ContextMode.REUSE_THREAD) + object.__setattr__(legacy, "thread_id", None) + + first = ScheduledTask.resolve_execution_thread(legacy) + second = ScheduledTask.resolve_execution_thread(legacy) + assert uuid.UUID(first), "a real thread id, not a placeholder" + assert first != second, "a fresh thread per call, like FRESH_THREAD_PER_RUN" + + +# ---------------------------------------------------------------- K. ScheduledRun + + +class TestScheduledRun: + def test_queued_run_occupies_the_active_slot(self): + run = ScheduledRun.queued(task_id="task-1", thread_id="thread-1", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED) + assert run.status is RunStatus.QUEUED + assert run.is_active is True + assert run.record_id.startswith("task-run-") + + def test_skipped_tombstone_is_terminal_and_never_queued(self): + """QUEUED falls inside uq_scheduled_task_run_active's predicate and + would collide with the pre-existing run still holding the slot.""" + run = ScheduledRun.skipped_tombstone(task_id="task-1", thread_id="thread-1", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED) + assert run.status is RunStatus.SKIPPED + assert run.is_active is False + + def test_each_run_gets_a_distinct_identity(self): + first = ScheduledRun.queued(task_id="task-1", thread_id="t", scheduled_for=NOW, trigger=TriggerKind.MANUAL) + second = ScheduledRun.queued(task_id="task-1", thread_id="t", scheduled_for=NOW, trigger=TriggerKind.MANUAL) + assert first.record_id != second.record_id + + @pytest.mark.parametrize( + ("status", "active"), + [ + (RunStatus.QUEUED, True), + (RunStatus.RUNNING, True), + (RunStatus.SUCCESS, False), + (RunStatus.FAILED, False), + (RunStatus.SKIPPED, False), + (RunStatus.INTERRUPTED, False), + ], + ) + def test_only_queued_and_running_occupy_the_active_slot(self, status, active): + """Stated as behaviour over every status rather than as an equality + against ACTIVE_RUN_STATUSES -- restating the constant proves nothing, + since editing it would edit the assertion with it. The other half of + this rule, that the set matches the partial unique index's predicate, + cannot be checked from a dependency-free domain test and lives in + test_scheduled_task_models.py. + """ + run = replace( + ScheduledRun.queued(task_id="task-1", thread_id="thread-1", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED), + status=status, + ) + assert run.is_active is active diff --git a/backend/tests/test_schedule_fakes.py b/backend/tests/test_schedule_fakes.py new file mode 100644 index 000000000..abed843ff --- /dev/null +++ b/backend/tests/test_schedule_fakes.py @@ -0,0 +1,565 @@ +"""Contract suite for the schedule repository ports. + +Every case here runs **twice**: once against the in-memory doubles and once +against the SQL adapters on a real file-backed sqlite database. That is the +point -- a rule stated in a port docstring has to hold for both, and a +divergence becomes a failure rather than a surprise in production. + +What the contract owns is single-threaded semantics: which rows `claim_due` +selects, that `add` refuses a second active record, what `protect_terminal` +preserves. What it deliberately does **not** own is atomicity -- the doubles +provide none, and a green run here says nothing about two dispatchers racing. +That is covered against a real database in +``test_schedule_dispatch_race.py``. Do not read a passing contract suite +as licence to run more than one scheduler. +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from datetime import UTC, datetime, timedelta + +import pytest +import pytest_asyncio +from schedule_fakes import ( + FakeRunLauncher, + FakeThreadLookup, + InMemoryScheduledRunRepository, + InMemoryScheduledTaskRepository, +) + +from app.adapters.schedule.scheduled_run_repository import SqlScheduledRunRepository +from app.adapters.schedule.scheduled_task_repository import SqlScheduledTaskRepository +from deerflow.config.database_config import DatabaseConfig +from deerflow.domain.schedule.exceptions import ActiveRunConflictError +from deerflow.domain.schedule.model import ContextMode, RunStatus, ScheduledRun, ScheduledTask, ScheduleSpec, TaskStatus, TriggerKind +from deerflow.domain.schedule.ports import ScheduledRunRepository, ScheduledTaskRepository +from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config + +pytestmark = pytest.mark.asyncio + +NOW = datetime(2026, 7, 27, 12, 0, tzinfo=UTC) +CRON = ScheduleSpec.cron_schedule("0 9 * * *", "UTC") +ONCE = ScheduleSpec.once_at(datetime(2026, 8, 1, 9, 0, tzinfo=UTC), "UTC") + + +@pytest_asyncio.fixture(params=["memory", "sql"]) +async def repos(request, tmp_path) -> AsyncIterator[tuple[ScheduledTaskRepository, ScheduledRunRepository]]: + """One parametrized fixture, two implementations of the same ports.""" + if request.param == "memory": + yield InMemoryScheduledTaskRepository(), InMemoryScheduledRunRepository() + return + await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) + try: + sf = get_session_factory() + assert sf is not None + yield SqlScheduledTaskRepository(sf), SqlScheduledRunRepository(sf) + finally: + await close_engine() + + +@pytest_asyncio.fixture +async def tasks(repos) -> ScheduledTaskRepository: + return repos[0] + + +@pytest_asyncio.fixture +async def runs(repos) -> ScheduledRunRepository: + return repos[1] + + +async def seed( + repo: ScheduledTaskRepository, + task: ScheduledTask, + *, + claimed_until: datetime | None = None, +) -> ScheduledTask: + """Install a task, optionally already carrying a claim. + + A claim cannot be installed through the port -- `claim_due` only stamps + tasks that are actually due, and these cases need shapes that claiming + would never produce (running with an *expired* claim, running with a live + one). So each implementation is set up directly: the fake exposes a seed + helper, and the SQL side writes the row. That is the one place this suite + reaches past the port, and it is why the assertions that follow go back + through it. + """ + stored = await repo.add(task) + if claimed_until is None: + return stored + + if isinstance(repo, InMemoryScheduledTaskRepository): + repo.seed(task, lease_owner="prior-worker", lease_expires_at=claimed_until) + return task + + from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow + + factory = get_session_factory() + assert factory is not None + async with factory() as session: + row = await session.get(ScheduledTaskRow, task.task_id) + row.lease_owner = "prior-worker" + row.lease_expires_at = claimed_until + await session.commit() + return task + + +def make_task( + task_id: str = "task-1", + *, + user_id: str = "user-1", + status: TaskStatus = TaskStatus.ENABLED, + next_run_at: datetime | None = None, + schedule: ScheduleSpec = CRON, + thread_id: str | None = None, + context_mode: ContextMode = ContextMode.FRESH_THREAD_PER_RUN, +) -> ScheduledTask: + return ScheduledTask( + task_id=task_id, + user_id=user_id, + title=task_id, + prompt="do the thing", + schedule=schedule, + status=status, + next_run_at=next_run_at, + thread_id=thread_id, + context_mode=context_mode, + ) + + +class TestProtocolConformance: + """`runtime_checkable` only checks that the methods exist, not their + signatures -- enough to catch a rename that updates one side only.""" + + async def test_repositories_satisfy_their_ports(self, tasks, runs): + assert isinstance(tasks, ScheduledTaskRepository) + assert isinstance(runs, ScheduledRunRepository) + + +class TestTaskOwnershipIsolation: + """Another user's task must read as absent, never as forbidden.""" + + async def test_get_hides_another_users_task(self, tasks): + await tasks.add(make_task(user_id="owner")) + assert await tasks.get("task-1", user_id="owner") is not None + assert await tasks.get("task-1", user_id="intruder") is None + + async def test_save_refuses_another_users_task(self, tasks): + await tasks.add(make_task(user_id="owner")) + assert await tasks.save(make_task(user_id="intruder")) is None + stored = await tasks.get("task-1", user_id="owner") + assert stored.user_id == "owner" + + async def test_delete_refuses_another_users_task(self, tasks): + await tasks.add(make_task(user_id="owner")) + assert await tasks.delete("task-1", user_id="intruder") is False + assert await tasks.get("task-1", user_id="owner") is not None + + async def test_list_by_user_excludes_other_owners(self, tasks): + await tasks.add(make_task("mine", user_id="owner")) + await tasks.add(make_task("theirs", user_id="someone-else")) + assert [t.task_id for t in await tasks.list_by_user("owner")] == ["mine"] + + async def test_list_by_thread_only_matches_bound_tasks(self, tasks): + await tasks.add(make_task("bound", context_mode=ContextMode.REUSE_THREAD, thread_id="thread-1")) + await tasks.add(make_task("unbound")) + listed = await tasks.list_by_user_and_thread("user-1", "thread-1") + assert [task.task_id for task in listed] == ["bound"] + + +class TestRoundTrip: + """What goes in comes back out -- including the value object, which the + SQL side has to rebuild from three separate columns.""" + + async def test_a_cron_task_round_trips(self, tasks): + original = await tasks.add(make_task(schedule=ScheduleSpec.cron_schedule("*/5 * * * *", "Asia/Shanghai"))) + stored = await tasks.get(original.task_id, user_id="user-1") + assert stored.schedule == original.schedule + assert stored.schedule.cron == "*/5 * * * *" + assert stored.schedule.timezone == "Asia/Shanghai" + + async def test_a_once_task_round_trips_to_the_same_instant(self, tasks): + original = await tasks.add(make_task(schedule=ONCE)) + stored = await tasks.get(original.task_id, user_id="user-1") + assert stored.schedule == original.schedule + + async def test_timestamps_come_back_timezone_aware(self, tasks): + """SQLite drops tzinfo on read; a naive datetime downstream would + compare wrong against an aware `now`.""" + await tasks.add(make_task(next_run_at=NOW)) + stored = await tasks.get("task-1", user_id="user-1") + assert stored.next_run_at.tzinfo is not None + assert stored.created_at.tzinfo is not None + + async def test_add_persists_the_aggregates_construction_instant(self, tasks): + """`created_at` has one source of truth: the aggregate's construction + instant. An adapter that mints its own timestamp on insert gives the + same fact two values -- domain tests and listings would disagree with + the stored row by however long the insert took.""" + task = make_task() + await tasks.add(task) + stored = await tasks.get(task.task_id, user_id="user-1") + assert stored.created_at == task.created_at + assert stored.updated_at == task.updated_at + + async def test_save_replaces_the_whole_aggregate(self, tasks): + from dataclasses import replace + + await tasks.add(make_task()) + stored = await tasks.get("task-1", user_id="user-1") + + await tasks.save(replace(stored, title="renamed", status=TaskStatus.PAUSED, run_count=7)) + + reloaded = await tasks.get("task-1", user_id="user-1") + assert reloaded.title == "renamed" + assert reloaded.status is TaskStatus.PAUSED + assert reloaded.run_count == 7 + + +class TestClaimDue: + async def test_claims_an_enabled_due_task(self, tasks): + await tasks.add(make_task(next_run_at=NOW - timedelta(minutes=1))) + + claimed = await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) + + assert [task.task_id for task in claimed] == ["task-1"] + assert claimed[0].status is TaskStatus.RUNNING + + async def test_a_claimed_task_is_not_claimed_again(self, tasks): + """The lease is not readable through the port, so the rule is asserted + the way it actually matters: a second claimer comes back empty.""" + await tasks.add(make_task(next_run_at=NOW - timedelta(minutes=1))) + assert len(await tasks.claim_due(now=NOW, lease_seconds=120, limit=10)) == 1 + + assert await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) == [] + + async def test_the_claim_expires(self, tasks): + await tasks.add(make_task(next_run_at=NOW - timedelta(minutes=1))) + await tasks.claim_due(now=NOW, lease_seconds=60, limit=10) + + later = NOW + timedelta(seconds=61) + reclaimed = await tasks.claim_due(now=later, lease_seconds=60, limit=10) + + assert [task.task_id for task in reclaimed] == ["task-1"] + + @pytest.mark.parametrize( + ("label", "task_kwargs"), + [ + ("not yet due", {"next_run_at": NOW + timedelta(minutes=1)}), + ("never scheduled", {"next_run_at": None}), + ("paused", {"status": TaskStatus.PAUSED, "next_run_at": NOW - timedelta(minutes=1)}), + ("completed", {"status": TaskStatus.COMPLETED, "next_run_at": NOW - timedelta(minutes=1)}), + ], + ) + async def test_does_not_claim(self, tasks, label, task_kwargs): + await tasks.add(make_task(**task_kwargs)) + assert await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) == [], label + + async def test_reclaims_a_task_stuck_mid_dispatch(self, tasks): + """The claimer died between claiming and launching: status is running, + the claim has expired, and the task must not stay unreachable.""" + await seed( + tasks, + make_task(status=TaskStatus.RUNNING, next_run_at=NOW - timedelta(minutes=1)), + claimed_until=NOW - timedelta(seconds=1), + ) + + claimed = await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) + + assert [task.task_id for task in claimed] == ["task-1"] + + async def test_claims_the_most_overdue_first_and_honours_the_limit(self, tasks): + await tasks.add(make_task("late", next_run_at=NOW - timedelta(hours=2))) + await tasks.add(make_task("later", next_run_at=NOW - timedelta(hours=1))) + + claimed = await tasks.claim_due(now=NOW, lease_seconds=120, limit=1) + + assert [task.task_id for task in claimed] == ["late"] + + +class TestRecordLaunch: + async def test_writes_bookkeeping_and_frees_the_task_for_the_next_round(self, tasks): + await tasks.add(make_task(next_run_at=NOW - timedelta(minutes=1))) + await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) + + await tasks.record_launch( + "task-1", + status=TaskStatus.ENABLED, + next_run_at=NOW - timedelta(seconds=1), + last_run_at=NOW, + last_run_id="run-1", + last_thread_id="thread-1", + last_error=None, + increment_run_count=True, + ) + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.ENABLED + assert task.last_run_id == "run-1" + assert task.run_count == 1 + # The claim was released, so the next round can take it again. + assert len(await tasks.claim_due(now=NOW, lease_seconds=120, limit=10)) == 1 + + async def test_protect_terminal_keeps_a_concurrently_finalized_verdict(self, tasks): + """A fast-failing run's completion hook lands before the launch path's + own write; the completion is authoritative.""" + await tasks.add(make_task(status=TaskStatus.COMPLETED, schedule=ONCE)) + + await tasks.record_launch( + "task-1", + status=TaskStatus.RUNNING, + next_run_at=None, + last_run_at=NOW, + last_run_id="run-1", + last_thread_id="thread-1", + last_error="stale", + increment_run_count=True, + protect_terminal=True, + ) + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.COMPLETED, "the terminal status must survive" + assert task.last_error is None, "the terminal error must survive" + assert task.last_run_id == "run-1", "bookkeeping is still recorded" + assert task.run_count == 1 + + async def test_without_protect_terminal_the_write_wins(self, tasks): + await tasks.add(make_task(status=TaskStatus.COMPLETED, schedule=ONCE)) + + await tasks.record_launch( + "task-1", + status=TaskStatus.FAILED, + next_run_at=None, + last_run_at=NOW, + last_run_id=None, + last_thread_id=None, + last_error="boom", + increment_run_count=False, + ) + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.FAILED + assert task.last_error == "boom" + + async def test_unknown_task_is_ignored(self, tasks): + await tasks.record_launch( + "nope", + status=TaskStatus.ENABLED, + next_run_at=None, + last_run_at=None, + last_run_id=None, + last_thread_id=None, + last_error=None, + increment_run_count=False, + ) + + +class TestRecordCompletion: + """The completion hook's write. + + Deliberately as narrow as `record_launch` is, and for the same reason: the + two race, so neither may write through the whole aggregate. This one owns + the terminal verdict and nothing else -- every scheduling field belongs to + the launch path, which may commit at any point around it. + """ + + async def test_records_the_terminal_status_and_error(self, tasks): + await tasks.add(make_task(status=TaskStatus.RUNNING, schedule=ONCE)) + + await tasks.record_completion("task-1", user_id="user-1", status=TaskStatus.FAILED, error="boom") + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.FAILED + assert task.last_error == "boom" + + async def test_a_none_status_records_the_error_and_leaves_the_status(self, tasks): + """A cron task's schedule outlives any single run, so only what went + wrong is recorded.""" + await tasks.add(make_task(status=TaskStatus.ENABLED, schedule=CRON)) + + await tasks.record_completion("task-1", user_id="user-1", status=None, error="boom") + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.ENABLED + assert task.last_error == "boom" + + async def test_never_rolls_back_a_concurrent_launch_write(self, tasks): + """The regression this method exists to prevent. + + A fast-failing run reaches the completion hook while the dispatch path + is still writing its bookkeeping. Whichever lands second must not undo + the other: the launch owns the schedule, the completion owns the + verdict. + """ + await tasks.add(make_task(next_run_at=NOW - timedelta(minutes=1))) + await tasks.claim_due(now=NOW, lease_seconds=120, limit=10) + + next_at = NOW + timedelta(days=1) + await tasks.record_launch( + "task-1", + status=TaskStatus.ENABLED, + next_run_at=next_at, + last_run_at=NOW, + last_run_id="run-1", + last_thread_id="thread-1", + last_error=None, + increment_run_count=True, + protect_terminal=True, + ) + + await tasks.record_completion("task-1", user_id="user-1", status=None, error="boom") + + task = await tasks.get("task-1", user_id="user-1") + assert task.next_run_at == next_at, "the launch path's next fire time must survive" + assert task.run_count == 1, "the launch path's run count must survive" + assert task.last_run_id == "run-1" + assert task.last_thread_id == "thread-1" + assert task.last_error == "boom", "the completion still records its verdict" + # The whole point: the task is still reachable by the next poll. + claimed = await tasks.claim_due(now=NOW + timedelta(days=2), lease_seconds=120, limit=10) + assert [t.task_id for t in claimed] == ["task-1"] + + async def test_another_users_task_is_untouched(self, tasks): + await tasks.add(make_task(status=TaskStatus.ENABLED)) + + await tasks.record_completion("task-1", user_id="someone-else", status=TaskStatus.FAILED, error="boom") + + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.ENABLED + assert task.last_error is None + + async def test_unknown_task_is_ignored(self, tasks): + await tasks.record_completion("nope", user_id="user-1", status=TaskStatus.FAILED, error="boom") + + +class TestCancelStuckOnceTasks: + async def test_cancels_a_launched_once_task_with_no_claim(self, tasks): + """Launched, so the claim was released; the completion hook then died + with the process. Expired-claim reclaim can never see this one.""" + await tasks.add(make_task(status=TaskStatus.RUNNING, schedule=ONCE)) + + assert await tasks.cancel_stuck_once_tasks(error="restarted") == 1 + task = await tasks.get("task-1", user_id="user-1") + assert task.status is TaskStatus.CANCELLED + assert task.last_error == "restarted" + + async def test_leaves_a_claimed_task_to_claim_expiry(self, tasks): + """Claimed but not launched -- expired-claim reclaim recovers it, and + cancelling here would throw away a dispatch that never happened.""" + await seed( + tasks, + make_task(status=TaskStatus.RUNNING, schedule=ONCE, next_run_at=NOW - timedelta(minutes=1)), + claimed_until=NOW + timedelta(seconds=60), + ) + assert await tasks.cancel_stuck_once_tasks(error="restarted") == 0 + + async def test_leaves_cron_tasks_alone(self, tasks): + await tasks.add(make_task(status=TaskStatus.RUNNING, schedule=CRON)) + assert await tasks.cancel_stuck_once_tasks(error="restarted") == 0 + + +class TestActiveSlot: + def _queued(self, task_id: str = "task-1") -> ScheduledRun: + return ScheduledRun.queued(task_id=task_id, thread_id="thread-1", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED) + + def _tombstone(self, task_id: str = "task-1") -> ScheduledRun: + return ScheduledRun.skipped_tombstone(task_id=task_id, thread_id="thread-1", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED) + + async def test_second_active_record_is_refused(self, runs): + await runs.add(self._queued()) + with pytest.raises(ActiveRunConflictError): + await runs.add(self._queued()) + + async def test_a_tombstone_never_conflicts(self, runs): + """Terminal from birth, so it sits outside the active-slot rule -- this + is why the skip path cannot reuse the queued factory.""" + await runs.add(self._queued()) + await runs.add(self._tombstone()) + assert await runs.count_active() == 1 + + async def test_another_task_is_unaffected(self, runs): + await runs.add(self._queued("task-1")) + await runs.add(self._queued("task-2")) + assert await runs.count_active() == 2 + + async def test_slot_frees_up_once_the_record_terminalizes(self, runs): + first = await runs.add(self._queued()) + await runs.update_status(first.record_id, status=RunStatus.SUCCESS, finished_at=NOW) + await runs.add(self._queued()) + assert await runs.count_active() == 1 + + async def test_has_active_is_scoped_to_one_task_while_count_is_global(self, runs): + await runs.add(self._queued("task-1")) + await runs.add(self._queued("task-2")) + assert await runs.has_active("task-1") is True + assert await runs.has_active("task-3") is False + assert await runs.count_active() == 2 + + async def test_a_run_round_trips(self, runs): + stored = await runs.add(self._queued()) + listed = await runs.list_by_task("task-1", limit=10, offset=0) + assert [r.record_id for r in listed] == [stored.record_id] + assert listed[0].trigger is TriggerKind.SCHEDULED + assert listed[0].scheduled_for.tzinfo is not None + + +class TestRunStatusWrites: + async def _one_queued(self, runs) -> ScheduledRun: + return await runs.add(ScheduledRun.queued(task_id="t", thread_id="th", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED)) + + async def test_protect_terminal_backfills_without_overwriting(self, runs): + run = await self._one_queued(runs) + await runs.update_status(run.record_id, status=RunStatus.FAILED, error="boom", finished_at=NOW) + + # The launch path's write arrives late. + await runs.update_status(run.record_id, status=RunStatus.RUNNING, run_id="run-1", started_at=NOW, protect_terminal=True) + + stored = (await runs.list_by_task("t", limit=10, offset=0))[0] + assert stored.status is RunStatus.FAILED + assert stored.error == "boom" + assert stored.run_id == "run-1", "the id the completion could not know is backfilled" + assert stored.started_at == NOW + + async def test_unknown_record_is_ignored(self, runs): + await runs.update_status("nope", status=RunStatus.SUCCESS) + + async def test_mark_stale_active_terminalizes_orphans(self, runs): + active = await self._one_queued(runs) + done = await runs.add(ScheduledRun.skipped_tombstone(task_id="t", thread_id="th", scheduled_for=NOW, trigger=TriggerKind.SCHEDULED)) + + assert await runs.mark_stale_active(error="gateway restarted") == 1 + + by_id = {run.record_id: run for run in await runs.list_by_task("t", limit=10, offset=0)} + assert by_id[active.record_id].status is RunStatus.INTERRUPTED + assert by_id[active.record_id].error == "gateway restarted" + assert by_id[done.record_id].status is RunStatus.SKIPPED + + +class TestLauncherAndThreadLookup: + """Fake-only: these two ports have no SQL implementation -- one starts a + run and the other asks the thread store, so both land in later adapters.""" + + async def test_launcher_records_the_call_and_echoes_the_thread(self): + launcher = FakeRunLauncher() + launched = await launcher.launch( + thread_id="thread-1", + assistant_id="lead_agent", + prompt="go", + owner_user_id="user-1", + metadata={"scheduled_task_id": "task-1"}, + ) + assert launched.thread_id == "thread-1" + assert launcher.calls[0]["metadata"] == {"scheduled_task_id": "task-1"} + + async def test_launcher_can_be_driven_into_either_failure_branch(self): + boom = RuntimeError("nope") + launcher = FakeRunLauncher(fail_with=boom) + with pytest.raises(RuntimeError): + await launcher.launch(thread_id="t", assistant_id=None, prompt="p", owner_user_id=None, metadata={}) + assert len(launcher.calls) == 1, "the attempt is still recorded" + + async def test_thread_lookup_requires_both_existence_and_ownership(self): + lookup = FakeThreadLookup({"thread-1": "user-1"}) + assert await lookup.exists_for_user("thread-1", "user-1") is True + assert await lookup.exists_for_user("thread-1", "user-2") is False + assert await lookup.exists_for_user("missing", "user-1") is False diff --git a/backend/tests/test_schedule_poller.py b/backend/tests/test_schedule_poller.py new file mode 100644 index 000000000..2db99354d --- /dev/null +++ b/backend/tests/test_schedule_poller.py @@ -0,0 +1,149 @@ +"""Tests for the scheduler poller. + +The poller owns everything about *when* the service is asked to work, and +nothing about what the work means. Two behaviours here are load-bearing and +were carried over deliberately from `app/scheduler/service.py`: + + - a failing poll must not kill the loop for the rest of the process life + (a transient "database is locked" would otherwise silently stop every + scheduled task until the next restart), and + - startup reconciliation must not block startup. +""" + +from __future__ import annotations + +import asyncio +from datetime import datetime + +import pytest + +from app.scheduler.poller import SchedulePoller + + +class _SpyService: + """Stands in for `ScheduleService`, recording how the poller drove it.""" + + def __init__(self, *, run_once_error: Exception | None = None, reconcile_error: Exception | None = None) -> None: + self.run_once_error = run_once_error + self.reconcile_error = reconcile_error + self.run_once_calls: list[datetime] = [] + self.reconcile_calls: list[str] = [] + self.polled = asyncio.Event() + + async def run_once(self, *, now: datetime) -> list: + self.run_once_calls.append(now) + self.polled.set() + if self.run_once_error is not None: + raise self.run_once_error + return [] + + async def reconcile_on_startup(self, *, error: str) -> tuple[int, int]: + self.reconcile_calls.append(error) + if self.reconcile_error is not None: + raise self.reconcile_error + return 2, 1 + + +async def _drain(poller: SchedulePoller, service: _SpyService, *, polls: int = 1) -> None: + """Start the poller, wait for it to poll, then stop it.""" + await poller.start() + try: + for _ in range(polls): + service.polled.clear() + await asyncio.wait_for(service.polled.wait(), timeout=2) + finally: + await poller.stop() + + +class TestStartupReconciliation: + @pytest.mark.asyncio + async def test_start_reconciles_before_polling(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service) + assert len(service.reconcile_calls) == 1 + assert "restart" in service.reconcile_calls[0] + + @pytest.mark.asyncio + async def test_a_failed_reconcile_does_not_block_the_loop(self): + """Whether a partial reconcile blocks startup is the caller's policy, + and this caller's policy is: log it and keep scheduling. A gateway that + refuses to start because of leftover rows is worse than one that runs + with them.""" + service = _SpyService(reconcile_error=RuntimeError("db down")) + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service) + assert service.run_once_calls + + +class TestPolling: + @pytest.mark.asyncio + async def test_polls_repeatedly(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service, polls=3) + assert len(service.run_once_calls) >= 3 + + @pytest.mark.asyncio + async def test_each_poll_passes_a_tz_aware_now(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service) + assert all(now.tzinfo is not None for now in service.run_once_calls) + + @pytest.mark.asyncio + async def test_a_failing_poll_does_not_kill_the_loop(self): + """The regression this guards: one transient DB error used to end + scheduling for the rest of the process life.""" + service = _SpyService(run_once_error=RuntimeError("database is locked")) + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service, polls=3) + assert len(service.run_once_calls) >= 3 + + +class TestLifecycle: + @pytest.mark.asyncio + async def test_stop_is_awaited_to_completion(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await poller.start() + await asyncio.wait_for(service.polled.wait(), timeout=2) + await poller.stop() + before = len(service.run_once_calls) + await asyncio.sleep(0.05) + assert len(service.run_once_calls) == before + + @pytest.mark.asyncio + async def test_stop_does_not_wait_out_the_poll_interval(self): + """The loop waits on a stop event rather than sleeping, so shutdown is + prompt even with a production-sized interval.""" + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=300) + await poller.start() + await asyncio.wait_for(service.polled.wait(), timeout=2) + await asyncio.wait_for(poller.stop(), timeout=2) + + @pytest.mark.asyncio + async def test_start_is_idempotent(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await poller.start() + await poller.start() + try: + await asyncio.wait_for(service.polled.wait(), timeout=2) + finally: + await poller.stop() + assert len(service.reconcile_calls) == 1 + + @pytest.mark.asyncio + async def test_stop_without_start_is_a_no_op(self): + poller = SchedulePoller(_SpyService(), poll_interval_seconds=0.01) + await poller.stop() + + @pytest.mark.asyncio + async def test_it_can_be_restarted(self): + service = _SpyService() + poller = SchedulePoller(service, poll_interval_seconds=0.01) + await _drain(poller, service) + await _drain(poller, service) + assert len(service.reconcile_calls) == 2 diff --git a/backend/tests/test_schedule_response_models.py b/backend/tests/test_schedule_response_models.py new file mode 100644 index 000000000..1866c9f5d --- /dev/null +++ b/backend/tests/test_schedule_response_models.py @@ -0,0 +1,176 @@ +"""Tests for the scheduled-task HTTP response models. + +Two things are being pinned, and they pull in opposite directions: + + - **the leak is closed** -- server-owned and scheduler-internal fields that + the pre-migration router dumped from the ORM row must not appear, and + - **nothing the client already reads changed** -- the field set and the + timestamp spelling have to stay byte-compatible with what the legacy + `to_dict()` + `coerce_iso` path emitted, or the frontend breaks. +""" + +from __future__ import annotations + +import json +from datetime import UTC, datetime, timedelta, timezone + +import pytest + +from app.gateway.routers.schedule.models import ScheduledRunResponse, ScheduledTaskResponse +from deerflow.domain.schedule.model import ContextMode, RunStatus, ScheduledRun, ScheduledTask, ScheduleSpec, TaskStatus, TriggerKind + +# Exactly the frontend's `ScheduledTask` type (frontend/src/core/scheduled-tasks/types.ts). +FRONTEND_TASK_FIELDS = { + "id", + "thread_id", + "context_mode", + "title", + "prompt", + "schedule_type", + "schedule_spec", + "timezone", + "status", + "next_run_at", + "last_run_at", + "last_run_id", + "last_thread_id", + "last_error", + "run_count", + "created_at", + "updated_at", +} + +# Exactly the frontend's `ScheduledTaskRun` type. +FRONTEND_RUN_FIELDS = { + "id", + "task_id", + "thread_id", + "run_id", + "scheduled_for", + "trigger", + "status", + "error", + "started_at", + "finished_at", + "created_at", +} + +# What the ORM dump used to expose. Lease fields never reached the domain, so +# they cannot leak now; the other three can, and must not. +LEAKED_FIELDS = {"user_id", "assistant_id", "overlap_policy", "lease_owner", "lease_expires_at"} + + +def _task(**overrides) -> ScheduledTask: + defaults = dict( + task_id="task-1", + user_id="user-1", + title="Morning digest", + prompt="summarize", + schedule=ScheduleSpec.cron_schedule("0 9 * * *", "Asia/Shanghai"), + context_mode=ContextMode.REUSE_THREAD, + thread_id="thread-1", + status=TaskStatus.ENABLED, + next_run_at=datetime(2026, 8, 1, 1, 0, tzinfo=UTC), + last_run_at=datetime(2026, 7, 31, 1, 0, tzinfo=UTC), + last_run_id="run-1", + last_thread_id="thread-1", + last_error=None, + run_count=3, + created_at=datetime(2026, 7, 1, 0, 0, tzinfo=UTC), + updated_at=datetime(2026, 7, 31, 1, 0, tzinfo=UTC), + ) + return ScheduledTask(**{**defaults, **overrides}) + + +def _run(**overrides) -> ScheduledRun: + defaults = dict( + record_id="task-run-1", + task_id="task-1", + thread_id="thread-1", + scheduled_for=datetime(2026, 8, 1, 1, 0, tzinfo=UTC), + trigger=TriggerKind.SCHEDULED, + status=RunStatus.SUCCESS, + run_id="run-1", + error=None, + started_at=datetime(2026, 8, 1, 1, 0, 1, tzinfo=UTC), + finished_at=datetime(2026, 8, 1, 1, 0, 9, tzinfo=UTC), + created_at=datetime(2026, 8, 1, 1, 0, tzinfo=UTC), + ) + return ScheduledRun(**{**defaults, **overrides}) + + +class TestTheLeakIsClosed: + def test_no_server_owned_field_is_published(self): + emitted = set(ScheduledTaskResponse.from_domain(_task()).model_dump()) + assert emitted & LEAKED_FIELDS == set() + + def test_the_field_set_is_exactly_what_the_frontend_declares(self): + """Both directions matter: an extra field is a leak, a missing one + breaks a client that reads it.""" + assert set(ScheduledTaskResponse.from_domain(_task()).model_dump()) == FRONTEND_TASK_FIELDS + + def test_run_records_publish_exactly_the_declared_fields(self): + assert set(ScheduledRunResponse.from_domain(_run()).model_dump()) == FRONTEND_RUN_FIELDS + + +class TestFieldMapping: + def test_the_aggregate_id_is_published_as_id(self): + assert ScheduledTaskResponse.from_domain(_task()).id == "task-1" + + def test_the_record_id_is_published_as_id(self): + assert ScheduledRunResponse.from_domain(_run()).id == "task-run-1" + + def test_the_schedule_value_object_is_flattened_into_three_fields(self): + """The client's shape predates the value object and keeps the three + columns separate; the flattening is this model's job, not the + domain's.""" + response = ScheduledTaskResponse.from_domain(_task()) + assert response.schedule_type == "cron" + assert response.schedule_spec == {"cron": "0 9 * * *"} + assert response.timezone == "Asia/Shanghai" + + def test_a_once_schedule_flattens_to_run_at(self): + task = _task(schedule=ScheduleSpec.once_at(datetime(2026, 8, 1, 9, 0, tzinfo=UTC), "UTC")) + response = ScheduledTaskResponse.from_domain(task) + assert response.schedule_type == "once" + assert response.schedule_spec == {"run_at": "2026-08-01T09:00:00+00:00"} + + def test_enums_are_emitted_as_their_string_values(self): + """`StrEnum` would serialize acceptably either way, but the client + compares against string literals, so the model declares `str`.""" + payload = json.loads(ScheduledTaskResponse.from_domain(_task()).model_dump_json()) + assert payload["status"] == "enabled" + assert payload["context_mode"] == "reuse_thread" + + +class TestTimestampCompatibility: + def test_utc_timestamps_keep_the_legacy_spelling(self): + """`coerce_iso` emitted `astimezone(UTC).isoformat()`; anything else + is a silent wire change for every client parsing these.""" + payload = json.loads(ScheduledTaskResponse.from_domain(_task()).model_dump_json()) + assert payload["next_run_at"] == "2026-08-01T01:00:00+00:00" + assert payload["created_at"] == "2026-07-01T00:00:00+00:00" + + def test_a_non_utc_timestamp_is_converted_not_echoed(self): + """The legacy path normalised the offset away. A value that reached + the aggregate in another zone must not start emitting `+08:00`.""" + shanghai = timezone(timedelta(hours=8)) + task = _task(next_run_at=datetime(2026, 8, 1, 9, 0, tzinfo=shanghai)) + payload = json.loads(ScheduledTaskResponse.from_domain(task).model_dump_json()) + assert payload["next_run_at"] == "2026-08-01T01:00:00+00:00" + + def test_a_naive_timestamp_is_assumed_utc(self): + """Same assumption the repository's `_tz_aware` makes on read.""" + task = _task(next_run_at=datetime(2026, 8, 1, 1, 0)) + payload = json.loads(ScheduledTaskResponse.from_domain(task).model_dump_json()) + assert payload["next_run_at"] == "2026-08-01T01:00:00+00:00" + + @pytest.mark.parametrize("field", ["next_run_at", "last_run_at"]) + def test_absent_timestamps_stay_null(self, field): + payload = json.loads(ScheduledTaskResponse.from_domain(_task(**{field: None})).model_dump_json()) + assert payload[field] is None + + def test_run_timestamps_use_the_same_spelling(self): + payload = json.loads(ScheduledRunResponse.from_domain(_run()).model_dump_json()) + assert payload["scheduled_for"] == "2026-08-01T01:00:00+00:00" + assert payload["finished_at"] == "2026-08-01T01:00:09+00:00" diff --git a/backend/tests/test_schedule_router.py b/backend/tests/test_schedule_router.py new file mode 100644 index 000000000..2a02eb7ac --- /dev/null +++ b/backend/tests/test_schedule_router.py @@ -0,0 +1,599 @@ +"""Behaviour tests for the scheduled-task HTTP adapter. + +Driven through the handlers with a **real** `ScheduleService` over in-memory +port fakes, not a mocked service. The router's whole remaining job is protocol +translation, and half of that is turning domain errors into status codes -- a +mocked service would let those assertions pass without a domain error ever +being raised. + +`__wrapped__` unwraps `@require_permission` (authorization is covered by its +own suite) while keeping `_map_domain_errors` in the call path, which is the +layer under test here. + +Every scenario the deleted legacy suite (`test_scheduled_task_router_behavior.py`) +pinned has a counterpart below, plus the response-shape and error-mapping +cases the old dict-returning router could not express. +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest +from fastapi import HTTPException +from schedule_fakes import ( + FakeRunLauncher, + FakeThreadLookup, + InMemoryScheduledRunRepository, + InMemoryScheduledTaskRepository, +) + +from app.gateway.routers.schedule import router as router_module +from deerflow.domain.schedule.commands import UNSET +from deerflow.domain.schedule.exceptions import LaunchFailedError, ThreadBusyError +from deerflow.domain.schedule.model import RunStatus, ScheduledRun, ScheduledTask, SchedulePolicy, ScheduleSpec, TaskStatus, TriggerKind +from deerflow.domain.schedule.service import ScheduleService + +USER = "user-1" +OTHER_USER = "user-2" +THREAD = "thread-1" + + +class _User: + def __init__(self, user_id: str | None) -> None: + self.id = user_id + + +@pytest.fixture +def tasks(): + return InMemoryScheduledTaskRepository() + + +@pytest.fixture +def runs(): + return InMemoryScheduledRunRepository() + + +@pytest.fixture +def launcher(): + return FakeRunLauncher() + + +@pytest.fixture +def service(tasks, runs, launcher): + return ScheduleService( + tasks=tasks, + runs=runs, + launcher=launcher, + threads=FakeThreadLookup({THREAD: USER}), + policy=SchedulePolicy(min_once_delay_seconds=60, max_concurrent_runs=3, lease_seconds=120), + ) + + +@pytest.fixture +def as_user(monkeypatch): + """Authenticate the handlers as a given user id (None = anonymous).""" + + def _apply(user_id: str | None = USER): + async def _resolve(_request): + return None if user_id is None else _User(user_id) + + monkeypatch.setattr(router_module, "get_optional_user_from_request", _resolve) + + _apply() + return _apply + + +def _call(handler, **kwargs): + """Invoke a route handler with authorization unwrapped.""" + return handler.__wrapped__(request=object(), **kwargs) + + +def _create_body(**overrides): + defaults = dict( + thread_id=None, + context_mode="fresh_thread_per_run", + title="Daily summary", + prompt="Summarize the thread", + schedule_type="cron", + schedule_spec={"cron": "0 9 * * *"}, + timezone="UTC", + ) + return router_module.CreateScheduledTaskRequest(**{**defaults, **overrides}) + + +def _update_body(**fields): + return router_module.UpdateScheduledTaskRequest(**fields) + + +async def _create(service, **overrides): + return await _call(router_module.create_scheduled_task, body=_create_body(**overrides), service=service) + + +class TestCreate: + @pytest.mark.asyncio + async def test_a_cron_task_is_created_and_rendered(self, service, as_user): + created = await _create(service) + assert created.title == "Daily summary" + assert created.schedule_type == "cron" + assert created.schedule_spec == {"cron": "0 9 * * *"} + assert created.status == "enabled" + assert created.next_run_at is not None + + @pytest.mark.asyncio + async def test_the_response_carries_no_server_owned_fields(self, service, as_user): + """The pre-migration router returned the ORM row, leaking `user_id`, + `overlap_policy` and `assistant_id` to every client.""" + rendered = (await _create(service)).model_dump() + assert {"user_id", "assistant_id", "overlap_policy", "lease_owner", "lease_expires_at"} & set(rendered) == set() + + @pytest.mark.asyncio + async def test_a_fresh_thread_task_needs_no_thread_id(self, service, as_user): + created = await _create(service, context_mode="fresh_thread_per_run", thread_id=None) + assert created.thread_id is None + + @pytest.mark.asyncio + async def test_reuse_thread_binds_the_thread(self, service, as_user): + created = await _create(service, context_mode="reuse_thread", thread_id=THREAD) + assert created.thread_id == THREAD + + @pytest.mark.asyncio + async def test_reuse_thread_without_a_thread_id_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, context_mode="reuse_thread", thread_id=None) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_reuse_of_an_unknown_thread_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, context_mode="reuse_thread", thread_id="thread-nope") + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_reuse_of_someone_elses_thread_is_also_404(self, service, as_user): + """Indistinguishable from "does not exist" on purpose: telling them + apart would let a caller probe for threads they cannot see.""" + as_user(OTHER_USER) + with pytest.raises(HTTPException) as caught: + await _create(service, context_mode="reuse_thread", thread_id=THREAD) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_an_unknown_schedule_type_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, schedule_type="teleport") + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_a_cron_without_its_key_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, schedule_type="cron", schedule_spec={}) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_a_malformed_cron_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, schedule_type="cron", schedule_spec={"cron": "0 9 * *"}) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_an_unknown_timezone_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, timezone="Mars/Olympus_Mons") + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_a_once_schedule_too_close_to_now_is_422(self, service, as_user): + """`min_once_delay_seconds` is operator policy, and it reaches the + aggregate through SchedulePolicy rather than being re-checked here.""" + soon = (datetime.now(UTC) + timedelta(seconds=5)).isoformat() + with pytest.raises(HTTPException) as caught: + await _create(service, schedule_type="once", schedule_spec={"run_at": soon}) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_a_once_schedule_in_the_past_is_422(self, service, as_user): + past = (datetime.now(UTC) - timedelta(days=1)).isoformat() + with pytest.raises(HTTPException) as caught: + await _create(service, schedule_type="once", schedule_spec={"run_at": past}) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_an_unknown_context_mode_is_422(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _create(service, context_mode="telepathy") + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_an_anonymous_caller_is_401(self, service, as_user): + as_user(None) + with pytest.raises(HTTPException) as caught: + await _create(service) + assert caught.value.status_code == 401 + + +class TestRead: + @pytest.mark.asyncio + async def test_a_task_is_fetched_by_id(self, service, as_user): + created = await _create(service) + fetched = await _call(router_module.get_scheduled_task, task_id=created.id, service=service) + assert fetched.id == created.id + + @pytest.mark.asyncio + async def test_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call(router_module.get_scheduled_task, task_id="task-nope", service=service) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_another_users_task_is_404(self, service, as_user): + created = await _create(service) + as_user(OTHER_USER) + with pytest.raises(HTTPException) as caught: + await _call(router_module.get_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_listing_returns_the_users_tasks(self, service, as_user): + await _create(service, title="One") + await _create(service, title="Two") + listed = await _call(router_module.list_scheduled_tasks, service=service) + assert {task.title for task in listed} == {"One", "Two"} + + @pytest.mark.asyncio + async def test_an_anonymous_listing_is_empty_not_401(self, service, as_user): + """Unchanged from the pre-migration router: this endpoint has always + answered an unauthenticated GET with an empty list.""" + as_user(None) + assert await _call(router_module.list_scheduled_tasks, service=service) == [] + + @pytest.mark.asyncio + async def test_thread_listing_filters_by_thread(self, service, as_user): + await _create(service, context_mode="reuse_thread", thread_id=THREAD, title="Bound") + await _create(service, title="Unbound") + listed = await _call(router_module.list_thread_scheduled_tasks, thread_id=THREAD, service=service) + assert [task.title for task in listed] == ["Bound"] + + +class TestToCommand: + """Transformation ① of the chain: wire shape -> command, owned by the + request models. Identity is injected as a parameter, never read off the + wire; the wire's `None` = "not supplied" convention becomes the command's + unambiguous `UNSET` here and nowhere else.""" + + def test_identity_is_injected_not_read_off_the_wire(self): + assert "user_id" not in router_module.CreateScheduledTaskRequest.model_fields + assert "user_id" not in router_module.UpdateScheduledTaskRequest.model_fields + cmd = _create_body().to_command(USER) + assert cmd.user_id == USER + + def test_create_carries_the_parsed_value_object(self): + cmd = _create_body().to_command(USER) + assert cmd.schedule.cron == "0 9 * * *" + assert cmd.title == "Daily summary" + + def test_update_translates_wire_none_to_unset(self): + cmd = _update_body(title="Renamed").to_command("task-1", USER, None) + assert cmd.title == "Renamed" + assert cmd.prompt is UNSET + assert cmd.schedule is UNSET + assert cmd.context is UNSET + + def test_update_completes_composite_fields_from_the_current_task(self): + current = ScheduledTask.create( + user_id=USER, + title="t", + prompt="p", + schedule=ScheduleSpec.cron_schedule("0 9 * * *", "Asia/Shanghai"), + context_mode="fresh_thread_per_run", + thread_id=None, + now=datetime(2026, 7, 27, tzinfo=UTC), + policy=SchedulePolicy(), + ) + cmd = _update_body(schedule_spec={"cron": "30 2 * * *"}).to_command(current.task_id, USER, current) + assert cmd.schedule.cron == "30 2 * * *" + assert cmd.schedule.timezone == "Asia/Shanghai", "the omitted zone comes from the current value object" + + cmd = _update_body(thread_id=THREAD).to_command(current.task_id, USER, current) + assert str(cmd.context.context_mode) == "fresh_thread_per_run", "the omitted mode comes from the current task" + assert cmd.context.thread_id == THREAD + + +class TestUpdate: + @pytest.mark.asyncio + async def test_a_title_is_updated_in_place(self, service, as_user): + created = await _create(service) + updated = await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(title="Renamed"), + service=service, + ) + assert updated.title == "Renamed" + assert updated.schedule_spec == created.schedule_spec + + @pytest.mark.asyncio + async def test_a_schedule_spec_change_keeps_the_existing_timezone(self, service, as_user): + """Only one of the three schedule parts was supplied; the router reads + the task to complete the value object rather than the service being + handed a partial one.""" + created = await _create(service, timezone="Asia/Shanghai") + updated = await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(schedule_spec={"cron": "30 2 * * *"}), + service=service, + ) + assert updated.schedule_spec == {"cron": "30 2 * * *"} + assert updated.timezone == "Asia/Shanghai" + + @pytest.mark.asyncio + async def test_a_timezone_change_keeps_the_existing_spec(self, service, as_user): + created = await _create(service) + updated = await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(timezone="Asia/Shanghai"), + service=service, + ) + assert updated.schedule_spec == created.schedule_spec + assert updated.timezone == "Asia/Shanghai" + + @pytest.mark.asyncio + async def test_a_timezone_change_on_a_once_task_keeps_the_same_instant(self, service, as_user): + """The `once` half of the fallback above. + + The omitted `run_at` is read straight off the current value object, and + the stored instant is already offset-aware, so re-zoning the schedule + must relabel it without moving it. + """ + created = await _create( + service, + schedule_type="once", + schedule_spec={"run_at": "2026-08-01T09:00:00+00:00"}, + timezone="UTC", + ) + updated = await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(timezone="Asia/Shanghai"), + service=service, + ) + assert updated.schedule_spec == created.schedule_spec + assert updated.timezone == "Asia/Shanghai" + assert updated.next_run_at == created.next_run_at + + @pytest.mark.asyncio + async def test_a_running_task_cannot_be_updated(self, service, tasks, as_user): + """Red line: the mutability gate lives in the aggregate now, and the + router only maps it onto 409.""" + created = await _create(service) + stored = await tasks.get(created.id, user_id=USER) + await tasks.save(stored.__class__(**{**stored.__dict__, "status": TaskStatus.RUNNING})) + with pytest.raises(HTTPException) as caught: + await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(title="Renamed"), + service=service, + ) + assert caught.value.status_code == 409 + + @pytest.mark.asyncio + async def test_updating_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call( + router_module.update_scheduled_task, + task_id="task-nope", + body=_update_body(title="Renamed"), + service=service, + ) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_a_malformed_new_schedule_is_422(self, service, as_user): + created = await _create(service) + with pytest.raises(HTTPException) as caught: + await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(schedule_spec={"cron": "0 9 * *"}), + service=service, + ) + assert caught.value.status_code == 422 + + @pytest.mark.asyncio + async def test_a_terminal_once_task_pushed_into_the_future_is_rearmed(self, service, tasks, as_user): + """Red line: without re-arming, the API answers 200 with a + `next_run_at` that can never fire, because claiming only admits + `enabled` rows.""" + future = (datetime.now(UTC) + timedelta(days=1)).isoformat() + created = await _create(service, schedule_type="once", schedule_spec={"run_at": future}) + stored = await tasks.get(created.id, user_id=USER) + await tasks.save(stored.__class__(**{**stored.__dict__, "status": TaskStatus.FAILED})) + + later = (datetime.now(UTC) + timedelta(days=2)).isoformat() + updated = await _call( + router_module.update_scheduled_task, + task_id=created.id, + body=_update_body(schedule_spec={"run_at": later}), + service=service, + ) + assert updated.status == "enabled" + assert updated.next_run_at is not None + + +class TestPauseResume: + @pytest.mark.asyncio + async def test_pause_then_resume(self, service, as_user): + created = await _create(service) + paused = await _call(router_module.pause_scheduled_task, task_id=created.id, service=service) + assert paused.status == "paused" + resumed = await _call(router_module.resume_scheduled_task, task_id=created.id, service=service) + assert resumed.status == "enabled" + + @pytest.mark.asyncio + async def test_pausing_a_running_task_is_409(self, service, tasks, as_user): + created = await _create(service) + stored = await tasks.get(created.id, user_id=USER) + await tasks.save(stored.__class__(**{**stored.__dict__, "status": TaskStatus.RUNNING})) + with pytest.raises(HTTPException) as caught: + await _call(router_module.pause_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 409 + + @pytest.mark.asyncio + async def test_pausing_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call(router_module.pause_scheduled_task, task_id="task-nope", service=service) + assert caught.value.status_code == 404 + + +class TestTrigger: + """Red line: this endpoint answers exactly 409 / 502 / 200, and the success + body is `{"id": ..., "triggered": true}`.""" + + @pytest.mark.asyncio + async def test_a_launched_trigger_is_200_with_the_documented_body(self, service, as_user): + created = await _create(service) + response = await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + assert response.model_dump() == {"id": created.id, "triggered": True} + + @pytest.mark.asyncio + async def test_a_busy_thread_is_409(self, service, launcher, as_user): + created = await _create(service) + launcher.fail_with = ThreadBusyError("thread already has an active run") + with pytest.raises(HTTPException) as caught: + await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 409 + + @pytest.mark.asyncio + async def test_an_active_run_for_the_same_task_is_409(self, service, runs, as_user): + """The other route to a conflict: the task's single active slot is + already taken, which the service reports without calling the launcher + at all.""" + created = await _create(service) + await runs.add( + ScheduledRun( + record_id="task-run-existing", + task_id=created.id, + thread_id="thread-x", + scheduled_for=datetime.now(UTC), + trigger=TriggerKind.SCHEDULED, + status=RunStatus.RUNNING, + ) + ) + with pytest.raises(HTTPException) as caught: + await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 409 + + @pytest.mark.asyncio + async def test_a_launch_failure_is_502(self, service, launcher, as_user): + """502, not 500: the failure is downstream of this API and the task + itself is intact.""" + created = await _create(service) + launcher.fail_with = LaunchFailedError("run backend unavailable") + with pytest.raises(HTTPException) as caught: + await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 502 + + @pytest.mark.asyncio + async def test_triggering_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call(router_module.trigger_scheduled_task, task_id="task-nope", service=service) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_a_paused_task_can_still_be_triggered(self, service, as_user): + """Manual dispatch is deliberately allowed while paused, and leaves the + task paused.""" + created = await _create(service) + await _call(router_module.pause_scheduled_task, task_id=created.id, service=service) + response = await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + assert response.triggered is True + after = await _call(router_module.get_scheduled_task, task_id=created.id, service=service) + assert after.status == "paused" + + +class TestDelete: + @pytest.mark.asyncio + async def test_a_task_is_deleted(self, service, as_user): + created = await _create(service) + response = await _call(router_module.delete_scheduled_task, task_id=created.id, service=service) + assert response.model_dump() == {"id": created.id, "deleted": True} + + @pytest.mark.asyncio + async def test_deleting_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call(router_module.delete_scheduled_task, task_id="task-nope", service=service) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_deleting_another_users_task_is_404(self, service, as_user): + created = await _create(service) + as_user(OTHER_USER) + with pytest.raises(HTTPException) as caught: + await _call(router_module.delete_scheduled_task, task_id=created.id, service=service) + assert caught.value.status_code == 404 + + +class TestRunHistory: + @pytest.mark.asyncio + async def test_runs_are_listed_for_an_owned_task(self, service, as_user): + created = await _create(service) + await _call(router_module.trigger_scheduled_task, task_id=created.id, service=service) + listed = await _call(router_module.list_scheduled_task_runs, task_id=created.id, service=service) + assert len(listed) == 1 + assert listed[0].task_id == created.id + assert listed[0].trigger == "manual" + + @pytest.mark.asyncio + async def test_history_of_an_unknown_task_is_404(self, service, as_user): + with pytest.raises(HTTPException) as caught: + await _call(router_module.list_scheduled_task_runs, task_id="task-nope", service=service) + assert caught.value.status_code == 404 + + @pytest.mark.asyncio + async def test_history_of_another_users_task_is_404(self, service, as_user): + """Ownership is checked on the parent task, so run history cannot be + read sideways.""" + created = await _create(service) + as_user(OTHER_USER) + with pytest.raises(HTTPException) as caught: + await _call(router_module.list_scheduled_task_runs, task_id=created.id, service=service) + assert caught.value.status_code == 404 + + +class TestErrorMapping: + def test_every_mapped_error_is_a_schedule_error(self): + from deerflow.domain.schedule.exceptions import ScheduleError + + assert all(issubclass(error, ScheduleError) for error in router_module._STATUS_BY_ERROR) + + @pytest.mark.asyncio + async def test_an_unclassified_domain_error_is_not_swallowed(self): + """A new domain error is a new protocol decision. Letting it through + surfaces as a 500 that has to be classified, rather than shipping as + whatever 4xx happened to be nearest.""" + from deerflow.domain.schedule.exceptions import ScheduleError + + class NewDomainError(ScheduleError): + pass + + @router_module._map_domain_errors + async def handler(): + raise NewDomainError("something new") + + with pytest.raises(NewDomainError): + await handler() + + @pytest.mark.asyncio + async def test_a_mapped_error_keeps_its_message_as_the_detail(self): + from deerflow.domain.schedule.exceptions import TaskNotFoundError + + @router_module._map_domain_errors + async def handler(): + raise TaskNotFoundError("Scheduled task not found") + + with pytest.raises(HTTPException) as caught: + await handler() + assert caught.value.detail == "Scheduled task not found" diff --git a/backend/tests/test_schedule_run_completion.py b/backend/tests/test_schedule_run_completion.py new file mode 100644 index 000000000..c029801b5 --- /dev/null +++ b/backend/tests/test_schedule_run_completion.py @@ -0,0 +1,176 @@ +"""Tests for the run-completion driving adapter. + +Every run in the process reaches the run runtime's completion callback, so +this adapter's first job is deciding which of them the schedule context has +any business with -- the filtering the legacy hook did inline, with four guard +clauses and an `if/elif` chain over runtime strings. A run that is not a +scheduled execution produces no call at all, which is why the service carries +no guard clauses of its own and never imports the run runtime. + +Driven through `__call__` rather than a conversion function: "was the use case +invoked, and with what" is the behaviour, and the previous split -- a pure +converter here plus the invoking closure in the composition root -- left that +second half untested, since the composition root is not where behaviour is +asserted. +""" + +from __future__ import annotations + +from datetime import UTC, datetime + +import pytest + +from app.adapters.schedule.run_completion import ScheduleRunCompletionListener +from deerflow.domain.schedule.model import RunStatus +from deerflow.domain.schedule.ports import RunOutcome +from deerflow.runtime.runs.manager import RunRecord +from deerflow.runtime.runs.schemas import DisconnectMode +from deerflow.runtime.runs.schemas import RunStatus as RuntimeRunStatus + +pytestmark = pytest.mark.asyncio + +TASK_METADATA = { + "scheduled_task_id": "task-1", + "scheduled_task_run_id": "rec-1", + "scheduled_trigger": "scheduled", +} + +_DEFAULT = object() + + +def _record(*, status, metadata=_DEFAULT, user_id="user-1", error=None, run_id="run-1"): + return RunRecord( + run_id=run_id, + thread_id="thread-1", + assistant_id=None, + status=status, + on_disconnect=DisconnectMode.continue_, + metadata=TASK_METADATA if metadata is _DEFAULT else metadata, + user_id=user_id, + error=error, + ) + + +class _RecordingService: + """Stands in for `ScheduleService`, capturing what the use case received.""" + + def __init__(self) -> None: + self.calls: list[tuple[RunOutcome, datetime]] = [] + + async def handle_run_completion(self, outcome: RunOutcome, *, now: datetime) -> None: + self.calls.append((outcome, now)) + + +async def _delivered(record) -> RunOutcome | None: + """The outcome the service was called with, or None when it was not.""" + service = _RecordingService() + await ScheduleRunCompletionListener(service)(record) + return service.calls[0][0] if service.calls else None + + +class TestTerminalStatusMapping: + async def test_success(self): + assert await _delivered(_record(status=RuntimeRunStatus.success)) == RunOutcome( + task_id="task-1", + record_id="rec-1", + run_id="run-1", + user_id="user-1", + status=RunStatus.SUCCESS, + error=None, + ) + + async def test_a_successful_run_carries_no_error_even_if_the_record_has_one(self): + """A stale `error` on a successful record must not be written back as + the task's `last_error`; success is success.""" + outcome = await _delivered(_record(status=RuntimeRunStatus.success, error="stale")) + assert outcome is not None and outcome.error is None + + @pytest.mark.parametrize("status", [RuntimeRunStatus.error, RuntimeRunStatus.timeout]) + async def test_error_and_timeout_both_become_failed(self, status): + outcome = await _delivered(_record(status=status, error="boom")) + assert outcome is not None + assert outcome.status is RunStatus.FAILED + assert outcome.error == "boom" + + async def test_interrupted_is_distinct_from_failed(self): + """Red line: a cancel or same-thread takeover is not an execution + failure -- the task must end CANCELLED, and only INTERRUPTED gets it + there.""" + outcome = await _delivered(_record(status=RuntimeRunStatus.interrupted)) + assert outcome is not None and outcome.status is RunStatus.INTERRUPTED + + async def test_an_interrupt_without_an_error_gets_an_explanatory_one(self): + outcome = await _delivered(_record(status=RuntimeRunStatus.interrupted, error=None)) + assert outcome is not None and outcome.error == "run was interrupted before completion" + + async def test_an_interrupts_own_error_is_preserved(self): + outcome = await _delivered(_record(status=RuntimeRunStatus.interrupted, error="cancelled by user")) + assert outcome is not None and outcome.error == "cancelled by user" + + +class TestFilteredOut: + """None of these reach the service at all -- not as an error, not as a + no-op call it has to recognise.""" + + @pytest.mark.parametrize("status", [RuntimeRunStatus.pending, RuntimeRunStatus.running]) + async def test_a_non_terminal_run_is_ignored(self, status): + assert await _delivered(_record(status=status)) is None + + async def test_an_ordinary_chat_run_is_ignored(self): + """Every run in the process reaches this callback; only scheduled + executions carry the metadata that makes one ours.""" + assert await _delivered(_record(status=RuntimeRunStatus.success, metadata={})) is None + + @pytest.mark.parametrize( + "metadata", + [ + {"scheduled_task_run_id": "rec-1"}, + {"scheduled_task_id": "task-1"}, + {"scheduled_task_id": "task-1", "scheduled_task_run_id": None}, + {"scheduled_task_id": 7, "scheduled_task_run_id": "rec-1"}, + {"scheduled_task_id": "task-1", "scheduled_task_run_id": 7}, + ], + ) + async def test_half_or_mistyped_metadata_is_ignored(self, metadata): + """Both ids are required and both must be strings -- `metadata` is a + free-form dict a client can influence, so its shape is not assumed.""" + assert await _delivered(_record(status=RuntimeRunStatus.success, metadata=metadata)) is None + + @pytest.mark.parametrize("user_id", [None, ""]) + async def test_a_run_without_an_owner_is_ignored(self, user_id): + """Every task read is scoped by user_id; without one there is no + authorized way to look the task up.""" + assert await _delivered(_record(status=RuntimeRunStatus.success, user_id=user_id)) is None + + async def test_none_metadata_is_ignored(self): + """`metadata` is typed as a dict but the legacy hook defended against + `None`; keep that defence rather than rediscovering it in production.""" + assert await _delivered(_record(status=RuntimeRunStatus.success, metadata=None)) is None + + async def test_the_service_is_left_entirely_alone(self): + """Explicitly: not called, rather than called with something falsy. + + This half had no coverage while the invoking closure lived in the + composition root. + """ + service = _RecordingService() + await ScheduleRunCompletionListener(service)(_record(status=RuntimeRunStatus.running)) + assert service.calls == [] + + +class TestUseCaseInvocation: + async def test_the_completion_is_stamped_with_the_current_instant(self): + service = _RecordingService() + before = datetime.now(UTC) + await ScheduleRunCompletionListener(service)(_record(status=RuntimeRunStatus.success)) + after = datetime.now(UTC) + + assert len(service.calls) == 1 + _, now = service.calls[0] + assert now.tzinfo is not None, "the domain is tz-aware throughout" + assert before <= now <= after + + async def test_one_finished_run_produces_exactly_one_call(self): + service = _RecordingService() + await ScheduleRunCompletionListener(service)(_record(status=RuntimeRunStatus.success)) + assert len(service.calls) == 1 diff --git a/backend/tests/test_schedule_run_launcher.py b/backend/tests/test_schedule_run_launcher.py new file mode 100644 index 000000000..90768427e --- /dev/null +++ b/backend/tests/test_schedule_run_launcher.py @@ -0,0 +1,149 @@ +"""Contract tests for the run-launcher anti-corruption layer. + +The `RunLauncher` port allows exactly two exceptions to escape -- `ThreadBusyError` +and `LaunchFailedError` -- and the domain branches on the difference: a busy +thread on a scheduled dispatch is a *skipped* occurrence, a genuine failure is a +*recorded* one. Everything the Gateway can raise is therefore classified here, +and this file is what pins that classification. + +That translation is the whole point of the adapter: it is what lets +`app/scheduler/service.py`'s `from fastapi import HTTPException` disappear +without the busy/failed distinction disappearing with it. + +There is deliberately no `isinstance(launcher, RunLauncher)` assertion. The +adapter inherits the port explicitly, which makes that check trivially true -- +and worse than useless: inheritance is exactly what turns a misspelled method +into a silent inherited `...` body returning `None`. Calling every port method +and asserting on what it returns, as this file does, is what actually catches +that. +""" + +from __future__ import annotations + +import asyncio + +import pytest +from fastapi import HTTPException + +from app.adapters.schedule.run_launcher import GatewayRunLauncher +from deerflow.domain.schedule.exceptions import LaunchFailedError, ThreadBusyError +from deerflow.domain.schedule.ports import LaunchedRun +from deerflow.runtime import ConflictError + +LAUNCH_KWARGS = { + "thread_id": "thread-1", + "assistant_id": "assistant-1", + "prompt": "do the thing", + "owner_user_id": "user-1", + "metadata": {"scheduled_task_id": "task-1", "scheduled_task_run_id": "rec-1", "scheduled_trigger": "scheduled"}, +} + + +def _launcher_returning(payload): + async def launch_run(**kwargs): + launch_run.calls.append(kwargs) + return payload + + launch_run.calls = [] + return GatewayRunLauncher(launch_run), launch_run + + +def _launcher_raising(exc): + async def launch_run(**kwargs): + raise exc + + return GatewayRunLauncher(launch_run) + + +class TestSuccessfulLaunch: + @pytest.mark.asyncio + async def test_returns_what_the_gateway_reported(self): + launcher, _ = _launcher_returning({"run_id": "run-9", "thread_id": "thread-other"}) + result = await launcher.launch(**LAUNCH_KWARGS) + assert result == LaunchedRun(run_id="run-9", thread_id="thread-other") + + @pytest.mark.asyncio + async def test_echoes_the_gateways_thread_not_the_requested_one(self): + """`LaunchedRun.thread_id` is documented as what actually ran, so the + adapter must not substitute the thread it asked for.""" + launcher, _ = _launcher_returning({"run_id": "run-9", "thread_id": "thread-substituted"}) + result = await launcher.launch(**LAUNCH_KWARGS) + assert result.thread_id == "thread-substituted" + + @pytest.mark.asyncio + async def test_carries_every_argument_through_untouched(self): + launcher, spy = _launcher_returning({"run_id": "r", "thread_id": "t"}) + await launcher.launch(**LAUNCH_KWARGS) + assert spy.calls == [LAUNCH_KWARGS] + + @pytest.mark.asyncio + async def test_a_malformed_gateway_payload_is_a_launch_failure(self): + """A missing id is not a busy thread -- it is the run path breaking its + own contract, which the domain records as a failure.""" + launcher, _ = _launcher_returning({"thread_id": "t"}) + with pytest.raises(LaunchFailedError): + await launcher.launch(**LAUNCH_KWARGS) + + +class TestBusyThreadTranslation: + @pytest.mark.asyncio + async def test_conflict_error_becomes_thread_busy(self): + launcher = _launcher_raising(ConflictError("thread already has an active run")) + with pytest.raises(ThreadBusyError): + await launcher.launch(**LAUNCH_KWARGS) + + @pytest.mark.asyncio + async def test_http_409_becomes_thread_busy(self): + """`start_run` rejects a busy thread as an HTTP 409 rather than a + ConflictError on some paths; both mean the same thing here.""" + launcher = _launcher_raising(HTTPException(status_code=409, detail="thread is busy")) + with pytest.raises(ThreadBusyError): + await launcher.launch(**LAUNCH_KWARGS) + + @pytest.mark.asyncio + async def test_the_cause_survives_in_the_message(self): + launcher = _launcher_raising(ConflictError("thread already has an active run")) + with pytest.raises(ThreadBusyError, match="thread already has an active run"): + await launcher.launch(**LAUNCH_KWARGS) + + @pytest.mark.asyncio + async def test_http_409_message_is_the_detail_not_the_repr(self): + launcher = _launcher_raising(HTTPException(status_code=409, detail="thread is busy")) + with pytest.raises(ThreadBusyError, match="^thread is busy$"): + await launcher.launch(**LAUNCH_KWARGS) + + +class TestFailureTranslation: + @pytest.mark.asyncio + @pytest.mark.parametrize("status_code", [400, 404, 422, 500, 502]) + async def test_any_other_http_error_is_a_launch_failure(self, status_code): + launcher = _launcher_raising(HTTPException(status_code=status_code, detail="nope")) + with pytest.raises(LaunchFailedError): + await launcher.launch(**LAUNCH_KWARGS) + + @pytest.mark.asyncio + async def test_an_arbitrary_exception_is_a_launch_failure(self): + launcher = _launcher_raising(RuntimeError("database is on fire")) + with pytest.raises(LaunchFailedError, match="database is on fire"): + await launcher.launch(**LAUNCH_KWARGS) + + @pytest.mark.asyncio + async def test_the_original_exception_is_chained(self): + """The domain only needs the two categories, but an operator reading a + log needs the real traceback.""" + original = RuntimeError("database is on fire") + launcher = _launcher_raising(original) + with pytest.raises(LaunchFailedError) as caught: + await launcher.launch(**LAUNCH_KWARGS) + assert caught.value.__cause__ is original + + +class TestCancellationIsNotSwallowed: + @pytest.mark.asyncio + async def test_cancelled_error_propagates(self): + """`CancelledError` is shutdown control flow, not a launch outcome. + Translating it to LaunchFailedError would record a spurious failure and + break cooperative cancellation of the poll loop.""" + launcher = _launcher_raising(asyncio.CancelledError()) + with pytest.raises(asyncio.CancelledError): + await launcher.launch(**LAUNCH_KWARGS) diff --git a/backend/tests/test_schedule_service.py b/backend/tests/test_schedule_service.py new file mode 100644 index 000000000..1890b0d97 --- /dev/null +++ b/backend/tests/test_schedule_service.py @@ -0,0 +1,905 @@ +"""Use-case tests for ScheduleService, run entirely on in-memory doubles. + +This file is the acceptance criterion of the hexagonal migration: the complete +scheduled-task lifecycle runs here with no HTTP, no database, and no run +runtime -- if any of that were still reachable from the domain, these tests +could not exist. + +The dispatch cases mirror the pre-migration behaviour deliberately. Where a +comment says two paths must be indistinguishable, that is a regression the +tests are here to catch, not a description of the implementation. +""" + +from __future__ import annotations + +from dataclasses import replace +from datetime import UTC, datetime, timedelta + +import pytest +from schedule_fakes import ( + FakeRunLauncher, + FakeThreadLookup, + InMemoryScheduledRunRepository, + InMemoryScheduledTaskRepository, +) + +from deerflow.domain.schedule.commands import ( + UNSET, + ContextChange, + CreateScheduledTask, + DeleteTask, + PauseTask, + ResumeTask, + TriggerTask, + UnsetType, + UpdateScheduledTask, +) +from deerflow.domain.schedule.exceptions import ( + InvalidScheduleError, + LaunchFailedError, + TaskNotFoundError, + TaskNotMutableError, + ThreadBusyError, + ThreadNotFoundError, +) +from deerflow.domain.schedule.model import ( + ContextMode, + DispatchOutcome, + RunStatus, + SchedulePolicy, + ScheduleSpec, + TaskStatus, + TriggerKind, +) +from deerflow.domain.schedule.ports import RunOutcome +from deerflow.domain.schedule.service import ScheduleService + +pytestmark = pytest.mark.asyncio + +NOW = datetime(2026, 7, 27, 12, 0, tzinfo=UTC) +CRON = ScheduleSpec.cron_schedule("0 9 * * *", "UTC") +POLICY = SchedulePolicy(min_once_delay_seconds=60, max_concurrent_runs=3, lease_seconds=120) + + +def once_spec(*, after_seconds: int = 3600) -> ScheduleSpec: + return ScheduleSpec.once_at(NOW + timedelta(seconds=after_seconds), "UTC") + + +class _BlindRunRepo(InMemoryScheduledRunRepository): + """`has_active` always misses. + + Reproduces the TOCTOU window the fast path cannot close: a concurrent + dispatch inserted its record after this one looked. The active-slot + rejection on `add` is then the only thing standing in the way. + """ + + async def has_active(self, _task_id: str) -> bool: + return False + + +class _LaunchWritesMidReadTaskRepo(InMemoryScheduledTaskRepository): + """`record_launch` commits between the completion hook's read and its write. + + That ordering is not exotic -- `dispatch_task` performs its two bookkeeping + writes after the launch returns, and a run that fails fast reaches the hook + in between. Modelling it here rather than with real concurrency keeps the + window deterministic. + """ + + def __init__(self, *, launch_next_run_at: datetime) -> None: + super().__init__() + self._launch_next_run_at = launch_next_run_at + self._armed = False + + def arm(self) -> None: + """Fire the interleaving on the next read, once.""" + self._armed = True + + async def get(self, task_id: str, *, user_id: str): + task = await super().get(task_id, user_id=user_id) + if self._armed: + self._armed = False + await self.record_launch( + task_id, + status=TaskStatus.ENABLED, + next_run_at=self._launch_next_run_at, + last_run_at=NOW, + last_run_id="run-1", + last_thread_id="thread-1", + last_error=None, + increment_run_count=True, + protect_terminal=True, + ) + return task + + +def make_service( + *, + tasks: InMemoryScheduledTaskRepository | None = None, + runs: InMemoryScheduledRunRepository | None = None, + launcher: FakeRunLauncher | None = None, + threads: FakeThreadLookup | None = None, + policy: SchedulePolicy = POLICY, +) -> ScheduleService: + return ScheduleService( + tasks=tasks if tasks is not None else InMemoryScheduledTaskRepository(), + runs=runs if runs is not None else InMemoryScheduledRunRepository(), + launcher=launcher if launcher is not None else FakeRunLauncher(), + threads=threads if threads is not None else FakeThreadLookup(), + policy=policy, + ) + + +async def create_cron_task(service: ScheduleService, *, schedule: ScheduleSpec = CRON): + return await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="Daily summary", + prompt="summarize", + schedule=schedule, + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + + +# ==================================================================== lifecycle + + +class TestFullLifecycle: + async def test_create_claim_dispatch_overlap_complete_pause_delete(self): + """The acceptance criterion: the whole feature, zero IO. + + Every step below is a real use case going through real domain rules -- + only the four ports are doubles. + """ + tasks = InMemoryScheduledTaskRepository() + runs = InMemoryScheduledRunRepository() + launcher = FakeRunLauncher() + service = make_service(tasks=tasks, runs=runs, launcher=launcher) + + # -- create --------------------------------------------------------- + task = await create_cron_task(service) + assert task.status is TaskStatus.ENABLED + assert task.next_run_at == datetime(2026, 7, 28, 9, 0, tzinfo=UTC) + + # -- become due, get claimed and dispatched -------------------------- + due_at = task.next_run_at + timedelta(seconds=1) + results = await service.run_once(now=due_at) + assert [r.outcome for r in results] == [DispatchOutcome.LAUNCHED] + assert len(launcher.calls) == 1 + launch = launcher.calls[0] + assert launch["prompt"] == "summarize" + assert launch["owner_user_id"] == "user-1" + assert launch["metadata"]["scheduled_task_id"] == task.task_id + assert launch["metadata"]["scheduled_trigger"] == "scheduled" + + after_launch = await service.get_task(task.task_id, user_id="user-1") + assert after_launch.status is TaskStatus.ENABLED, "a cron task stays claimable" + assert after_launch.run_count == 1 + assert after_launch.last_run_id == "run-1" + assert after_launch.next_run_at > due_at + + # -- next occurrence overlaps the still-running one ------------------ + overlap_at = after_launch.next_run_at + timedelta(seconds=1) + overlapped = await service.run_once(now=overlap_at) + assert [r.outcome for r in overlapped] == [DispatchOutcome.SKIPPED] + assert len(launcher.calls) == 1, "the overlapping occurrence must not launch" + + after_skip = await service.get_task(task.task_id, user_id="user-1") + assert after_skip.run_count == 1, "a skip is not an execution" + assert after_skip.last_run_id == after_launch.last_run_id, "bookkeeping carried over" + assert after_skip.status is TaskStatus.ENABLED + # Only a lost one-shot occurrence is worth surfacing; a cron task simply + # waits for its next turn, so the skip must not leave a user-visible + # error behind (service.py:455). The `once` half is covered by + # TestOnceTaskDispatch.test_a_skipped_once_task_is_failed_not_completed. + assert after_skip.last_error is None, "a routine cron overlap is not an error to report" + + # -- the first run finally finishes ---------------------------------- + record = next(r for r in runs.all_runs() if r.status is RunStatus.RUNNING) + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=record.record_id, + run_id="run-1", + user_id="user-1", + status=RunStatus.SUCCESS, + error=None, + ), + now=overlap_at, + ) + assert await runs.count_active() == 0, "the active slot is free again" + + # -- pause / resume --------------------------------------------------- + paused = await service.pause_task(PauseTask(task_id=task.task_id, user_id="user-1")) + assert paused.status is TaskStatus.PAUSED + assert await service.run_once(now=overlap_at + timedelta(days=2)) == [], "a paused task is not claimed" + + resumed = await service.resume_task(ResumeTask(task_id=task.task_id, user_id="user-1")) + assert resumed.status is TaskStatus.ENABLED + + # -- history and delete ---------------------------------------------- + history = await service.list_task_runs(task.task_id, user_id="user-1") + assert sorted(run.status for run in history) == [RunStatus.SKIPPED, RunStatus.SUCCESS] + + await service.delete_task(DeleteTask(task_id=task.task_id, user_id="user-1")) + with pytest.raises(TaskNotFoundError): + await service.get_task(task.task_id, user_id="user-1") + + +# ==================================================================== dispatch + + +class TestDispatchOutcomes: + async def test_launch_records_the_run_and_the_bookkeeping(self): + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await create_cron_task(service) + + result = await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + assert result.outcome is DispatchOutcome.LAUNCHED + assert result.run_id == "run-1" + assert result.error is None + stored = runs.all_runs()[0] + assert stored.status is RunStatus.RUNNING + assert stored.run_id == "run-1" + assert stored.started_at == NOW + + async def test_manual_trigger_against_an_active_run_is_a_conflict_with_no_record(self): + """Nothing was scheduled to happen, so there is no occurrence to + account for -- the caller gets a conflict and the history stays clean.""" + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await create_cron_task(service) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + before = len(runs.all_runs()) + + result = await service.trigger_task(TriggerTask(task_id=task.task_id, user_id="user-1"), now=NOW) + + assert result.outcome is DispatchOutcome.CONFLICT + assert result.record_id is None + assert result.error == "task already has an active run" + assert len(runs.all_runs()) == before, "no history row for a rejected manual trigger" + + async def test_scheduled_overlap_records_a_terminal_tombstone(self): + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await create_cron_task(service) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + result = await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + assert result.outcome is DispatchOutcome.SKIPPED + assert result.error == "skipped: a previous run of this task is still active" + tombstone = next(r for r in runs.all_runs() if r.status is RunStatus.SKIPPED) + assert tombstone.is_active is False, "a queued tombstone would collide with the live run" + assert tombstone.started_at == tombstone.finished_at == NOW + + async def test_launch_failure_is_recorded_as_failed(self): + runs = InMemoryScheduledRunRepository() + launcher = FakeRunLauncher(fail_with=LaunchFailedError("provider exploded")) + service = make_service(runs=runs, launcher=launcher) + task = await create_cron_task(service) + + result = await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + assert result.outcome is DispatchOutcome.FAILED + assert result.run_id is None + assert "provider exploded" in result.error + assert runs.all_runs()[0].status is RunStatus.FAILED + + async def test_a_failed_launch_replaces_the_previous_run_bookkeeping(self): + """A failed launch is still an execution *attempt*, so it overwrites the + launch bookkeeping rather than carrying it over the way a skip does + (contrast `_finalize_skip`, which passes the current values back). + + `record_launch` assigns every field unconditionally, so this is what + `last_run_id=None` in `_fail` actually means for a task that had already + run successfully: the id of the previous, unrelated run must not be left + pointing at a launch that never happened. + """ + launcher = FakeRunLauncher() + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs, launcher=launcher) + task = await create_cron_task(service) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + succeeded = await service.get_task(task.task_id, user_id="user-1") + assert succeeded.last_run_id == "run-1", "precondition: a successful launch was recorded" + + # The slot has to be free, or the next dispatch is an overlap instead. + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=runs.all_runs()[0].record_id, + run_id="run-1", + user_id="user-1", + status=RunStatus.SUCCESS, + error=None, + ), + now=NOW, + ) + launcher.fail_with = LaunchFailedError("provider exploded") + reloaded = await service.get_task(task.task_id, user_id="user-1") + + await service.dispatch_task(reloaded, now=NOW, trigger=TriggerKind.SCHEDULED) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.last_run_id is None, "no run started, so no run id to point at" + assert after.last_error == "provider exploded" + assert after.last_run_at == NOW, "the attempt itself is timestamped" + assert after.run_count == 1, "a failed launch is not a completed execution" + + async def test_busy_thread_on_the_scheduled_path_degrades_to_a_skip(self): + launcher = FakeRunLauncher(fail_with=ThreadBusyError("thread busy")) + service = make_service(launcher=launcher) + task = await create_cron_task(service) + + result = await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + assert result.outcome is DispatchOutcome.SKIPPED + + async def test_busy_thread_on_a_manual_trigger_stays_a_conflict(self): + """Not a skip: the user asked for this one, so it is reported rather + than quietly accounted for. The router maps it to 409, not 502.""" + launcher = FakeRunLauncher(fail_with=ThreadBusyError("thread busy")) + service = make_service(launcher=launcher) + task = await create_cron_task(service) + + result = await service.dispatch_task(task, now=NOW, trigger=TriggerKind.MANUAL) + + assert result.outcome is DispatchOutcome.CONFLICT + assert result.record_id is not None, "the attempt itself is still recorded" + + +class TestConflictCollapse: + """The fast path and the active-slot rejection must be indistinguishable. + + Two concurrent dispatches can both pass `has_active`; whichever loses is + rejected by the repository instead. A caller must not be able to tell which + of the two mechanisms stopped it, or retry behaviour diverges. + """ + + async def _dispatch_second(self, run_repo, trigger: TriggerKind): + # reuse_thread so the execution thread is fixed: a fresh-thread task + # mints a new uuid per dispatch, which would make the two runs differ + # for a reason that has nothing to do with the collapse. + service = make_service(runs=run_repo, threads=FakeThreadLookup({"thread-1": "user-1"})) + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="t", + prompt="p", + schedule=CRON, + context_mode=ContextMode.REUSE_THREAD, + thread_id="thread-1", + ), + now=NOW, + ) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + return await service.dispatch_task(task, now=NOW, trigger=trigger) + + @pytest.mark.parametrize("trigger", [TriggerKind.SCHEDULED, TriggerKind.MANUAL]) + async def test_both_paths_produce_the_same_result(self, trigger): + via_fast_path = await self._dispatch_second(InMemoryScheduledRunRepository(), trigger) + via_slot_rejection = await self._dispatch_second(_BlindRunRepo(), trigger) + + # record_id is freshly generated; every other field must match exactly. + assert replace(via_fast_path, record_id=None) == replace(via_slot_rejection, record_id=None) + + async def test_the_losing_scheduled_dispatch_still_leaves_a_tombstone(self): + runs = _BlindRunRepo() + result = await self._dispatch_second(runs, TriggerKind.SCHEDULED) + assert result.outcome is DispatchOutcome.SKIPPED + assert any(r.status is RunStatus.SKIPPED for r in runs.all_runs()) + + async def test_the_losing_manual_dispatch_leaves_nothing(self): + runs = _BlindRunRepo() + result = await self._dispatch_second(runs, TriggerKind.MANUAL) + assert result.outcome is DispatchOutcome.CONFLICT + assert not any(r.status is RunStatus.SKIPPED for r in runs.all_runs()) + + +class TestOnceTaskDispatch: + async def test_a_once_task_waits_in_running_for_its_completion(self): + """Declaring it complete at launch would stick if the run failed or the + process died before the hook could correct it.""" + service = make_service() + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="one shot", + prompt="go", + schedule=once_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.status is TaskStatus.RUNNING + + async def test_a_skipped_once_task_is_failed_not_completed(self): + """The single occurrence was lost; `completed` would claim an execution + that never happened.""" + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="one shot", + prompt="go", + schedule=once_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + reloaded = await service.get_task(task.task_id, user_id="user-1") + + await service.dispatch_task(reloaded, now=NOW, trigger=TriggerKind.SCHEDULED) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.status is TaskStatus.FAILED + assert after.last_error == "skipped: a previous run of this task is still active" + + +# ==================================================================== run_once + + +class TestRunOnceBudget: + async def test_claims_nothing_when_the_global_budget_is_exhausted(self): + """The cap is on active runs across all tasks, not on one poll's batch: + long runs accumulate across cycles.""" + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs, policy=replace(POLICY, max_concurrent_runs=1)) + first = await create_cron_task(service) + await service.dispatch_task(first, now=NOW, trigger=TriggerKind.SCHEDULED) + + second = await create_cron_task(service) + results = await service.run_once(now=second.next_run_at + timedelta(seconds=1)) + + assert results == [] + + async def test_claims_only_into_the_remaining_budget(self): + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks, policy=replace(POLICY, max_concurrent_runs=2)) + for _ in range(3): + await create_cron_task(service) + + due_at = datetime(2026, 7, 28, 9, 0, 1, tzinfo=UTC) + results = await service.run_once(now=due_at) + + assert len(results) == 2 + assert all(r.outcome is DispatchOutcome.LAUNCHED for r in results) + + async def test_the_claim_is_held_across_the_launch_and_released_after(self): + """Claiming is what makes the task uneditable while it is being + dispatched, so the claim has to be live *at the moment of the launch* -- + not merely taken at some point and released by the end. + + The launch is the only place that ordering is observable, so the + launcher double reads the repository from inside `launch`. Asserting + only on the end state would pass even if the claim were taken after the + run had already started. + """ + tasks = InMemoryScheduledTaskRepository() + observed: dict = {} + + class _ObservingLauncher(FakeRunLauncher): + async def launch(self, **kwargs): + task_id = kwargs["metadata"]["scheduled_task_id"] + observed["lease"] = tasks.lease_of(task_id) + observed["status"] = (await tasks.get(task_id, user_id="user-1")).status + return await super().launch(**kwargs) + + service = make_service(tasks=tasks, launcher=_ObservingLauncher()) + task = await create_cron_task(service) + + await service.run_once(now=task.next_run_at + timedelta(seconds=1)) + + assert observed["status"] is TaskStatus.RUNNING, "the claim marks the task running before the launch" + assert observed["lease"][1] is not None, "and the lease is still held while the run starts" + assert tasks.lease_of(task.task_id) == (None, None), "released only once the dispatch is done" + + +# ==================================================================== completion + + +class TestRunCompletion: + async def _launched_once_task(self): + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="one shot", + prompt="go", + schedule=once_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + record = runs.all_runs()[0] + return service, task, record + + @pytest.mark.parametrize( + ("outcome_status", "expected"), + [ + (RunStatus.SUCCESS, TaskStatus.COMPLETED), + (RunStatus.FAILED, TaskStatus.FAILED), + (RunStatus.INTERRUPTED, TaskStatus.CANCELLED), + ], + ) + async def test_once_task_terminal_mapping(self, outcome_status, expected): + service, task, record = await self._launched_once_task() + + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=record.record_id, + run_id="run-1", + user_id="user-1", + status=outcome_status, + error=None if outcome_status is RunStatus.SUCCESS else "boom", + ), + now=NOW, + ) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.status is expected + + async def test_a_cron_task_keeps_its_status_but_records_the_error(self): + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await create_cron_task(service) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + record = runs.all_runs()[0] + + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=record.record_id, + run_id="run-1", + user_id="user-1", + status=RunStatus.FAILED, + error="boom", + ), + now=NOW, + ) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.status is TaskStatus.ENABLED, "the schedule outlives any single run" + assert after.last_error == "boom" + + async def test_a_task_deleted_mid_flight_still_finalizes_its_run(self): + """A task deleted while its run was in flight has nothing to write back, + and that is not an error -- but the *run* record must still be closed. + + The hook writes the record first and only then reads the task, so + asserting on the record is what keeps that ordering honest; a test that + only checked "no exception" would stay green if the record write were + dropped entirely. + """ + runs = InMemoryScheduledRunRepository() + service = make_service(runs=runs) + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="one shot", + prompt="go", + schedule=once_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + record = runs.all_runs()[0] + await service.delete_task(DeleteTask(task_id=task.task_id, user_id="user-1")) + + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=record.record_id, + run_id="run-1", + user_id="user-1", + status=RunStatus.SUCCESS, + error=None, + ), + now=NOW, + ) + + stored = runs.all_runs()[0] + assert stored.status is RunStatus.SUCCESS, "the execution record is finalized regardless" + assert stored.finished_at == NOW + assert await runs.count_active() == 0, "the active slot is freed" + + async def test_a_cron_task_survives_a_launch_write_landing_mid_completion(self): + """The interleaving `dispatch_task` already admits is possible. + + A run that fails fast reaches this hook while the dispatch path has not + yet written its bookkeeping, so the hook reads a task still carrying the + elapsed `next_run_at`. If the write-back replays that whole snapshot, + the launch path's fresh fire time is rolled back and the task lands in + `running` with no live claim -- a shape neither `claim_due` branch nor + `cancel_stuck_once_tasks` can reach, i.e. permanently unschedulable. + """ + tasks = _LaunchWritesMidReadTaskRepo(launch_next_run_at=NOW + timedelta(days=1)) + runs = InMemoryScheduledRunRepository() + service = make_service(tasks=tasks, runs=runs) + task = await create_cron_task(service) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + record = runs.all_runs()[0] + tasks.arm() + + await service.handle_run_completion( + RunOutcome( + task_id=task.task_id, + record_id=record.record_id, + run_id="run-1", + user_id="user-1", + status=RunStatus.FAILED, + error="boom", + ), + now=NOW, + ) + + after = await service.get_task(task.task_id, user_id="user-1") + assert after.last_error == "boom", "the completion still records its verdict" + assert after.next_run_at == NOW + timedelta(days=1), "the launch path's fire time must survive" + claimed = await tasks.claim_due(now=NOW + timedelta(days=2), lease_seconds=120, limit=10) + assert [t.task_id for t in claimed] == [task.task_id], "the task must remain schedulable" + + +class TestReconcileOnStartup: + async def test_sweeps_orphaned_runs_and_stuck_once_tasks(self): + runs = InMemoryScheduledRunRepository() + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks, runs=runs) + task = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="one shot", + prompt="go", + schedule=once_spec(), + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + await service.dispatch_task(task, now=NOW, trigger=TriggerKind.SCHEDULED) + # The process dies here: the run is active and the once task is parked + # in running with its claim already released. + + stale_runs, stuck_tasks = await service.reconcile_on_startup(error="gateway restarted") + + assert (stale_runs, stuck_tasks) == (1, 1) + assert runs.all_runs()[0].status is RunStatus.INTERRUPTED + after = await service.get_task(task.task_id, user_id="user-1") + assert after.status is TaskStatus.CANCELLED + + +# ==================================================================== CRUD + + +class TestTaskManagement: + async def test_reuse_thread_requires_an_accessible_thread(self): + service = make_service(threads=FakeThreadLookup({"thread-1": "user-1"})) + + created = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="t", + prompt="p", + schedule=CRON, + context_mode=ContextMode.REUSE_THREAD, + thread_id="thread-1", + ), + now=NOW, + ) + assert created.thread_id == "thread-1" + + with pytest.raises(ThreadNotFoundError): + await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-2", + title="t", + prompt="p", + schedule=CRON, + context_mode=ContextMode.REUSE_THREAD, + thread_id="thread-1", + ), + now=NOW, + ) + + async def test_an_invalid_schedule_is_rejected_before_any_write(self): + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks) + + with pytest.raises(InvalidScheduleError): + await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="t", + prompt="p", + schedule=once_spec(after_seconds=10), # inside min_once_delay + context_mode=ContextMode.FRESH_THREAD_PER_RUN, + thread_id=None, + ), + now=NOW, + ) + assert await service.list_tasks("user-1") == [] + + async def test_commands_are_dumb_data(self): + # A command carries intent without validating it: business rules stay + # on the aggregate, so error attribution (a malformed schedule before + # an unknown thread) is owned by the handler's construction order, + # not by the command's own constructor. + cmd = CreateScheduledTask(user_id="u", title="", prompt="", schedule=CRON, context_mode="not-a-mode", thread_id=None) + assert cmd.context_mode == "not-a-mode" + + async def test_unset_is_a_singleton_distinct_from_none(self): + # Three states, not two: an update field is UNSET (leave it alone), + # None can stay a meaningful value elsewhere, and UNSET is falsy so + # it cannot masquerade as a supplied value. + assert UnsetType() is UNSET + cmd = UpdateScheduledTask(task_id="t", user_id="u") + assert cmd.title is UNSET + assert cmd.title is not None + assert not UNSET + + async def test_update_leaves_omitted_fields_alone(self): + service = make_service() + task = await create_cron_task(service) + + updated = await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="user-1", title="renamed"), now=NOW) + + assert updated.title == "renamed" + assert updated.prompt == task.prompt + assert updated.schedule == task.schedule + + async def test_update_with_everything_unset_changes_nothing(self): + service = make_service() + task = await create_cron_task(service) + + updated = await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="user-1"), now=NOW) + + assert updated == task + + async def test_update_can_change_the_prompt(self): + service = make_service() + task = await create_cron_task(service) + + updated = await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="user-1", prompt="new instructions"), now=NOW) + + assert updated.prompt == "new instructions" + assert updated.title == task.title + + async def test_a_task_deleted_mid_update_reports_as_missing(self): + """get_task saw it, save no longer does -- a concurrent delete landed + in between. The caller gets the same not-found it would have got a + moment earlier, rather than a None leaking out.""" + + class _VanishingRepo(InMemoryScheduledTaskRepository): + async def save(self, _task): + return None + + tasks = _VanishingRepo() + service = make_service(tasks=tasks) + task = await create_cron_task(service) + + with pytest.raises(TaskNotFoundError): + await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="user-1", title="x"), now=NOW) + with pytest.raises(TaskNotFoundError): + await service.pause_task(PauseTask(task_id=task.task_id, user_id="user-1")) + + async def test_context_is_changed_through_the_packaged_value(self): + """context_mode and thread_id move together, so they are supplied + together -- which is what lets every other update field use plain + `None` for "not supplied".""" + service = make_service(threads=FakeThreadLookup({"thread-1": "user-1"})) + task = await create_cron_task(service) + + bound = await service.update_scheduled_task( + UpdateScheduledTask(task_id=task.task_id, user_id="user-1", context=ContextChange(ContextMode.REUSE_THREAD, "thread-1")), + now=NOW, + ) + assert bound.context_mode is ContextMode.REUSE_THREAD + assert bound.thread_id == "thread-1" + + unbound = await service.update_scheduled_task( + UpdateScheduledTask(task_id=task.task_id, user_id="user-1", context=ContextChange(ContextMode.FRESH_THREAD_PER_RUN)), + now=NOW, + ) + assert unbound.thread_id is None, "switching to a fresh thread clears the binding" + + async def test_changing_context_to_an_inaccessible_thread_is_rejected(self): + service = make_service(threads=FakeThreadLookup({"thread-1": "someone-else"})) + task = await create_cron_task(service) + + with pytest.raises(ThreadNotFoundError): + await service.update_scheduled_task( + UpdateScheduledTask(task_id=task.task_id, user_id="user-1", context=ContextChange(ContextMode.REUSE_THREAD, "thread-1")), + now=NOW, + ) + + async def test_rescheduling_a_terminal_task_re_arms_it(self): + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks) + task = await create_cron_task(service) + tasks.seed(replace(task, status=TaskStatus.FAILED)) + + updated = await service.update_scheduled_task( + UpdateScheduledTask(task_id=task.task_id, user_id="user-1", schedule=ScheduleSpec.cron_schedule("0 10 * * *", "UTC")), + now=NOW, + ) + + assert updated.status is TaskStatus.ENABLED, "otherwise it would never be claimed again" + + async def test_a_running_task_cannot_be_edited(self): + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks) + task = await create_cron_task(service) + tasks.seed(replace(task, status=TaskStatus.RUNNING)) + + with pytest.raises(TaskNotMutableError): + await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="user-1", title="nope"), now=NOW) + with pytest.raises(TaskNotMutableError): + await service.pause_task(PauseTask(task_id=task.task_id, user_id="user-1")) + + async def test_a_running_task_can_still_be_deleted(self): + """The pre-migration router gated update/pause/resume on this, but not + delete -- that asymmetry is preserved.""" + tasks = InMemoryScheduledTaskRepository() + service = make_service(tasks=tasks) + task = await create_cron_task(service) + tasks.seed(replace(task, status=TaskStatus.RUNNING)) + + await service.delete_task(DeleteTask(task_id=task.task_id, user_id="user-1")) + + @pytest.mark.parametrize("call", ["get", "update", "pause", "resume", "delete", "runs"]) + async def test_another_users_task_is_reported_as_missing(self, call): + service = make_service() + task = await create_cron_task(service) + with pytest.raises(TaskNotFoundError): + if call == "get": + await service.get_task(task.task_id, user_id="intruder") + elif call == "update": + await service.update_scheduled_task(UpdateScheduledTask(task_id=task.task_id, user_id="intruder", title="x"), now=NOW) + elif call == "pause": + await service.pause_task(PauseTask(task_id=task.task_id, user_id="intruder")) + elif call == "resume": + await service.resume_task(ResumeTask(task_id=task.task_id, user_id="intruder")) + elif call == "delete": + await service.delete_task(DeleteTask(task_id=task.task_id, user_id="intruder")) + else: + await service.list_task_runs(task.task_id, user_id="intruder") + + async def test_tasks_are_listed_per_user_and_per_thread(self): + service = make_service(threads=FakeThreadLookup({"thread-1": "user-1"})) + bound = await service.create_scheduled_task( + CreateScheduledTask( + user_id="user-1", + title="bound", + prompt="p", + schedule=CRON, + context_mode=ContextMode.REUSE_THREAD, + thread_id="thread-1", + ), + now=NOW, + ) + await create_cron_task(service) + + assert len(await service.list_tasks("user-1")) == 2 + assert await service.list_tasks("someone-else") == [] + by_thread = await service.list_tasks_by_thread("user-1", "thread-1") + assert [t.task_id for t in by_thread] == [bound.task_id] diff --git a/backend/tests/test_schedule_thread_lookup.py b/backend/tests/test_schedule_thread_lookup.py new file mode 100644 index 000000000..acb71e607 --- /dev/null +++ b/backend/tests/test_schedule_thread_lookup.py @@ -0,0 +1,71 @@ +"""Contract tests for the thread-lookup anti-corruption layer. + +The port asks one question -- "does this thread exist AND may this user use +it?" -- and deliberately answers both halves with a single bool, so a caller +cannot probe for the existence of threads they cannot see. These tests pin that +the adapter does not accidentally widen it back into two answers. +""" + +from __future__ import annotations + +import pytest + +from app.adapters.schedule.thread_lookup import ThreadStoreThreadLookup + + +class _RecordingThreadStore: + """Stands in for `ThreadMetaStore`, recording how it was asked.""" + + def __init__(self, owners: dict[str, str] | None = None) -> None: + self._owners = dict(owners or {}) + self.calls: list[tuple[str, str, bool]] = [] + + async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool: + self.calls.append((thread_id, user_id, require_existing)) + owner = self._owners.get(thread_id) + if owner is None: + # Mirrors the real store: absent rows pass a non-strict check. + return not require_existing + return owner == user_id + + +class TestTheOneQuestion: + @pytest.mark.asyncio + async def test_an_owned_thread_exists_for_its_user(self): + lookup = ThreadStoreThreadLookup(_RecordingThreadStore({"thread-1": "user-1"})) + assert await lookup.exists_for_user("thread-1", "user-1") is True + + @pytest.mark.asyncio + async def test_someone_elses_thread_does_not(self): + lookup = ThreadStoreThreadLookup(_RecordingThreadStore({"thread-1": "user-1"})) + assert await lookup.exists_for_user("thread-1", "user-2") is False + + @pytest.mark.asyncio + async def test_a_missing_thread_does_not(self): + """This is the half `require_existing` buys: without it the store + treats an absent row as accessible, and a task could be bound to a + thread that does not exist.""" + lookup = ThreadStoreThreadLookup(_RecordingThreadStore()) + assert await lookup.exists_for_user("thread-nope", "user-1") is False + + @pytest.mark.asyncio + async def test_missing_and_forbidden_are_indistinguishable(self): + lookup = ThreadStoreThreadLookup(_RecordingThreadStore({"thread-1": "user-1"})) + missing = await lookup.exists_for_user("thread-nope", "user-2") + forbidden = await lookup.exists_for_user("thread-1", "user-2") + assert missing == forbidden is False + + +class TestHowTheStoreIsAsked: + @pytest.mark.asyncio + async def test_require_existing_is_always_set(self): + store = _RecordingThreadStore({"thread-1": "user-1"}) + await ThreadStoreThreadLookup(store).exists_for_user("thread-1", "user-1") + assert store.calls == [("thread-1", "user-1", True)] + + @pytest.mark.asyncio + async def test_the_result_is_a_real_bool(self): + """The port is typed `bool`; a truthy row object leaking through would + satisfy the domain's `if` and still be the wrong contract.""" + store = _RecordingThreadStore({"thread-1": "user-1"}) + assert await ThreadStoreThreadLookup(store).exists_for_user("thread-1", "user-1") is True diff --git a/backend/tests/test_scheduled_task_claims.py b/backend/tests/test_scheduled_task_claims.py deleted file mode 100644 index 83db44628..000000000 --- a/backend/tests/test_scheduled_task_claims.py +++ /dev/null @@ -1,152 +0,0 @@ -from datetime import UTC, datetime, timedelta - -import pytest - -from deerflow.config.database_config import DatabaseConfig -from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config -from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository - - -@pytest.mark.asyncio -async def test_claim_due_tasks_claims_only_due_rows(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - repo = ScheduledTaskRepository(sf) - - due = datetime.now(UTC) - timedelta(minutes=1) - future = datetime.now(UTC) + timedelta(hours=1) - - await repo.create( - task_id="due-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Due", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=due, - ) - await repo.create( - task_id="future-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Future", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=future, - ) - - claimed = await repo.claim_due_tasks( - now=datetime.now(UTC), - lease_owner="worker-1", - lease_seconds=120, - limit=10, - ) - assert [task["id"] for task in claimed] == ["due-1"] - - await close_engine() - - -@pytest.mark.asyncio -async def test_claim_reclaims_task_stuck_in_running_with_expired_lease(tmp_path): - """A task whose claiming process died mid-dispatch must stay reclaimable. - - Regression for the lease dead-end bug: claim flips status to ``running``, - and the old claim query only selected ``status == 'enabled'``, so a crash - between claim and dispatch left the task permanently un-triggerable. - """ - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - repo = ScheduledTaskRepository(sf) - - now = datetime.now(UTC) - due = now - timedelta(minutes=5) - - await repo.create( - task_id="stuck-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Stuck", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=due, - ) - - first_claim = await repo.claim_due_tasks( - now=now, - lease_owner="dead-worker", - lease_seconds=60, - limit=10, - ) - assert first_claim[0]["id"] == "stuck-1" - assert first_claim[0]["status"] == "running" - - # Simulate the claiming process dying: lease expires, status stays "running". - expired_now = now + timedelta(seconds=120) - reclaimed = await repo.claim_due_tasks( - now=expired_now, - lease_owner="new-worker", - lease_seconds=60, - limit=10, - ) - assert [task["id"] for task in reclaimed] == ["stuck-1"] - assert reclaimed[0]["lease_owner"] == "new-worker" - - await close_engine() - - -@pytest.mark.asyncio -async def test_claim_skips_task_with_active_lease(tmp_path): - """A task whose lease has not expired must not be reclaimed.""" - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - repo = ScheduledTaskRepository(sf) - - now = datetime.now(UTC) - due = now - timedelta(minutes=5) - - await repo.create( - task_id="active-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Active", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=due, - ) - - await repo.claim_due_tasks( - now=now, - lease_owner="worker-1", - lease_seconds=300, - limit=10, - ) - - # Lease still valid — second claim within the same process must not re-grab it. - reclaimed = await repo.claim_due_tasks( - now=now + timedelta(seconds=10), - lease_owner="worker-2", - lease_seconds=300, - limit=10, - ) - assert reclaimed == [] - - await close_engine() diff --git a/backend/tests/test_scheduled_task_dispatch_race.py b/backend/tests/test_scheduled_task_dispatch_race.py deleted file mode 100644 index 6dcc01dc0..000000000 --- a/backend/tests/test_scheduled_task_dispatch_race.py +++ /dev/null @@ -1,219 +0,0 @@ -"""Concurrency regression tests for the scheduled-task dispatch TOCTOU. - -``ScheduledTaskService.dispatch_task`` guards the "at most one active run per -task when overlap_policy=skip" invariant with a non-atomic -``has_active_runs`` check followed by a separate ``create(status="queued")`` -insert. Two concurrent dispatches (double-click, client retry, or a manual -trigger racing the poller) can both pass the check and both launch. The fix -makes the database the atomic arbiter via the partial unique index -``uq_scheduled_task_run_active`` (``task_id WHERE status IN -('queued','running')``); the losing insert is translated to the typed -``ActiveScheduledRunConflict`` and collapsed to the same outcome as the -fast-path check. - -These tests drive the REAL ``ScheduledTaskRunRepository`` + ``ScheduledTaskService`` -against a real file-backed ``sqlite+aiosqlite`` DB (so the index is actually -enforced), with a fake ``launch_run`` that only records launches. -""" - -from __future__ import annotations - -import asyncio -from datetime import UTC, datetime - -import pytest - -from app.scheduler.service import ScheduledTaskService -from deerflow.config.database_config import DatabaseConfig -from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config -from deerflow.persistence.scheduled_task_runs import ActiveScheduledRunConflict, ScheduledTaskRunRepository -from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository - -pytestmark = pytest.mark.asyncio - -_ACTIVE_STATUSES = {"queued", "running"} - - -class _BarrierRunRepo(ScheduledTaskRunRepository): - """Real repository that only releases both dispatchers past - ``has_active_runs`` once both have read it, so their ``create()`` calls - genuinely race for the task's single active slot — a deterministic - reproduction of the check-then-insert TOCTOU.""" - - def __init__(self, session_factory, barrier: asyncio.Barrier | None) -> None: - super().__init__(session_factory) - self._barrier = barrier - - async def has_active_runs(self, task_id: str) -> bool: - result = await super().has_active_runs(task_id) - if self._barrier is not None: - await self._barrier.wait() - return result - - -def _make_service(task_repo, run_repo, launched: list) -> ScheduledTaskService: - async def fake_launch(**kwargs): - # Yield so a truly-concurrent sibling can interleave, then record. - await asyncio.sleep(0) - launched.append(kwargs) - return {"run_id": f"run-{len(launched)}", "thread_id": kwargs["thread_id"]} - - return ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=10, - ) - - -async def _seed_task(task_repo: ScheduledTaskRepository, task_id: str) -> dict: - # fresh_thread_per_run: every dispatch gets a NEW thread_id, so #4003's - # per-thread uq_runs_thread_active can never fire for two dispatches of the - # same task — this is precisely the gap the per-task index closes. - await task_repo.create( - task_id=task_id, - user_id="user-1", - thread_id=None, - context_mode="fresh_thread_per_run", - assistant_id="lead_agent", - title=task_id, - prompt="do the thing", - schedule_type="cron", - schedule_spec={"cron": "*/5 * * * *"}, - timezone="UTC", - next_run_at=None, - ) - task = await task_repo.get(task_id, user_id="user-1") - assert task is not None - assert task["overlap_policy"] == "skip" - return task - - -async def _active_run_count(run_repo: ScheduledTaskRunRepository, task_id: str) -> int: - rows = await run_repo.list_by_task(task_id, limit=100) - return sum(1 for row in rows if row["status"] in _ACTIVE_STATUSES) - - -async def test_two_concurrent_manual_dispatches_launch_exactly_once(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - try: - sf = get_session_factory() - assert sf is not None - task_repo = ScheduledTaskRepository(sf) - run_repo = _BarrierRunRepo(sf, asyncio.Barrier(2)) - launched: list = [] - service = _make_service(task_repo, run_repo, launched) - task = await _seed_task(task_repo, "task-race-manual") - now = datetime.now(UTC) - - results = await asyncio.gather( - service.dispatch_task(dict(task), now=now, trigger="manual"), - service.dispatch_task(dict(task), now=now, trigger="manual"), - ) - - outcomes = sorted(result["outcome"] for result in results) - # Exactly one wins the active slot; the loser is a 409-style conflict. - assert outcomes == ["conflict", "launched"], outcomes - assert len(launched) == 1, launched - assert await _active_run_count(run_repo, "task-race-manual") == 1 - # The manual loser records no run-history row (nothing was scheduled). - conflict = next(r for r in results if r["outcome"] == "conflict") - assert conflict["task_run_id"] is None - finally: - await close_engine() - - -async def test_scheduled_and_manual_dispatch_launch_exactly_once(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - try: - sf = get_session_factory() - assert sf is not None - task_repo = ScheduledTaskRepository(sf) - run_repo = _BarrierRunRepo(sf, asyncio.Barrier(2)) - launched: list = [] - service = _make_service(task_repo, run_repo, launched) - task = await _seed_task(task_repo, "task-race-mixed") - now = datetime.now(UTC) - - results = await asyncio.gather( - service.dispatch_task(dict(task), now=now, trigger="scheduled"), - service.dispatch_task(dict(task), now=now, trigger="manual"), - ) - - outcomes = sorted(result["outcome"] for result in results) - # Whichever won launched; the loser is conflict (manual) or skipped - # (scheduled). Which one wins is timing-dependent, but exactly one runs. - assert outcomes.count("launched") == 1, outcomes - assert set(outcomes) <= {"launched", "conflict", "skipped"}, outcomes - assert len(launched) == 1, launched - assert await _active_run_count(run_repo, "task-race-mixed") == 1 - finally: - await close_engine() - - -async def test_natural_timing_concurrent_dispatch_launches_exactly_once(tmp_path): - # No barrier: exercise the fix under the same natural interleaving that - # reproduced the bug (5/5 both-launch on main). The fix must hold whether - # the second dispatch is caught by the has_active_runs fast path or by the - # index-violation path. - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - try: - sf = get_session_factory() - assert sf is not None - task_repo = ScheduledTaskRepository(sf) - run_repo = ScheduledTaskRunRepository(sf) - for i in range(5): - launched: list = [] - service = _make_service(task_repo, run_repo, launched) - task_id = f"task-natural-{i}" - task = await _seed_task(task_repo, task_id) - now = datetime.now(UTC) - - results = await asyncio.gather( - service.dispatch_task(dict(task), now=now, trigger="manual"), - service.dispatch_task(dict(task), now=now, trigger="manual"), - ) - - outcomes = sorted(result["outcome"] for result in results) - assert outcomes.count("launched") == 1, (i, outcomes) - assert len(launched) == 1, (i, launched) - assert await _active_run_count(run_repo, task_id) == 1, i - finally: - await close_engine() - - -async def test_partial_unique_index_enforces_one_active_run_per_task(tmp_path): - # Focused repository-level test of the index semantics + the typed conflict. - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - try: - sf = get_session_factory() - assert sf is not None - run_repo = ScheduledTaskRunRepository(sf) - now = datetime(2026, 7, 2, 1, 0, tzinfo=UTC) - - await run_repo.create(run_record_id="r1", task_id="t1", thread_id="th1", scheduled_for=now, trigger="scheduled", status="queued") - - # queued -> running is a same-row UPDATE: keeps the one active slot, no - # violation (this is the normal launch transition). - await run_repo.update_status("r1", status="running", run_id="run-1", started_at=now) - assert await run_repo.has_active_runs("t1") is True - - # A second active insert for the same task is a domain conflict. - with pytest.raises(ActiveScheduledRunConflict): - await run_repo.create(run_record_id="r2", task_id="t1", thread_id="th2", scheduled_for=now, trigger="manual", status="queued") - - # Terminal-status rows for the same task are outside the index predicate. - await run_repo.create(run_record_id="r3", task_id="t1", thread_id="th3", scheduled_for=now, trigger="scheduled", status="skipped") - - # A different task's active row is independent. - await run_repo.create(run_record_id="r4", task_id="t2", thread_id="th4", scheduled_for=now, trigger="scheduled", status="queued") - - # Finishing the active run frees the slot; a fresh active row is allowed. - await run_repo.update_status("r1", status="success", run_id="run-1", finished_at=now) - assert await run_repo.has_active_runs("t1") is False - await run_repo.create(run_record_id="r5", task_id="t1", thread_id="th5", scheduled_for=now, trigger="scheduled", status="queued") - assert await run_repo.has_active_runs("t1") is True - finally: - await close_engine() diff --git a/backend/tests/test_scheduled_task_models.py b/backend/tests/test_scheduled_task_models.py index c4f7b1f4b..47cfbc94f 100644 --- a/backend/tests/test_scheduled_task_models.py +++ b/backend/tests/test_scheduled_task_models.py @@ -1,4 +1,9 @@ +import re + +from sqlalchemy import Index + from deerflow.config.app_config import AppConfig +from deerflow.domain.schedule.model import ACTIVE_RUN_STATUSES from deerflow.persistence.models import ScheduledTaskRow, ScheduledTaskRunRow @@ -17,3 +22,37 @@ def test_app_config_exposes_scheduler_section(): def test_scheduled_task_models_registered(): assert ScheduledTaskRow.__tablename__ == "scheduled_tasks" assert ScheduledTaskRunRow.__tablename__ == "scheduled_task_runs" + + +def _active_run_index() -> Index: + return next(arg for arg in ScheduledTaskRunRow.__table_args__ if isinstance(arg, Index) and arg.name == "uq_scheduled_task_run_active") + + +def test_active_run_index_arbitrates_one_active_run_per_task(): + """The index is the atomic arbiter of the overlap rule, so its shape is a + contract, not a detail: unique, keyed on `task_id` alone.""" + index = _active_run_index() + assert index.unique is True + assert [column.name for column in index.expressions] == ["task_id"] + + +def test_active_run_index_predicate_matches_the_domain_constant(): + """`ACTIVE_RUN_STATUSES` and this predicate must stay in lockstep. + + The domain's fast path (`has_active`) and the index disagree the moment + they drift, which silently decouples the overlap check from its arbiter. + The domain tests cannot assert this -- they are deliberately + dependency-free and cannot import an ORM model -- so the assertion the + `ACTIVE_RUN_STATUSES` docstring promises lives here. + + Both dialect predicates are checked: `create_all` renders the SQLite one + and production renders the Postgres one, so a drift in either is real. + """ + index = _active_run_index() + expected = {str(status) for status in ACTIVE_RUN_STATUSES} + assert expected == {"queued", "running"}, "domain constant changed -- update the ORM predicates below" + + predicates = {key: str(value) for key, value in index.dialect_kwargs.items() if key.endswith("_where")} + assert set(predicates) == {"sqlite_where", "postgresql_where"}, "a dialect lost its partial-index predicate" + for dialect, predicate in predicates.items(): + assert set(re.findall(r"'([^']+)'", predicate)) == expected, f"{dialect} predicate drifted from ACTIVE_RUN_STATUSES" diff --git a/backend/tests/test_scheduled_task_repository.py b/backend/tests/test_scheduled_task_repository.py deleted file mode 100644 index a9101f440..000000000 --- a/backend/tests/test_scheduled_task_repository.py +++ /dev/null @@ -1,324 +0,0 @@ -from datetime import UTC, datetime - -import pytest - -from deerflow.config.database_config import DatabaseConfig -from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config -from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository -from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository - - -@pytest.mark.asyncio -async def test_scheduled_task_repository_create_and_list(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRepository(sf) - created = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize this thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="Asia/Shanghai", - next_run_at=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - ) - - assert created["id"] == "task-1" - listed = await repo.list_by_user("user-1") - assert [task["id"] for task in listed] == ["task-1"] - - await close_engine() - - -@pytest.mark.asyncio -async def test_scheduled_task_run_repository_records_history(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRunRepository(sf) - row = await repo.create( - run_record_id="task-run-1", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="manual", - status="queued", - ) - - assert row["id"] == "task-run-1" - history = await repo.list_by_task("task-1") - assert [entry["id"] for entry in history] == ["task-run-1"] - - await close_engine() - - -@pytest.mark.asyncio -async def test_mark_stale_active_runs_fails_orphaned_runs(tmp_path): - """Runs stuck in queued/running after a process crash are swept to interrupted.""" - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRunRepository(sf) - # The queued and running rows live on different tasks: mark_stale_active_runs - # is a global sweep (no task filter), and the uq_scheduled_task_run_active - # partial unique index forbids two active rows on one task_id, so the pair - # that proves both active statuses get swept must be spread across tasks. - await repo.create( - run_record_id="task-run-queued", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="scheduled", - status="queued", - ) - await repo.create( - run_record_id="task-run-running", - task_id="task-2", - thread_id="thread-2", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="scheduled", - status="running", - ) - # A terminal row on task-1 (outside the index predicate) coexists with the - # active queued row and must be left untouched by the sweep. - await repo.create( - run_record_id="task-run-success", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="scheduled", - status="success", - ) - - swept = await repo.mark_stale_active_runs(error="interrupted: gateway restarted") - assert swept == 2 - - by_id = {entry["id"]: entry for entry in await repo.list_by_task("task-1")} - by_id.update({entry["id"]: entry for entry in await repo.list_by_task("task-2")}) - assert by_id["task-run-queued"]["status"] == "interrupted" - assert by_id["task-run-running"]["status"] == "interrupted" - assert by_id["task-run-success"]["status"] == "success" - - await close_engine() - - -@pytest.mark.asyncio -async def test_update_status_protect_terminal_keeps_completion_result(tmp_path): - """The launch-path "running" write must not clobber a terminal status - already committed by the completion hook (launch/completion race).""" - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRunRepository(sf) - await repo.create( - run_record_id="task-run-race", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="scheduled", - status="queued", - ) - # Completion hook wins the race and commits the terminal state first. - await repo.update_status("task-run-race", status="failed", run_id="run-1", error="boom", finished_at=datetime(2026, 7, 2, 1, 1, tzinfo=UTC)) - # Late launch-path write: keeps terminal status/error, backfills started_at. - await repo.update_status("task-run-race", status="running", run_id="run-1", started_at=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), protect_terminal=True) - - entry = (await repo.list_by_task("task-1"))[0] - assert entry["status"] == "failed" - assert entry["error"] == "boom" - assert entry["started_at"] is not None - - await close_engine() - - -@pytest.mark.asyncio -async def test_has_active_runs_sees_only_queued_and_running(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRunRepository(sf) - assert await repo.has_active_runs("task-1") is False - await repo.create( - run_record_id="task-run-active", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - trigger="scheduled", - status="running", - ) - assert await repo.has_active_runs("task-1") is True - await repo.update_status("task-run-active", status="success", run_id="run-1") - assert await repo.has_active_runs("task-1") is False - - await close_engine() - - -@pytest.mark.asyncio -async def test_cancel_stuck_once_tasks_reconciles_orphaned_running(tmp_path): - """Launched (lease cleared) once tasks stuck in running are cancelled at - startup; leased ones are left for expired-lease reclaim.""" - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRepository(sf) - for task_id, schedule_type, status in ( - ("task-once-stuck", "once", "running"), - ("task-once-done", "once", "completed"), - ("task-cron-running", "cron", "running"), - ): - await repo.create( - task_id=task_id, - user_id="user-1", - thread_id=None, - context_mode="fresh_thread_per_run", - assistant_id="lead_agent", - title=task_id, - prompt="p", - schedule_type=schedule_type, - schedule_spec={"run_at": "2026-07-02T01:00:00+00:00"} if schedule_type == "once" else {"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - await repo.update(task_id, user_id="user-1", updates={"status": status}) - # A claimed-but-not-launched once task still holds its lease: keep it. - await repo.create( - task_id="task-once-leased", - user_id="user-1", - thread_id=None, - context_mode="fresh_thread_per_run", - assistant_id="lead_agent", - title="task-once-leased", - prompt="p", - schedule_type="once", - schedule_spec={"run_at": "2026-07-02T01:00:00+00:00"}, - timezone="UTC", - next_run_at=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - ) - await repo.update("task-once-leased", user_id="user-1", updates={"status": "running", "lease_expires_at": datetime(2026, 7, 2, 1, 2, tzinfo=UTC)}) - - cancelled = await repo.cancel_stuck_once_tasks(error="interrupted: gateway restarted") - assert cancelled == 1 - - by_id = {t["id"]: t for t in await repo.list_by_user("user-1")} - assert by_id["task-once-stuck"]["status"] == "cancelled" - assert by_id["task-once-stuck"]["last_error"] == "interrupted: gateway restarted" - assert by_id["task-once-done"]["status"] == "completed" - assert by_id["task-cron-running"]["status"] == "running" - assert by_id["task-once-leased"]["status"] == "running" - - await close_engine() - - -@pytest.mark.asyncio -async def test_update_after_launch_protect_terminal_keeps_hook_result(tmp_path): - """The launch-path bookkeeping write must not clobber a terminal task - status committed first by the completion hook (fast-failing run).""" - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRepository(sf) - await repo.create( - task_id="task-race", - user_id="user-1", - thread_id=None, - context_mode="fresh_thread_per_run", - assistant_id="lead_agent", - title="task-race", - prompt="p", - schedule_type="once", - schedule_spec={"run_at": "2026-07-02T01:00:00+00:00"}, - timezone="UTC", - next_run_at=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - ) - # Completion hook wins the race: task finalized as failed. - await repo.update("task-race", user_id="user-1", updates={"status": "failed", "last_error": "boom"}) - # Late launch-path write with protection keeps the hook's outcome. - await repo.update_after_launch( - "task-race", - status="running", - next_run_at=None, - last_run_at=datetime(2026, 7, 2, 1, 0, tzinfo=UTC), - last_run_id="run-1", - last_thread_id="thread-1", - last_error=None, - increment_run_count=True, - protect_terminal=True, - ) - - task = await repo.get("task-race", user_id="user-1") - assert task is not None - assert task["status"] == "failed" - assert task["last_error"] == "boom" - # Launch bookkeeping still recorded. - assert task["last_run_id"] == "run-1" - assert task["run_count"] == 1 - - await close_engine() - - -@pytest.mark.asyncio -async def test_list_by_task_paginates(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRunRepository(sf) - for i in range(5): - await repo.create( - run_record_id=f"task-run-{i}", - task_id="task-1", - thread_id="thread-1", - scheduled_for=datetime(2026, 7, 2, 1, i, tzinfo=UTC), - trigger="scheduled", - status="success", - ) - - assert await repo.count_active_runs() == 0 - page1 = await repo.list_by_task("task-1", limit=2) - page2 = await repo.list_by_task("task-1", limit=2, offset=2) - assert len(page1) == 2 - assert len(page2) == 2 - assert {e["id"] for e in page1}.isdisjoint({e["id"] for e in page2}) - - await close_engine() - - -@pytest.mark.asyncio -async def test_list_by_user_and_thread_filters_in_sql(tmp_path): - await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path))) - sf = get_session_factory() - assert sf is not None - - repo = ScheduledTaskRepository(sf) - for task_id, thread_id in (("task-a", "thread-1"), ("task-b", "thread-2"), ("task-c", "thread-1")): - await repo.create( - task_id=task_id, - user_id="user-1", - thread_id=thread_id, - context_mode="reuse_thread", - assistant_id="lead_agent", - title=task_id, - prompt="p", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - - listed = await repo.list_by_user_and_thread("user-1", "thread-1") - assert sorted(t["id"] for t in listed) == ["task-a", "task-c"] - assert await repo.list_by_user_and_thread("user-2", "thread-1") == [] - - await close_engine() diff --git a/backend/tests/test_scheduled_task_router.py b/backend/tests/test_scheduled_task_router.py deleted file mode 100644 index 4c28161c1..000000000 --- a/backend/tests/test_scheduled_task_router.py +++ /dev/null @@ -1,38 +0,0 @@ -from fastapi import FastAPI -from fastapi.testclient import TestClient - -from app.gateway.routers import scheduled_tasks - - -def test_router_registers_list_endpoint(): - app = FastAPI() - app.include_router(scheduled_tasks.router) - client = TestClient(app) - response = client.get("/api/scheduled-tasks") - assert response.status_code != 404 - - -def test_router_registers_trigger_route(): - app = FastAPI() - app.include_router(scheduled_tasks.router) - client = TestClient(app) - response = client.post("/api/scheduled-tasks/task-1/trigger") - assert response.status_code != 404 - - -def test_router_registers_create_route(): - app = FastAPI() - app.include_router(scheduled_tasks.router) - client = TestClient(app) - response = client.post( - "/api/scheduled-tasks", - json={ - "thread_id": "thread-1", - "title": "Daily summary", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - }, - ) - assert response.status_code != 404 diff --git a/backend/tests/test_scheduled_task_router_behavior.py b/backend/tests/test_scheduled_task_router_behavior.py deleted file mode 100644 index 70e516752..000000000 --- a/backend/tests/test_scheduled_task_router_behavior.py +++ /dev/null @@ -1,652 +0,0 @@ -from datetime import UTC, datetime, timedelta -from types import SimpleNamespace -from unittest.mock import AsyncMock - -import pytest - -from app.gateway.routers import scheduled_tasks - - -class _Repo: - def __init__(self) -> None: - self.created = [] - self.items = {} - - async def list_by_user(self, user_id: str): - return [item for item in self.items.values() if item["user_id"] == user_id] - - async def list_by_user_and_thread(self, user_id: str, thread_id: str): - return [item for item in self.items.values() if item["user_id"] == user_id and item["thread_id"] == thread_id] - - async def create(self, **kwargs): - item = { - "id": kwargs["task_id"], - "user_id": kwargs["user_id"], - "thread_id": kwargs["thread_id"], - "context_mode": kwargs["context_mode"], - "title": kwargs["title"], - "prompt": kwargs["prompt"], - "schedule_type": kwargs["schedule_type"], - "schedule_spec": kwargs["schedule_spec"], - "timezone": kwargs["timezone"], - "status": "enabled", - "next_run_at": kwargs["next_run_at"], - } - self.items[item["id"]] = item - self.created.append(item) - return item - - async def get(self, task_id: str, *, user_id: str): - item = self.items.get(task_id) - if item is None or item["user_id"] != user_id: - return None - return item - - async def update(self, task_id: str, *, user_id: str, updates): - item = await self.get(task_id, user_id=user_id) - if item is None: - return None - item.update(updates) - return item - - async def delete(self, task_id: str, *, user_id: str): - item = await self.get(task_id, user_id=user_id) - if item is None: - return False - self.items.pop(task_id, None) - return True - - async def list_by_task(self, task_id: str): - return [] - - -class _Service: - def __init__(self) -> None: - self.calls = [] - self.result = {"outcome": "launched"} - - async def dispatch_task(self, task, *, now, trigger): - self.calls.append((task, now, trigger)) - return self.result - - -class _RunStore: - def __init__(self, runs): - self.runs = runs - - async def get(self, run_id: str, *, user_id: str): - run = self.runs.get(run_id) - if run is None or run.get("user_id") != user_id: - return None - return run - - -class _Config: - def __init__(self, min_once_delay_seconds: int = 60) -> None: - self.scheduler = SimpleNamespace(min_once_delay_seconds=min_once_delay_seconds) - - -@pytest.mark.asyncio -async def test_create_scheduled_task_uses_repo(): - repo = _Repo() - request = SimpleNamespace() - body = scheduled_tasks.ScheduledTaskCreateRequest( - thread_id="thread-1", - title="Daily summary", - prompt="Summarize thread", - schedule_type="once", - schedule_spec={"run_at": "2027-01-01T01:00:00+00:00"}, - timezone="UTC", - ) - - user = SimpleNamespace(id="user-1") - thread_store = SimpleNamespace(check_access=AsyncMock(return_value=True)) - config = _Config() - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_thread_store = scheduled_tasks.get_thread_store - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_thread_store = lambda _request: thread_store - scheduled_tasks.get_config = lambda: config - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - created = await scheduled_tasks.create_scheduled_task.__wrapped__( - request=request, - body=body, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_thread_store = old_thread_store - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert created["title"] == "Daily summary" - assert created["user_id"] == "user-1" - assert created["next_run_at"] == datetime(2027, 1, 1, 1, 0, tzinfo=UTC) - - -@pytest.mark.asyncio -async def test_create_fresh_thread_task_does_not_require_thread_id(): - repo = _Repo() - request = SimpleNamespace() - body = scheduled_tasks.ScheduledTaskCreateRequest( - context_mode="fresh_thread_per_run", - thread_id=None, - title="Fresh task", - prompt="Run in fresh thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - ) - - user = SimpleNamespace(id="user-1") - thread_store = SimpleNamespace(check_access=AsyncMock(return_value=True)) - config = _Config() - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_thread_store = scheduled_tasks.get_thread_store - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_thread_store = lambda _request: thread_store - scheduled_tasks.get_config = lambda: config - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - created = await scheduled_tasks.create_scheduled_task.__wrapped__( - request=request, - body=body, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_thread_store = old_thread_store - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert created["context_mode"] == "fresh_thread_per_run" - assert created["thread_id"] is None - - -@pytest.mark.asyncio -async def test_trigger_scheduled_task_dispatches_manual_run(): - repo = _Repo() - service = _Service() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_service = scheduled_tasks.get_scheduled_task_service - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_scheduled_task_service = lambda _request: service - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.trigger_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_scheduled_task_service = old_service - scheduled_tasks.get_optional_user_from_request = old_user - - assert result == {"id": "task-1", "triggered": True} - assert len(service.calls) == 1 - assert service.calls[0][2] == "manual" - - -@pytest.mark.asyncio -async def test_trigger_scheduled_task_returns_conflict_when_dispatch_conflicts(): - repo = _Repo() - service = _Service() - service.result = {"outcome": "conflict", "error": "Thread thread-1 already has an active run"} - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_service = scheduled_tasks.get_scheduled_task_service - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_scheduled_task_service = lambda _request: service - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - with pytest.raises(Exception) as exc_info: - await scheduled_tasks.trigger_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_scheduled_task_service = old_service - scheduled_tasks.get_optional_user_from_request = old_user - - assert "already has an active run" in str(exc_info.value) - - -@pytest.mark.asyncio -async def test_update_scheduled_task_writes_repo(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - config = _Config() - thread_store = SimpleNamespace(check_access=AsyncMock(return_value=True)) - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_thread_store = scheduled_tasks.get_thread_store - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_thread_store = lambda _request: thread_store - scheduled_tasks.get_config = lambda: config - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.update_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - body=scheduled_tasks.ScheduledTaskUpdateRequest(title="Updated title"), - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_thread_store = old_thread_store - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert result["title"] == "Updated title" - - -@pytest.mark.asyncio -async def test_delete_scheduled_task_deletes_repo_row(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.delete_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_optional_user_from_request = old_user - - assert result == {"id": "task-1", "deleted": True} - assert repo.items == {} - - -@pytest.mark.asyncio -async def test_pause_and_resume_scheduled_task_update_status(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - paused = await scheduled_tasks.pause_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - paused_status = paused["status"] - resumed = await scheduled_tasks.resume_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_optional_user_from_request = old_user - - assert paused_status == "paused" - assert resumed["status"] == "enabled" - - -@pytest.mark.asyncio -async def test_pause_rejects_running_task(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - task["status"] = "running" - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - with pytest.raises(Exception) as exc_info: - await scheduled_tasks.pause_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_optional_user_from_request = old_user - - assert "currently running" in str(exc_info.value) - - -@pytest.mark.asyncio -async def test_update_rejects_running_task(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Daily summary", - prompt="Summarize thread", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - task["status"] = "running" - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - config = _Config() - thread_store = SimpleNamespace(check_access=AsyncMock(return_value=True)) - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_thread_store = scheduled_tasks.get_thread_store - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_thread_store = lambda _request: thread_store - scheduled_tasks.get_config = lambda: config - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - with pytest.raises(Exception) as exc_info: - await scheduled_tasks.update_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - body=scheduled_tasks.ScheduledTaskUpdateRequest(title="Updated title"), - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_thread_store = old_thread_store - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert "currently running" in str(exc_info.value) - - -@pytest.mark.asyncio -async def test_list_thread_scheduled_tasks_filters_by_thread_id(): - repo = _Repo() - await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Thread one task", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - await repo.create( - task_id="task-2", - user_id="user-1", - thread_id="thread-2", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Thread two task", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.list_thread_scheduled_tasks.__wrapped__( - thread_id="thread-1", - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_optional_user_from_request = old_user - - assert [task["id"] for task in result] == ["task-1"] - - -@pytest.mark.asyncio -async def test_list_scheduled_task_runs_returns_persisted_rows_without_side_effects(): - repo = _Repo() - task = await repo.create( - task_id="task-1", - user_id="user-1", - thread_id="thread-1", - context_mode="reuse_thread", - assistant_id="lead_agent", - title="Task", - prompt="Prompt", - schedule_type="cron", - schedule_spec={"cron": "0 9 * * *"}, - timezone="UTC", - next_run_at=None, - ) - run_repo = SimpleNamespace( - list_by_task=AsyncMock( - return_value=[ - { - "id": "task-run-1", - "task_id": "task-1", - "thread_id": "thread-1", - "run_id": "run-1", - "status": "running", - "error": None, - } - ] - ), - ) - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_task_repo = scheduled_tasks.get_scheduled_task_repo - old_run_repo = scheduled_tasks.get_scheduled_task_run_repo - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_scheduled_task_run_repo = lambda _request: run_repo - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.list_scheduled_task_runs.__wrapped__( - task_id=task["id"], - request=request, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_task_repo - scheduled_tasks.get_scheduled_task_run_repo = old_run_repo - scheduled_tasks.get_optional_user_from_request = old_user - - assert result[0]["status"] == "running" - - -@pytest.mark.asyncio -async def test_create_once_task_enforces_minimum_delay(): - repo = _Repo() - request = SimpleNamespace() - body = scheduled_tasks.ScheduledTaskCreateRequest( - thread_id="thread-1", - title="Soon task", - prompt="Run soon", - schedule_type="once", - schedule_spec={"run_at": (datetime.now(UTC) + timedelta(seconds=30)).isoformat()}, - timezone="UTC", - ) - user = SimpleNamespace(id="user-1") - thread_store = SimpleNamespace(check_access=AsyncMock(return_value=True)) - config = _Config(min_once_delay_seconds=60) - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_thread_store = scheduled_tasks.get_thread_store - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_thread_store = lambda _request: thread_store - scheduled_tasks.get_config = lambda: config - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - with pytest.raises(Exception) as exc_info: - await scheduled_tasks.create_scheduled_task.__wrapped__( - request=request, - body=body, - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_thread_store = old_thread_store - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert "once schedule must be at least" in str(exc_info.value) - - -@pytest.mark.asyncio -async def test_update_terminal_once_task_with_future_run_at_rearms_it(): - """PATCHing a fresh future run_at onto a completed/failed/cancelled once - task must reset status to enabled — claim_due_tasks only admits enabled - rows, so keeping the terminal status returns a next_run_at that never fires.""" - repo = _Repo() - task = await repo.create( - task_id="task-terminal", - user_id="user-1", - thread_id=None, - context_mode="fresh_thread_per_run", - assistant_id="lead_agent", - title="Once done", - prompt="p", - schedule_type="once", - schedule_spec={"run_at": "2026-07-01T00:00:00+00:00"}, - timezone="UTC", - next_run_at=None, - ) - task["status"] = "completed" - future_run_at = (datetime.now(UTC) + timedelta(hours=1)).isoformat() - request = SimpleNamespace() - user = SimpleNamespace(id="user-1") - - old_repo = scheduled_tasks.get_scheduled_task_repo - old_config = scheduled_tasks.get_config - old_user = scheduled_tasks.get_optional_user_from_request - try: - scheduled_tasks.get_scheduled_task_repo = lambda _request: repo - scheduled_tasks.get_config = lambda: _Config() - scheduled_tasks.get_optional_user_from_request = AsyncMock(return_value=user) - - result = await scheduled_tasks.update_scheduled_task.__wrapped__( - task_id=task["id"], - request=request, - body=scheduled_tasks.ScheduledTaskUpdateRequest(schedule_spec={"run_at": future_run_at}), - ) - finally: - scheduled_tasks.get_scheduled_task_repo = old_repo - scheduled_tasks.get_config = old_config - scheduled_tasks.get_optional_user_from_request = old_user - - assert result["status"] == "enabled" - assert result["next_run_at"] is not None diff --git a/backend/tests/test_scheduled_task_schedules.py b/backend/tests/test_scheduled_task_schedules.py deleted file mode 100644 index fcaac8676..000000000 --- a/backend/tests/test_scheduled_task_schedules.py +++ /dev/null @@ -1,49 +0,0 @@ -from datetime import UTC, datetime - -import pytest - -from deerflow.scheduler.schedules import ( - next_run_at, - normalize_cron_expression, - validate_timezone, -) - - -def test_validate_timezone_accepts_iana_name(): - assert validate_timezone("Asia/Shanghai") == "Asia/Shanghai" - - -def test_validate_timezone_rejects_unknown_name(): - with pytest.raises(ValueError): - validate_timezone("Mars/Base") - - -def test_normalize_cron_accepts_five_fields(): - assert normalize_cron_expression("0 9 * * 1") == "0 9 * * 1" - - -def test_normalize_cron_rejects_seconds_field(): - with pytest.raises(ValueError): - normalize_cron_expression("0 0 9 * * 1") - - -def test_next_run_at_for_once_returns_none_after_fire_time(): - now = datetime(2026, 7, 2, 2, 0, tzinfo=UTC) - result = next_run_at( - "once", - {"run_at": "2026-07-02T01:00:00+00:00"}, - "UTC", - now=now, - ) - assert result is None - - -def test_next_run_at_for_cron_uses_timezone(): - now = datetime(2026, 7, 1, 0, 30, tzinfo=UTC) - result = next_run_at( - "cron", - {"cron": "0 9 * * *"}, - "Asia/Shanghai", - now=now, - ) - assert result == datetime(2026, 7, 1, 1, 0, tzinfo=UTC) diff --git a/backend/tests/test_scheduled_task_service.py b/backend/tests/test_scheduled_task_service.py deleted file mode 100644 index a071ad298..000000000 --- a/backend/tests/test_scheduled_task_service.py +++ /dev/null @@ -1,795 +0,0 @@ -from datetime import UTC, datetime, timedelta - -import pytest - -from app.scheduler.service import ScheduledTaskService -from deerflow.runtime import ConflictError, RunStatus -from deerflow.runtime.runs.manager import RunRecord -from deerflow.runtime.runs.schemas import DisconnectMode - - -class DummyTaskRepo: - def __init__(self, rows): - self.rows = rows - self.claimed = False - self.updated = None - self.cancelled_stuck_once = None - - async def cancel_stuck_once_tasks(self, *, error): - self.cancelled_stuck_once = error - return 0 - - async def claim_due_tasks(self, **_kwargs): - if self.claimed: - return [] - self.claimed = True - return self.rows - - async def update_after_launch(self, *args, **kwargs): - self.updated = (args, kwargs) - - async def get(self, task_id: str, *, user_id: str): - row = next((item for item in self.rows if item["id"] == task_id and item["user_id"] == user_id), None) - return dict(row) if row is not None else None - - async def update(self, task_id: str, *, user_id: str, updates): - row = next((item for item in self.rows if item["id"] == task_id and item["user_id"] == user_id), None) - if row is None: - return None - row.update(updates) - return dict(row) - - -class DummyRunRepo: - def __init__(self, *, active=False, active_count=0): - self.created = None - self.updated = [] - self.active = active - self.active_count = active_count - self.stale_marked = None - - async def count_active_runs(self): - return self.active_count - - async def create(self, **kwargs): - self.created = kwargs - return {"id": kwargs["run_record_id"]} - - async def update_status(self, run_record_id, **kwargs): - self.updated.append((run_record_id, kwargs)) - - async def has_active_runs(self, task_id): - return self.active - - async def mark_stale_active_runs(self, *, error): - self.stale_marked = error - return 0 - - -@pytest.mark.asyncio -async def test_service_claims_and_dispatches_due_task(): - async def fake_launch(**kwargs): - assert kwargs["owner_user_id"] == "user-1" - assert kwargs["metadata"]["scheduled_task_id"] == "task-1" - assert kwargs["metadata"]["scheduled_trigger"] == "scheduled" - return {"run_id": "run-1", "thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo( - [ - { - "id": "task-1", - "user_id": "user-1", - "thread_id": "thread-1", - "context_mode": "reuse_thread", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "once", - "schedule_spec": {"run_at": "2026-07-02T01:00:00+00:00"}, - "timezone": "UTC", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - await service.run_once(now=datetime.now(UTC) + timedelta(days=1)) - - assert run_repo.created["task_id"] == "task-1" - assert run_repo.updated[0][1]["status"] == "running" - assert run_repo.updated[0][1]["protect_terminal"] is True - # `once` terminal status is owned by handle_run_completion, not the launch. - assert task_repo.updated[1]["status"] == "running" - - -@pytest.mark.asyncio -async def test_manual_trigger_keeps_paused_cron_task_paused(): - async def fake_launch(**kwargs): - return {"run_id": "run-2", "thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo( - [ - { - "id": "task-2", - "user_id": "user-1", - "thread_id": "thread-1", - "context_mode": "reuse_thread", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - "status": "paused", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - await service.dispatch_task( - task_repo.rows[0], - now=datetime.now(UTC), - trigger="manual", - ) - - assert task_repo.updated[1]["status"] == "paused" - - -@pytest.mark.asyncio -async def test_fresh_thread_per_run_creates_new_execution_thread(): - async def fake_launch(**kwargs): - assert kwargs["thread_id"] != "thread-template" - return {"run_id": "run-3", "thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo( - [ - { - "id": "task-3", - "user_id": "user-1", - "thread_id": "thread-template", - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - "status": "enabled", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - await service.dispatch_task( - task_repo.rows[0], - now=datetime.now(UTC), - trigger="scheduled", - ) - - assert run_repo.created["thread_id"] != "thread-template" - assert task_repo.updated[1]["last_thread_id"] == run_repo.created["thread_id"] - - -@pytest.mark.asyncio -async def test_scheduled_overlap_conflict_is_recorded_as_skip(): - async def fake_launch(**_kwargs): - raise ConflictError("Thread thread-1 already has an active run") - - task_repo = DummyTaskRepo( - [ - { - "id": "task-4", - "user_id": "user-1", - "thread_id": "thread-1", - "context_mode": "reuse_thread", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - "status": "running", - "overlap_policy": "skip", - "last_run_id": "run-old", - "last_thread_id": "thread-1", - "last_run_at": "2026-07-01T00:00:00+00:00", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - result = await service.dispatch_task( - task_repo.rows[0], - now=datetime.now(UTC), - trigger="scheduled", - ) - - assert result["outcome"] == "skipped" - assert run_repo.updated[-1][1]["status"] == "skipped" - assert task_repo.updated[1]["status"] == "enabled" - - -@pytest.mark.asyncio -async def test_manual_overlap_conflict_returns_conflict(): - async def fake_launch(**_kwargs): - raise ConflictError("Thread thread-1 already has an active run") - - task_repo = DummyTaskRepo( - [ - { - "id": "task-5", - "user_id": "user-1", - "thread_id": "thread-1", - "context_mode": "reuse_thread", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - "status": "enabled", - "overlap_policy": "skip", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - result = await service.dispatch_task( - task_repo.rows[0], - now=datetime.now(UTC), - trigger="manual", - ) - - assert result["outcome"] == "conflict" - assert run_repo.updated[-1][1]["status"] == "failed" - - -@pytest.mark.asyncio -async def test_handle_run_completion_persists_success(): - task_repo = DummyTaskRepo( - [ - { - "id": "task-6", - "user_id": "user-1", - "thread_id": None, - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "cron", - "schedule_spec": {"cron": "0 9 * * *"}, - "timezone": "UTC", - "status": "enabled", - } - ] - ) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=lambda **_kwargs: None, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - record = RunRecord( - run_id="run-6", - thread_id="thread-6", - assistant_id="lead_agent", - status=RunStatus.success, - on_disconnect=DisconnectMode.continue_, - metadata={ - "scheduled_task_id": "task-6", - "scheduled_task_run_id": "task-run-6", - }, - user_id="user-1", - ) - - await service.handle_run_completion(record) - - assert run_repo.updated[-1][0] == "task-run-6" - assert run_repo.updated[-1][1]["status"] == "success" - assert task_repo.rows[0]["last_error"] is None - - -def _make_service(task_repo, run_repo): - return ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=lambda **_kwargs: None, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - -def _once_task_row(task_id="task-once", status="running"): - return { - "id": task_id, - "user_id": "user-1", - "thread_id": None, - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "Summarize thread", - "schedule_type": "once", - "schedule_spec": {"run_at": "2026-07-02T01:00:00+00:00"}, - "timezone": "UTC", - "status": status, - } - - -def _completion_record(status, *, task_id="task-once", error=None): - return RunRecord( - run_id="run-x", - thread_id="thread-x", - assistant_id="lead_agent", - status=status, - on_disconnect=DisconnectMode.continue_, - metadata={ - "scheduled_task_id": task_id, - "scheduled_task_run_id": "task-run-x", - }, - user_id="user-1", - error=error, - ) - - -@pytest.mark.asyncio -async def test_once_task_completes_only_via_completion_hook(): - task_repo = DummyTaskRepo([_once_task_row()]) - run_repo = DummyRunRepo() - service = _make_service(task_repo, run_repo) - - await service.handle_run_completion(_completion_record(RunStatus.success)) - - assert run_repo.updated[-1][1]["status"] == "success" - assert task_repo.rows[0]["status"] == "completed" - - -@pytest.mark.asyncio -async def test_once_task_failed_run_marks_task_failed(): - task_repo = DummyTaskRepo([_once_task_row()]) - run_repo = DummyRunRepo() - service = _make_service(task_repo, run_repo) - - await service.handle_run_completion(_completion_record(RunStatus.error, error="boom")) - - assert run_repo.updated[-1][1]["status"] == "failed" - assert run_repo.updated[-1][1]["error"] == "boom" - assert task_repo.rows[0]["status"] == "failed" - assert task_repo.rows[0]["last_error"] == "boom" - - -@pytest.mark.asyncio -async def test_interrupted_run_is_distinct_and_cancels_once_task(): - task_repo = DummyTaskRepo([_once_task_row()]) - run_repo = DummyRunRepo() - service = _make_service(task_repo, run_repo) - - await service.handle_run_completion(_completion_record(RunStatus.interrupted)) - - run_update = run_repo.updated[-1][1] - assert run_update["status"] == "interrupted" - assert run_update["error"] == "run was interrupted before completion" - assert task_repo.rows[0]["status"] == "cancelled" - - -@pytest.mark.asyncio -async def test_interrupted_cron_run_keeps_task_enabled(): - row = _once_task_row(task_id="task-cron") - row.update({"schedule_type": "cron", "schedule_spec": {"cron": "0 9 * * *"}, "status": "enabled"}) - task_repo = DummyTaskRepo([row]) - run_repo = DummyRunRepo() - service = _make_service(task_repo, run_repo) - - await service.handle_run_completion(_completion_record(RunStatus.interrupted, task_id="task-cron")) - - assert run_repo.updated[-1][1]["status"] == "interrupted" - assert task_repo.rows[0]["status"] == "enabled" - - -@pytest.mark.asyncio -async def test_skip_policy_applies_to_fresh_thread_runs(): - launched = [] - - async def fake_launch(**kwargs): - launched.append(kwargs) - return {"run_id": "run-9", "thread_id": kwargs["thread_id"]} - - row = _once_task_row(task_id="task-9") - row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "status": "running", "overlap_policy": "skip"}) - task_repo = DummyTaskRepo([row]) - run_repo = DummyRunRepo(active=True) - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="scheduled") - - assert result["outcome"] == "skipped" - assert launched == [] - # The skip tombstone is created directly as terminal "skipped" (not the - # transient "queued" the launch path uses): a queued row is active and would - # itself trip the uq_scheduled_task_run_active partial unique index against - # the pre-existing run still holding the task's single active slot. - assert run_repo.created["status"] == "skipped" - assert run_repo.updated[-1][1]["status"] == "skipped" - assert task_repo.updated[1]["status"] == "enabled" - assert task_repo.updated[1]["increment_run_count"] is False - - -@pytest.mark.asyncio -async def test_startup_sweep_reconciles_stale_runs_and_stuck_once_tasks(): - task_repo = DummyTaskRepo([]) - run_repo = DummyRunRepo() - service = _make_service(task_repo, run_repo) - - await service.start() - await service.stop() - - assert run_repo.stale_marked is not None - assert task_repo.cancelled_stuck_once == run_repo.stale_marked - - -@pytest.mark.asyncio -async def test_manual_trigger_with_active_run_returns_conflict_without_launching(): - launched = [] - - async def fake_launch(**kwargs): - launched.append(kwargs) - return {"run_id": "run-x", "thread_id": kwargs["thread_id"]} - - row = _once_task_row(task_id="task-manual-busy") - row.update({"schedule_type": "cron", "schedule_spec": {"cron": "* * * * *"}, "status": "enabled", "overlap_policy": "skip"}) - task_repo = DummyTaskRepo([row]) - run_repo = DummyRunRepo(active=True) - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - result = await service.dispatch_task(row, now=datetime.now(UTC), trigger="manual") - - assert result["outcome"] == "conflict" - assert launched == [] - # Nothing was scheduled to happen, so no run-history row is recorded. - assert run_repo.created is None - assert result["task_run_id"] is None - - -@pytest.mark.asyncio -async def test_run_once_claims_only_into_remaining_global_budget(): - claim_limits = [] - - class BudgetTaskRepo(DummyTaskRepo): - async def claim_due_tasks(self, **kwargs): - claim_limits.append(kwargs["limit"]) - return [] - - task_repo = BudgetTaskRepo([]) - run_repo = DummyRunRepo(active_count=2) - service = _make_service(task_repo, run_repo) - - await service.run_once(now=datetime.now(UTC)) - assert claim_limits == [1] - - run_repo.active_count = 3 - await service.run_once(now=datetime.now(UTC)) - # Budget exhausted: no claim at all this cycle. - assert claim_limits == [1] - - -@pytest.mark.asyncio -async def test_launch_bookkeeping_passes_protect_terminal(): - async def fake_launch(**kwargs): - return {"run_id": "run-pt", "thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo([_once_task_row(task_id="task-pt", status="enabled")]) - run_repo = DummyRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - await service.dispatch_task(task_repo.rows[0], now=datetime.now(UTC), trigger="scheduled") - - assert task_repo.updated[1]["protect_terminal"] is True - - -class _StatefulRunRepo: - """Stateful fake ``ScheduledTaskRunRepository`` for the #4452 tests. - - Mirrors just enough of the real repository to let a second dispatch - observe the active slot held by the first: - - * ``create()`` tracks each row by id, carrying its ``status`` and - ``run_id``; - * with ``fail_first_update=True`` the very FIRST ``update_status()`` - call raises, simulating a transient DB failure on the - ``queued -> running`` write that fires right after ``_launch_run`` - returns a live ``run_id``; every later ``update_status()`` applies; - * ``has_active_runs()`` reflects whether any tracked row for the task - is still in an active status (``queued``/``running``), exactly like - the partial unique index ``uq_scheduled_task_run_active``. - """ - - _ACTIVE = {"queued", "running"} - - def __init__(self, *, fail_first_update: bool = False) -> None: - self.created: list[dict] = [] - self.updates: list[tuple[str, dict]] = [] - self.rows: dict[str, dict] = {} - self._fail_first_update = fail_first_update - self._first_update_raised = False - - async def count_active_runs(self) -> int: - return sum(1 for row in self.rows.values() if row["status"] in self._ACTIVE) - - async def create(self, **kwargs) -> dict: - self.created.append(kwargs) - self.rows[kwargs["run_record_id"]] = { - "task_id": kwargs["task_id"], - "status": kwargs["status"], - "run_id": None, - } - return {"id": kwargs["run_record_id"]} - - async def update_status(self, run_record_id: str, **kwargs) -> None: - self.updates.append((run_record_id, kwargs)) - if self._fail_first_update and not self._first_update_raised: - # The launch-path queued->running write fails once, AFTER - # _launch_run has already returned a live run_id. - self._first_update_raised = True - raise RuntimeError("simulated transient DB error on queued->running write") - row = self.rows.get(run_record_id) - if row is None: - return - if "status" in kwargs: - row["status"] = kwargs["status"] - if kwargs.get("run_id") is not None: - row["run_id"] = kwargs["run_id"] - - async def has_active_runs(self, task_id: str) -> bool: - return any(row["task_id"] == task_id and row["status"] in self._ACTIVE for row in self.rows.values()) - - async def mark_stale_active_runs(self, *, error: str) -> int: - return 0 - - -@pytest.mark.asyncio -async def test_post_launch_bookkeeping_failure_does_not_release_active_slot(): - """Regression for issue #4452. - - A transient failure in the ``queued -> running`` bookkeeping write - (after ``_launch_run`` has already returned a live ``run_id``) must NOT - flip the task-run row to ``failed``: ``failed`` is outside the partial - unique index ``uq_scheduled_task_run_active``, so releasing the slot - would let the next dispatch launch a DUPLICATE run. The fix keeps the - row ``running`` with the launched ``run_id`` retained for recovery, - reconciliation, and cancellation. - """ - launched: list[dict] = [] - - async def fake_launch(**kwargs): - launched.append(kwargs) - return {"run_id": f"run-{len(launched)}", "thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo( - [ - { - "id": "task-4452", - "user_id": "user-1", - "thread_id": None, - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "do the thing", - "schedule_type": "cron", - "schedule_spec": {"cron": "*/5 * * * *"}, - "timezone": "UTC", - "status": "enabled", - "overlap_policy": "skip", - } - ] - ) - run_repo = _StatefulRunRepo(fail_first_update=True) - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - now = datetime.now(UTC) - task = dict(task_repo.rows[0]) - - first = await service.dispatch_task(task, now=now, trigger="scheduled") - # The run launched despite the post-launch bookkeeping error; the - # outcome and run_id reflect that a live run is in flight. - assert first["outcome"] == "launched" - assert first["run_id"] == "run-1" - assert first["error"] is not None # the bookkeeping error is surfaced, not hidden - - # Second dispatch must observe the active slot held by run-1 and NOT - # launch a duplicate. On main (bug) this would launch run-2 here. - second = await service.dispatch_task(task, now=now, trigger="scheduled") - assert len(launched) == 1, launched - assert second["outcome"] in {"skipped", "conflict"}, second - - # The launched run_id is retained on the task-run row (status "running", - # not "failed") so reconciliation / cancellation can still reach it. - first_row_id = run_repo.created[0]["run_record_id"] - assert run_repo.rows[first_row_id]["status"] == "running" - assert run_repo.rows[first_row_id]["run_id"] == "run-1" - - # The bookkeeping transient is NOT surfaced as the parent task's - # last_error: the run launched and is still in flight, so the task list - # must not show an error on an actively running task (matching the - # success path's clear-on-launch model). The real terminal outcome is - # written by handle_run_completion. - assert task_repo.updated[1]["last_error"] is None - - -@pytest.mark.asyncio -async def test_pre_launch_failure_still_releases_active_slot(): - """Complement to the #4452 fix: when ``_launch_run`` itself fails (no run - was ever started), the task-run row is marked ``failed`` and the active - slot is released as before -- the post-launch retention path does not - apply because there is no live run to protect. - """ - launched: list[dict] = [] - - async def fake_launch(**kwargs): - launched.append(kwargs) - raise RuntimeError("runtime refused to start the run") - - task_repo = DummyTaskRepo( - [ - { - "id": "task-4452-pre", - "user_id": "user-1", - "thread_id": None, - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "do the thing", - "schedule_type": "cron", - "schedule_spec": {"cron": "*/5 * * * *"}, - "timezone": "UTC", - "status": "enabled", - "overlap_policy": "skip", - } - ] - ) - run_repo = _StatefulRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - result = await service.dispatch_task(dict(task_repo.rows[0]), now=datetime.now(UTC), trigger="scheduled") - - assert result["outcome"] == "failed" - assert result["run_id"] is None - # launch was attempted (and raised), so exactly one launch attempt, and - # the row is terminal -> the slot is released for the next dispatch. - assert len(launched) == 1 - first_row_id = run_repo.created[0]["run_record_id"] - assert run_repo.rows[first_row_id]["status"] == "failed" - assert run_repo.rows[first_row_id]["run_id"] is None - - -@pytest.mark.asyncio -async def test_malformed_launch_result_still_retains_active_slot(): - """Defense-in-depth for the #4452 invariant. - - If ``_launch_run`` returns a malformed result (e.g. missing ``run_id``), - the unpacking line raises AFTER a live run was already created. The - dispatch must still take the retention path (keep the row active so the - slot stays held and no duplicate launches) rather than the pre-launch - generic-failure path, which would mark the row ``failed`` and release - the slot while a run is in flight. Keyed off ``launch_succeeded``, not - ``launched_run_id is not None``. - """ - launched: list[dict] = [] - - async def fake_launch(**kwargs): - launched.append(kwargs) - # Live run started, but the result payload is malformed. - return {"thread_id": kwargs["thread_id"]} - - task_repo = DummyTaskRepo( - [ - { - "id": "task-4452-malformed", - "user_id": "user-1", - "thread_id": None, - "context_mode": "fresh_thread_per_run", - "assistant_id": "lead_agent", - "prompt": "do the thing", - "schedule_type": "cron", - "schedule_spec": {"cron": "*/5 * * * *"}, - "timezone": "UTC", - "status": "enabled", - "overlap_policy": "skip", - } - ] - ) - run_repo = _StatefulRunRepo() - service = ScheduledTaskService( - task_repo=task_repo, - task_run_repo=run_repo, - launch_run=fake_launch, - poll_interval_seconds=5, - lease_seconds=120, - max_concurrent_runs=3, - ) - - now = datetime.now(UTC) - task = dict(task_repo.rows[0]) - - first = await service.dispatch_task(task, now=now, trigger="scheduled") - # Launch succeeded, so the outcome is "launched" (a run is in flight) - # even though the result unpacking raised; run_id is unknown. - assert first["outcome"] == "launched" - assert first["run_id"] is None - - # Second dispatch must observe the active slot still held (row stays in - # an active status, NOT "failed") and NOT launch a duplicate. - second = await service.dispatch_task(task, now=now, trigger="scheduled") - assert len(launched) == 1, launched - assert second["outcome"] in {"skipped", "conflict"}, second - - first_row_id = run_repo.created[0]["run_record_id"] - assert run_repo.rows[first_row_id]["status"] == "running"