9 Commits

Author	SHA1	Message	Date
Totoro	cd0e74edaf	fix(scheduler): reconcile stuck once tasks from committed run outcome (#5035) ... * fix(scheduler): reconcile stuck once tasks from committed run outcome Restart recovery (cancel_stuck_once_tasks and the multi-instance reconcile_stuck_once_tasks) blindly flipped every stuck once-task to 'cancelled'. When handle_run_completion crashed between its two transactions, a once-task whose run had already committed 'success' was permanently reported as cancelled. Both reconciliation paths now read the latest scheduled_task_runs row without a status filter and finalize the parent to match: success -> completed (last_error cleared), failed -> failed with the run's error, interrupted -> cancelled with the run's error when present, skipped -> cancelled (no work performed). Active occurrences (queued/launching/running) are left untouched — a concurrent completion or a later recovery pass will finalize them once the run reaches a terminal state. Tasks without a terminal run row keep the previous generic cancellation. Review follow-ups (willem-bd / Huixin615): - Extract _finalise_once_task_from_run() so both recovery paths share one outcome mapping (no more drift between single- and multi-instance paths). Returns bool (True = finalised, False = active/no-op) for explicit counter management at call sites. - Fix a no-op (`run_row.error or None` -> `run_row.error`) in the skipped branch. - Drop the unused `status` parameter from the test task helpers. - Use TERMINAL_RUN_STATUSES / ACTIVE_RUN_STATUSES constants (local copies to avoid circular import; kept in sync with scheduled_task_runs.sql). - [P1] Read the latest run AFTER acquiring the parent task row lock, not from a pre-lock batch snapshot. The latest-run lookup now runs per task under the lock with populate_existing so a concurrently committed status is read back fresh. - [P2] Race tests now use monkeypatch to actually enter the race window: _intercepted_fetch commits success in a separate session at the moment the per-task fetch fires, so a reverted pre-lock batch implementation fails the test, while the current post-lock implementation passes. - [P1] Do not finalize parent for active occurrences. A non-terminal scheduled occurrence means the run is still in progress — the parent must be left untouched until the completion path or a later recovery pass establishes a terminal outcome. - [P2] Add cancel_stuck_once_tasks to the single-instance poll loop so stuck once-tasks are not left permanently "running" when the startup sweep fails (mirrors multi-instance _reconcile_active_state behavior). - Fix stale docstrings in cancel_stuck_once_tasks and _fetch_latest_run. Adds regression tests for multiple historical runs (older success + newer skipped/active) on both paths, monkeypatch-based race tests that prove a concurrent completion committing success is reflected as completed, and active-run tests that verify the parent is left unchanged. Documents the behavior in AGENTS.md. Fixes #5034 * fix(scheduler): address review comments on completion-consistency fix - _fetch_latest_run: drop arbitrary id DESC tie-break; order by scheduled_for DESC (deterministic recency on schedule position) - _finalise_once_task_from_run: annotate bool return type - Centralize TERMINAL/ACTIVE_RUN_STATUSES in scheduled_tasks/model.py; stop duplicating them in scheduled_tasks/sql.py and scheduled_task_runs/sql.py (removes stale circular-import workaround) - cancel_stuck_once_tasks: run unconditionally in single-instance poll loop (remove try/except swallow) - tests: pin created_at/scheduled_for in _create_run so recency ordering is actually exercised; correct docstrings that described the active-occurrence branch as 'generic cancel' instead of 'left unchanged' * fix(scheduler): correct finalizer return annotation * fix: order scheduled task runs by creation time * fix(scheduler): stabilize latest run reconciliation ordering * fix(scheduler): order latest runs by creation time * test: update trace scheduler stub * fix(scheduler): clarify reconciliation diagnostics Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> * fix(scheduler): fail closed on startup recovery Keep single-instance parent reconciliation at startup so it cannot race manual admission. Propagate recovery failures through the Gateway lifespan before channel startup, preventing a half-started scheduler. Tests cover both recovery failure stages and a queued occurrence that survives startup before the ordinary poll drain launches it. Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> * fix(scheduler): order occurrences and fence stale parent writes Allocate per-task occurrence sequences under the parent lock and guard parent projection across launch, recovery, completion, and queue failure paths. Track launch accounting separately so stale occurrences are counted once without replacing newer results. Commit completion and accounting atomically, preserve legacy history, and cover migrations and reordered execution on SQLite and PostgreSQL. * fix(scheduler): tighten completion projection and launch fencing diagnostics Share the once-task outcome mapping between completion and both recovery paths, validate the terminal status before opening the completion transaction, leave cron parent status untouched on completion, log the fenced launch update when an occurrence does not belong to the launched run, and drop the README capability line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(scheduler): compare caller time only against unsequenced occurrences Among sequenced rows the parent-locked occurrence_seq is the only recency key. An unsequenced row can only be legacy history or an admission by a pre-upgrade Gateway writer, so recovery prefers it over the sequence winner only when its caller timestamp is later, which is the previous ordering for that pair. A rolling upgrade therefore degrades to the pre-sequence behaviour instead of ranking every pre-upgrade admission below every sequenced one. Document that boundary instead of requiring every Gateway writer to stop before the upgrade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(scheduler): gate once-task recovery on the same projection rule Recovery now finalises a once-task parent only from the occurrence that can_project() accepts: the highest sequenced occurrence whenever one exists, or the timestamp-latest row for a task whose history is entirely unsequenced. An unsequenced row admitted by a pre-upgrade writer can no longer cancel a parent whose sequenced occurrence is still live, nor stall finalisation of a parent whose sequenced occurrence already completed. Document that pre-upgrade instances project their own admissions during a rolling upgrade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(scheduler): defer once-task recovery while any occurrence is live uq_scheduled_task_run_active allows one non-terminal occurrence per task, so a live row is the task's newest admission whatever its caller clock and whether it carries a sequence. Both once-task recovery paths now probe for any active occurrence after the fresh latest-run read and leave the parent untouched while one exists; cancel_stuck_once_tasks also locks the parent row so admission cannot insert a queued occurrence between that probe and the commit. Once no occurrence is live, the sequence winner decides and a terminalised unsequenced row never overrides it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(persistence): follow the local head past canonical 0019 Main's forward-revision tests assumed 0019_thread_incarnations was the local chain head. With 0022_scheduled_occurrence_seq chained after it, seed the canonical-0019 shape explicitly, assert the real head where a database is upgraded, derive the 0020 rollback binary's revision set from the ancestors of its head, and step the PostgreSQL restart scenario back to canonical 0019 before the rollback binary restarts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(migrations): describe the chain through 0022_scheduled_occurrence_seq The rolling-forward section still ended the local chain at canonical 0019; it now names 0022_scheduled_occurrence_seq as the head and lists it among the revisions the 0020 rollback-floor binary does not know. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(scheduler): accept CI's sync Postgres URL in occurrence fixtures CI hands over TEST_POSTGRES_URI as postgresql://...?sslmode=disable. The occurrence, ordering and 0022 migration fixtures built async engines from it directly, so SQLAlchemy chose psycopg2, which is not installed. Normalize the scheme to postgresql+asyncpg and drop libpq-only query keys, matching the existing 0019 migration tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(scheduler): keep the backend AGENTS.md chain within its budget The middlewares guidance chain was already above the hard limit on main, so any added byte in backend/AGENTS.md fails the agent guidance check. Leave backend/AGENTS.md identical to main and record the recovery projection rule in the 0022 migration entry, which already describes the occurrence fields. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-12 07:35:07 +08:00
Hyeonsang Cho	9146bfa03d	feature(gateway): issue request trace ids unconditionally (#5119) ... * refactor(gateway): issue request trace ids unconditionally The request trace id was gated behind logging.enhance.enabled at every entry point, so downstream code had to keep asking whether one existed: a header-provenance flag in its own ContextVar, a precedence resolver, and three-level carrier fallbacks at each consumer. Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP; ensure_trace_context covers the entry points that never touch ASGI -- scheduled occurrences, MCP task notification runs, IM channel messages, and the embedded client -- each scoped to one unit of work so a long-lived worker task cannot leak one occurrence's id into the next. The ContextVar becomes the only source; the response header, runtime context, run metadata and log records are derived outputs. Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and drop their presence guards. Removed: resolve_deerflow_trace_id, the header-provenance flag and its three helpers, set/reset_current_trace_id, is_trace_correlation_enabled and its gateway alias. BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and it cannot be turned off; logging.enhance.enabled controls log output only. Installations on the default enabled: false will start seeing the header. No config keys were added or removed. * fix(gateway): stop persisting a caller-supplied trace id on the run record body.metadata forks two ways: through build_run_config into the live run config, which the run worker restamps, and through create_or_reject into the run record that the runs API echoes verbatim. Only the first was covered, so a client sending metadata.deerflow_trace_id made the most durable and most visible surface of a run disagree with the X-Trace-Id and the log lines the same request produced -- a correlation id that does not match the logs is worse than none. Stamp the server-issued id once at the trust boundary so both forks receive it, preserving the caller's own metadata keys. Close the same gap on config.context, which reaches the runtime context by a separate path: _build_runtime_context no longer merges server-owned keys from the caller, and _install_runtime_context assigns rather than setdefaults. A thread's metadata is no longer seeded with the run-scoped id of whichever run created it -- one thread spans many runs and as many trace ids. Found by driving a real run through the Gateway and reading the run back from the runs API; every unit test built its metadata by hand and so could not see it. * fix(gateway): expose X-Trace-Id to split-origin browser clients X-Trace-Id is not on the CORS safelist, so a browser client served from a separate origin could not read it -- and those are exactly the clients that cannot read the Gateway's logs either, leaving them with nothing to quote in a bug report. Same-origin nginx deployments were unaffected, which is why this stayed hidden. Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing TRACE_ID_HEADER rather than repeating the literal. * fix(gateway): keep X-Trace-Id on unhandled-exception 500s Starlette's ServerErrorMiddleware sits outside every user middleware and emits unhandled-exception 500s through the raw send, so those responses never pass TraceMiddleware's header-writing wrapper. The 500 for a server bug is exactly the response a user most needs to correlate with a log line, and it was the one response that shipped without the id. TraceMiddleware now tracks whether http.response.start has been sent. On an exception with no response started it emits its own plain 500 carrying the header, then re-raises: the outer ServerErrorMiddleware sees the response already started and only re-raises too, so the server's exception logging is untouched. An exception mid-stream keeps propagating unchanged — a second response start cannot be sent, and the already-written header stands. The trace id is printable ASCII by construction (normalize_trace_id / generate_trace_id), which is what makes the raw latin-1 header encoding safe. * fix(gateway): strip the forged trace id from the persisted request echo The run-record fix stopped a forged metadata.deerflow_trace_id on the authoritative metadata surface, but the raw request echo still carried one: create_or_reject persists body.config verbatim as runs.kwargs_json, which the runs API serves back. A client posting config.context.deerflow_trace_id therefore still got its forged value stored and echoed on one API surface while the header, logs, run metadata, and checkpoint all carried the real id — the id is ignored as input there, so echoing it back only manufactures disagreement. Two changes close it. redact_config_secrets — already the shared scrub for that echo, applied at admission and again at serve time, so historical records are covered too — now also drops deerflow_trace_id from config.metadata and config.context. And build_run_config now merges run metadata onto a copy of the caller's config["metadata"] instead of updating it in place: the nested values of the request config are reference copies, so the in-place merge was writing the server-stamped key through into body.config, contaminating the "what the client sent" record before it was persisted (and incidentally masking the forged-value echo on the metadata container). The regression test posts a forged id through body.metadata, config.metadata, and config.context at once and reads the kwargs echo back off the run record, failing if either leak returns. * docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence The trace section of the harness AGENTS.md now covers the two fixes that close the derived-output rule (the kwargs-echo scrub in redact_config_secrets plus build_run_config's copy merge, and TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains their Fixed entries. It also writes down the one accepted divergence: a crash-recovered scheduled launch reuses the durable run through its idempotency key, and start_run returns early on idempotency_reused without restamping — so the run record keeps the first attempt's deerflow_trace_id while the retry's own log lines carry the freshly minted id of its ensure_trace_context binding. The divergence is confined to the crash-recovery window and is accepted rather than fixed: restamping on reuse would rewrite a persisted record for a run that already exists, which is worse than two ids that each correlate their own attempt's logs. Written down so the next reader of the scheduler recovery path does not diagnose it as a bug. * docs(config): align the logging.enhance schema note with the unconditional trace id The config-module AGENTS.md still described logging.enhance as the gate for the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is gone: ids are issued unconditionally and this block decides log output only. Left as-is, the stale wording invites an agent to "restore" a header gate it believes was lost. Reworded to match the sibling AGENTS.md files and config.example.yaml, with a pointer to the Request Trace Context section that owns the full model. * docs(changelog): link the trace entries to #5119 The five new entries pointed at the ([#XXXX]) placeholder with no reference definition, rendering as literal text instead of a link — and RELEASING.md step 2 relies on those references when the section becomes release notes. All five now point at #5119, with the definition appended to the reference block. * refactor(harness): rename _stream_without_trace_context to _stream_turn The name asserted the opposite of what the method now does. It was accurate while logging.enhance.enabled could route stream() around the trace scope; with the gate gone it is the only stream implementation left, and it binds the id itself via ensure_trace_id(). Private, so the rename touches only the definition and the one stream() call site. * docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget The expanded Request Trace Context section pushed the effective AGENTS.md chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the section from 7,359 to 4592 bytes with no facts removed: the entry-point table, the derived-output rule and its enforcement points, the accepted scheduled-retry divergence, the two resolution helpers, the stream() binding rationale, the log-output-only gate, the CORS listing, the 500 fallback, and the test map all remain. Sized against the merge, not just the branch: current main grew the same chain by ~724 bytes, so the check was verified on the merged tree as well (97,772 bytes; branch tree 97,048). * fix(gateway): declare content-length on the fallback 500 The pre-response 500 declared content-type but no content-length, leaving the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response it replaces, which sends content-length: 21. The explicit header keeps the fallback byte-identical to what clients saw before. * docs(readme): drop the trace-correlation condition from the translations The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id matches X-Trace-Id "when request trace correlation is enabled". The id now always matches and that condition no longer exists, so each bullet states the unconditional match and that logging.enhance.enabled only controls whether the id is printed into logs — the one piece of the feature a user can still configure. * test(gateway): pin TraceMiddleware wiring through create_app() Every X-Trace-Id test exercised a hand-built four-route app, so the real stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting it — or short-circuiting above it — passed CI while silently dropping both the response header and the ambient id the run-record stamp and enhanced log records derive from. One case now drives /health through create_app() and asserts the inbound id round-trips; mutation-checked by removing the wiring line, which fails exactly this test. * docs(gateway): note the fallback 500 is CORS-opaque The pre-response 500 is emitted outside CORSMiddleware — the exception has already unwound past it — so it carries no Access-Control-Allow-Origin and a split-origin browser client cannot read the id on this one response, unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the class and in the CHANGELOG entry rather than fixed: replicating the origin allowlist outside CORSMiddleware would let the two policies drift. * fix(harness): keep abandoned-stream cleanup inside the trace binding stream() binds the turn's id around each next(inner) and resets it before yielding, but the finally's inner.close() ran after that binding was gone. Abandoning the stream therefore drove the inner LangGraph generator's GeneratorExit/finally path with no trace id — or an unrelated ambient one from whichever context ran the close — so cancellation and finalization logs and callbacks did not correlate with the turn they belong to. inner.close() is now wrapped in a local bind/reset of the same turn id. The token is set and reset in the same frame, never across a yield, so the per-step cross-context safety is preserved even when GC closes the generator from another Context — pinned by the existing copy_context close test, which now exercises this path. The regression test records the id from the inner generator's finally and fails without the binding. * test(harness): teach the worker-trace fake about RunManager.cleanup Upstream #5112 (bound gateway memory after terminal runs) added a run_manager.cleanup(run_id) call to run_agent's finalization, so the merge-commit CI run failed all five worker-trace-binding tests with AttributeError on this PR's _FakeRunManager. The fake gains the same no-op shape as its other methods. * docs(gateway): bring the gateway AGENTS.md back under its soft budget Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over the 40,960 soft budget that test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes enforces — its Unit Tests run on main was cancelled by push concurrency, so main is currently red on that test and every PR merge-run inherits the failure. Two whitespace/wording trims in the row #5092 touched (a doubled space, and "its configured `context_window`" → "its `context_window`") bring the file to 40,953 with no content change. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-01 16:49:39 +08:00
Aari	645ca08f16	fix(scheduler): enqueue busy scheduled task runs (#4918) ... * fix(frontend): clarify reuse-thread scheduling behavior * fix(scheduler): enqueue overlapping scheduled runs * fix(scheduler): preserve queue lease fencing * fix(scheduler): close queue concurrency races * fix(scheduler): harden queue timeout bookkeeping * fix(scheduler): preserve manual failure schedule --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-24 15:38:32 +08:00
Copilot	1dd6ba1acb	fix: enforce global concurrent-run budget for manual triggers (#4769) ... * Initial plan * fix: enforce global concurrent-run budget for manual triggers Manual triggers now check count_active_runs() before dispatching and return a conflict result (409 at the router) when max_concurrent_runs is already reached, preventing the global cap from being exceeded. Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-08-15 00:08:39 +08:00
rain02333z-spec	828363705a	fix(scheduler): support safe multi-instance scheduler recovery (#4713) ... * fix(scheduler): reject unsafe multi-worker startup * fix(scheduler): support safe multi-instance recovery * fix(scheduler): make multi-instance recovery lease-safe * fix(scheduler): address multi-instance review feedback * docs(scheduler): add multi-instance upgrade notes --------- Co-authored-by: rain02333z-spec <225106191+rain02333z-spec@users.noreply.github.com>	2026-08-14 23:50:40 +08:00
Vanzeren	095092418c	fix(gateway):unify thread id validation (#4589) ... * fix(gateway): unify thread ID validation at the API boundary Thread ID entry points accepted arbitrary strings while downstream consumers (filesystem paths, Kubernetes Provisioner, JSONL event store) each enforced different character restrictions, so invalid IDs were persisted first and only failed later during sandbox/workspace init. Centralize validation in deerflow.utils.thread_id (pattern ^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest, scheduler dispatch, paths.py, JSONL store, embedded client, and align the Provisioner pattern (pinned by a parity test). UUIDs are still generated only when no ID is supplied; caller-supplied opaque IDs stay supported. Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy cleanup escape hatch (filesystem cleanup guarded), read-only client.get_thread stays unvalidated, and scheduler rows with legacy invalid IDs record a failed dispatch instead of raising out of the poll loop. * docs: document canonical thread ID contract README: caller-supplied thread IDs need not be UUIDs; the canonical pattern and per-endpoint behavior. AGENTS.md: the shared deerflow.utils.thread_id contract, its enforcement boundaries, and the legacy-ID escape hatches. * fix(gateway): close thread ID validation gaps at remaining entry points Follow-up to the canonical thread ID contract: a full audit found the uniform-422 coverage only reached about half of the thread_id surfaces. - routers: 18 routes still took a bare thread_id: str — 13 in thread_runs.py (including the five messages/events/workspace-changes reads that returned 500 on the JSONL event store vs 404/empty on the DB store), 4 read routes in threads.py, and the suggestions route flagged in review. DELETE /api/threads/{id} keeps str as the declared legacy-cleanup escape hatch. - client: upload_files/delete_upload/list_uploads/get_artifact now validate up front, fulfilling the RFC's 'all mutating entry points' clause (get_thread stays unvalidated as the declared legacy read path). - tui: the /resume literal-ref fallback validates against the canonical contract and reports a descriptive error instead of failing deep in the client. - scripts/support_bundle.py: replace the drifted dot-allowing pattern with a byte-identical copy of THREAD_ID_PATTERN (kept local so the script still runs with a broken venv). * test(gateway): guard the canonical thread ID contract against regressions - test_thread_id_route_contract.py: static AST sweep asserting every route handler with a thread_id parameter annotates ThreadId (whitelist: the DELETE escape hatch), plus a runtime sweep hitting all 44 thread_id routes with a non-canonical ID and asserting a 422 that names thread_id, plus a websocket upgrade-rejection case. - test_thread_id_validation.py: client entry-point validation, support_bundle pattern parity, and TUI literal-ref fallback tests. - Align two tests that encoded the old contract (dotted IDs).	2026-08-01 19:42:44 +08:00
now-ing	f37c734406	fix(scheduler): retain launched run when post-launch bookkeeping fails (#4452) (#4504) ... * fix(scheduler): retain launched run when post-launch bookkeeping fails `dispatch_task()` created a `queued` task-run row, then `_launch_run()` returned a live `run_id`, and only afterward did the queued->running bookkeeping (`update_status` + `update_after_launch`) run. When that bookkeeping raised on a transient DB error, the `except` handler marked the task-run `failed` with `last_run_id=None`. Because `failed` is outside the partial unique index `uq_scheduled_task_run_active`, this released the task's single active slot: the next dispatch cycle could no longer see the still-live run and launched a duplicate. The launched `run_id` was also dropped, breaking later recovery / reconciliation / cancellation. Track `launched_run_id` / `launched_thread_id`, set only after `_launch_run` returns. In the `except` handler: - If launch already succeeded, keep the task-run row `running` (so it keeps holding the active slot and no duplicate launch can occur) and persist the launched `run_id` on the parent task for retention. The bookkeeping retries are best-effort with logging; if they fail too the row stays `queued`, which is still active and still holds the slot, so we still report the run as launched. - If launch itself failed (no live run was created), behave as before: mark the task-run `failed` and release the active slot. The overlap-skip branch is now guarded by `launched_run_id is None` so a run that already launched can never be reclassified as a skip / failed. Adds a stateful regression test (`test_post_launch_bookkeeping_failure_does_not_release_active_slot`) that injects a failure on the queued->running write and asserts a second dispatch does not launch another run (`launch_count` stays 1) while the first `run_id` is retained on the task-run row. The test is verified to fail on `main` and pass with this change. A complement test pins the pre-launch-failure path (launch itself raises) to ensure the slot is still released when no live run exists. Fixes #4452 * style(scheduler): apply ruff format to fix lint-backend CI Reformat the two files touched by the previous commit with `ruff format` (line-length=240 config joins the hand-wrapped condition/log lines). No semantic change. Fixes the `lint-backend` CI failure on PR #4504. Co-Authored-By: Claude <noreply@anthropic.com> * fix(scheduler): key retention on launch_succeeded flag The previous invariant keyed the retention branch off `launched_run_id is not None`, but the assignment `launched_run_id = result["run_id"]` is itself post-launch code that can raise (KeyError/TypeError on a malformed _launch_run result). In that case launched_run_id stays None and the dispatch falls through to the pre-launch generic-failure path, marking the task-run row failed and releasing the active slot -- even though a live run was just created (same class of bug as #4452, narrower trigger). Flip a `launch_succeeded` flag immediately after `await _launch_run(...)` returns, before any further code that can raise, and key both the overlap-conflict guard and the retention branch off that flag. Add a regression test with a malformed launch result (missing run_id): the dispatch reports outcome="launched", the row stays running, and a second dispatch does not launch a duplicate. Addresses willem-bd review point 1 on #4504. Co-Authored-By: Claude <noreply@anthropic.com> * fix(scheduler): don't surface bookkeeping transient as task last_error In the post-launch retention path the parent task's last_error was set to the bookkeeping exception -- an infrastructure-level transient, not a run-level failure. Between the failed bookkeeping write and the run completing, the task list showed an error on a task whose run was actively running. Clear last_error (like the success path's clear-on-launch model): the run's real terminal outcome is written by handle_run_completion, and the transient itself is already recorded via logger.exception. Assert in the retention regression test that the parent task update carries last_error=None. Addresses willem-bd review point 2 on #4504 (taking the drop option). Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: now-ing <24534365+now-ing@users.noreply.github.com> Co-authored-by: now-ing <now-ing@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 23:12:41 +08:00
Daoyuan Li	ca3e510b7d	fix(scheduler): close duplicate dispatch race (#4105) ... Enforce one queued or running scheduled-task run per task with a partial unique index. The migration resolves legacy duplicates before creating the index, and losing inserts use the existing conflict or skip outcomes.	2026-07-24 21:41:09 +08:00
Xinmin Zeng	4fc08b4f15	feat: add scheduled tasks MVP (#3898) ... * feat: add scheduled tasks MVP * fix: harden scheduled task execution semantics * feat(scheduled-tasks): preset-driven schedule form with timezone and live preview Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom) plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local for one-time tasks, a timezone selector defaulting to the browser timezone, and a live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers). * feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin Localize the rest of the scheduled-tasks page (filters, detail pane, actions, edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four built-in recipe templates (GitHub Trending, news digest, issue triage, weekly report) exposed as a chip row that pre-fills title + prompt + schedule. Pin Playwright locale to en-US so E2E selectors stay stable against i18n. No backend change, no new deps. * fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test Merge with main surfaced three CI failures: - 0003_scheduled_tasks create_table collided with legacy test seeds that build from full metadata; guard with inspector.has_table so the revision no-ops when the table already exists (0004/0005 are already idempotent via _helpers.py). - persistence bootstrap concurrency/regression tests pinned HEAD to main's 0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable. - once-task router test used a fixed past run_at and tripped the must-be-in-the-future validation; use a future date. * address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences - fetchThreadScheduledTasks now checks response.ok like the other fetchers. - trigger endpoint returns 502 (not 409) when dispatch fails outright, so clients can distinguish a real conflict from a server-side failure. - E2E mock normalizes scheduled-task objects with context_mode/last_thread_id and nullable thread_id, matching the backend contract the UI renders against. - Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its revision id (file was renamed in spirit already; filename now follows). - CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence after the Scheduler notes so the sections render correctly. * fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review * fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review - defer a once task's terminal status to the run-completion hook; the task stays running until the real outcome, and a startup sweep cancels once tasks orphaned by a crash (launch-time 'completed' could stick forever) - record interrupted runs as a distinct 'interrupted' run status with a readable message; an interrupted once task ends 'cancelled', not 'failed' - enforce overlap_policy=skip for fresh_thread_per_run via an active-run pre-check (same-thread ConflictError can never fire across fresh threads) - protect terminal run statuses from the late launch-path 'running' write - honor context.non_interactive only for internally-authenticated callers; arbitrary clients can no longer strip ask_clarification - fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the offset at the resolved instant (once tasks fired an hour late around spring-forward and the create->edit round-trip diverged) - drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway API error helper between channels and scheduled-tasks frontends * fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics - scrub internal-only context keys (non_interactive) from the assembled run config for non-internal callers: gating body.context alone left the same key smuggle-able through the free-form body.config copied verbatim by build_run_config - guard update_after_launch with protect_terminal so the launch bookkeeping write cannot clobber a once task already finalized by a fast-failing run's completion hook (parent-row sibling of the run-row guard) - reject a manual trigger while the task has an active run (409) instead of launching a duplicate concurrent run on fresh_thread_per_run - re-arm a terminal once task to enabled when PATCH pushes run_at into the future; previously the endpoint returned 200 with a next_run_at that could never be claimed - make max_concurrent_runs a real global cap: each poll claims only into the remaining budget of active (queued/running) scheduled runs - paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the thread filter of /threads/{id}/scheduled-tasks into SQL - stamp context.user_id on scheduler-launched runs, matching IM channels, so user-scoped guardrail providers see the owning user --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-04 21:51:57 +08:00