github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-25 22:16:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,380 Commits 42 Branches 10 Tags
Commit Graph

8 Commits

Author SHA1 Message Date
Totoro
cd0e74edaf
fix(scheduler): reconcile stuck once tasks from committed run outcome (#5035) 
* fix(scheduler): reconcile stuck once tasks from committed run outcome

Restart recovery (cancel_stuck_once_tasks and the multi-instance
reconcile_stuck_once_tasks) blindly flipped every stuck once-task to
'cancelled'. When handle_run_completion crashed between its two
transactions, a once-task whose run had already committed 'success'
was permanently reported as cancelled.

Both reconciliation paths now read the latest scheduled_task_runs row
without a status filter and finalize the parent to match:
success -> completed (last_error cleared),
failed -> failed with the run's error,
interrupted -> cancelled with the run's error when present,
skipped -> cancelled (no work performed).
Active occurrences (queued/launching/running) are left untouched — a
concurrent completion or a later recovery pass will finalize them once
the run reaches a terminal state.
Tasks without a terminal run row keep the previous generic cancellation.

Review follow-ups (willem-bd / Huixin615):
- Extract _finalise_once_task_from_run() so both recovery paths share one
  outcome mapping (no more drift between single- and multi-instance paths).
  Returns bool (True = finalised, False = active/no-op) for explicit
  counter management at call sites.
- Fix a no-op (`run_row.error or None` -> `run_row.error`) in the skipped
  branch.
- Drop the unused `status` parameter from the test task helpers.
- Use TERMINAL_RUN_STATUSES / ACTIVE_RUN_STATUSES constants (local copies
  to avoid circular import; kept in sync with scheduled_task_runs.sql).
- [P1] Read the latest run AFTER acquiring the parent task row lock, not from
  a pre-lock batch snapshot. The latest-run lookup now runs per task under
  the lock with populate_existing so a concurrently committed status is read
  back fresh.
- [P2] Race tests now use monkeypatch to actually enter the race window:
  _intercepted_fetch commits success in a separate session at the moment the
  per-task fetch fires, so a reverted pre-lock batch implementation fails the
  test, while the current post-lock implementation passes.
- [P1] Do not finalize parent for active occurrences. A non-terminal
  scheduled occurrence means the run is still in progress — the parent must
  be left untouched until the completion path or a later recovery pass
  establishes a terminal outcome.
- [P2] Add cancel_stuck_once_tasks to the single-instance poll loop so
  stuck once-tasks are not left permanently "running" when the startup sweep
  fails (mirrors multi-instance _reconcile_active_state behavior).
- Fix stale docstrings in cancel_stuck_once_tasks and _fetch_latest_run.

Adds regression tests for multiple historical runs (older success +
newer skipped/active) on both paths, monkeypatch-based race tests that
prove a concurrent completion committing success is reflected as
completed, and active-run tests that verify the parent is left
unchanged. Documents the behavior in AGENTS.md.

Fixes #5034

* fix(scheduler): address review comments on completion-consistency fix

- _fetch_latest_run: drop arbitrary id DESC tie-break; order by
  scheduled_for DESC (deterministic recency on schedule position)
- _finalise_once_task_from_run: annotate bool return type
- Centralize TERMINAL/ACTIVE_RUN_STATUSES in scheduled_tasks/model.py;
  stop duplicating them in scheduled_tasks/sql.py and
  scheduled_task_runs/sql.py (removes stale circular-import workaround)
- cancel_stuck_once_tasks: run unconditionally in single-instance poll
  loop (remove try/except swallow)
- tests: pin created_at/scheduled_for in _create_run so recency ordering
  is actually exercised; correct docstrings that described the
  active-occurrence branch as 'generic cancel' instead of 'left unchanged'

* fix(scheduler): correct finalizer return annotation

* fix: order scheduled task runs by creation time

* fix(scheduler): stabilize latest run reconciliation ordering

* fix(scheduler): order latest runs by creation time

* test: update trace scheduler stub

* fix(scheduler): clarify reconciliation diagnostics

Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>

* fix(scheduler): fail closed on startup recovery

Keep single-instance parent reconciliation at startup so it cannot race manual admission. Propagate recovery failures through the Gateway lifespan before channel startup, preventing a half-started scheduler.

Tests cover both recovery failure stages and a queued occurrence that survives startup before the ordinary poll drain launches it.

Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>

* fix(scheduler): order occurrences and fence stale parent writes

Allocate per-task occurrence sequences under the parent lock and guard parent projection across launch, recovery, completion, and queue failure paths. Track launch accounting separately so stale occurrences are counted once without replacing newer results. Commit completion and accounting atomically, preserve legacy history, and cover migrations and reordered execution on SQLite and PostgreSQL.

* fix(scheduler): tighten completion projection and launch fencing diagnostics

Share the once-task outcome mapping between completion and both recovery paths, validate the terminal status before opening the completion transaction, leave cron parent status untouched on completion, log the fenced launch update when an occurrence does not belong to the launched run, and drop the README capability line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(scheduler): compare caller time only against unsequenced occurrences

Among sequenced rows the parent-locked occurrence_seq is the only recency key. An unsequenced row can only be legacy history or an admission by a pre-upgrade Gateway writer, so recovery prefers it over the sequence winner only when its caller timestamp is later, which is the previous ordering for that pair. A rolling upgrade therefore degrades to the pre-sequence behaviour instead of ranking every pre-upgrade admission below every sequenced one. Document that boundary instead of requiring every Gateway writer to stop before the upgrade.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(scheduler): gate once-task recovery on the same projection rule

Recovery now finalises a once-task parent only from the occurrence that can_project() accepts: the highest sequenced occurrence whenever one exists, or the timestamp-latest row for a task whose history is entirely unsequenced. An unsequenced row admitted by a pre-upgrade writer can no longer cancel a parent whose sequenced occurrence is still live, nor stall finalisation of a parent whose sequenced occurrence already completed. Document that pre-upgrade instances project their own admissions during a rolling upgrade.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(scheduler): defer once-task recovery while any occurrence is live

uq_scheduled_task_run_active allows one non-terminal occurrence per task, so a live row is the task's newest admission whatever its caller clock and whether it carries a sequence. Both once-task recovery paths now probe for any active occurrence after the fresh latest-run read and leave the parent untouched while one exists; cancel_stuck_once_tasks also locks the parent row so admission cannot insert a queued occurrence between that probe and the commit. Once no occurrence is live, the sequence winner decides and a terminalised unsequenced row never overrides it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(persistence): follow the local head past canonical 0019

Main's forward-revision tests assumed 0019_thread_incarnations was the local chain head. With 0022_scheduled_occurrence_seq chained after it, seed the canonical-0019 shape explicitly, assert the real head where a database is upgraded, derive the 0020 rollback binary's revision set from the ancestors of its head, and step the PostgreSQL restart scenario back to canonical 0019 before the rollback binary restarts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(migrations): describe the chain through 0022_scheduled_occurrence_seq

The rolling-forward section still ended the local chain at canonical 0019; it now names 0022_scheduled_occurrence_seq as the head and lists it among the revisions the 0020 rollback-floor binary does not know.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(scheduler): accept CI's sync Postgres URL in occurrence fixtures

CI hands over TEST_POSTGRES_URI as postgresql://...?sslmode=disable. The occurrence, ordering and 0022 migration fixtures built async engines from it directly, so SQLAlchemy chose psycopg2, which is not installed. Normalize the scheme to postgresql+asyncpg and drop libpq-only query keys, matching the existing 0019 migration tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(scheduler): keep the backend AGENTS.md chain within its budget

The middlewares guidance chain was already above the hard limit on main, so any added byte in backend/AGENTS.md fails the agent guidance check. Leave backend/AGENTS.md identical to main and record the recovery projection rule in the 0022 migration entry, which already describes the occurrence fields.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-12 07:35:07 +08:00
Aari
645ca08f16
fix(scheduler): enqueue busy scheduled task runs (#4918) 
* fix(frontend): clarify reuse-thread scheduling behavior

* fix(scheduler): enqueue overlapping scheduled runs

* fix(scheduler): preserve queue lease fencing

* fix(scheduler): close queue concurrency races

* fix(scheduler): harden queue timeout bookkeeping

* fix(scheduler): preserve manual failure schedule

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-08-24 15:38:32 +08:00
Copilot
1dd6ba1acb
fix: enforce global concurrent-run budget for manual triggers (#4769) 
* Initial plan

* fix: enforce global concurrent-run budget for manual triggers

Manual triggers now check count_active_runs() before dispatching and
return a conflict result (409 at the router) when max_concurrent_runs
is already reached, preventing the global cap from being exceeded.

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-08-15 00:08:39 +08:00
rain02333z-spec
828363705a
fix(scheduler): support safe multi-instance scheduler recovery (#4713) 
* fix(scheduler): reject unsafe multi-worker startup

* fix(scheduler): support safe multi-instance recovery

* fix(scheduler): make multi-instance recovery lease-safe

* fix(scheduler): address multi-instance review feedback

* docs(scheduler): add multi-instance upgrade notes

---------

Co-authored-by: rain02333z-spec <225106191+rain02333z-spec@users.noreply.github.com>
 2026-08-14 23:50:40 +08:00
Vanzeren
095092418c
fix(gateway):unify thread id validation (#4589) 
* fix(gateway): unify thread ID validation at the API boundary

Thread ID entry points accepted arbitrary strings while downstream
consumers (filesystem paths, Kubernetes Provisioner, JSONL event store)
each enforced different character restrictions, so invalid IDs were
persisted first and only failed later during sandbox/workspace init.

Centralize validation in deerflow.utils.thread_id (pattern
^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest,
scheduler dispatch, paths.py, JSONL store, embedded client, and align
the Provisioner pattern (pinned by a parity test). UUIDs are still
generated only when no ID is supplied; caller-supplied opaque IDs stay
supported.

Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy
cleanup escape hatch (filesystem cleanup guarded), read-only
client.get_thread stays unvalidated, and scheduler rows with legacy
invalid IDs record a failed dispatch instead of raising out of the
poll loop.

* docs: document canonical thread ID contract

README: caller-supplied thread IDs need not be UUIDs; the canonical
pattern and per-endpoint behavior. AGENTS.md: the shared
deerflow.utils.thread_id contract, its enforcement boundaries, and the
legacy-ID escape hatches.

* fix(gateway): close thread ID validation gaps at remaining entry points

Follow-up to the canonical thread ID contract: a full audit found the
uniform-422 coverage only reached about half of the thread_id surfaces.

- routers: 18 routes still took a bare thread_id: str — 13 in
  thread_runs.py (including the five messages/events/workspace-changes
  reads that returned 500 on the JSONL event store vs 404/empty on the
  DB store), 4 read routes in threads.py, and the suggestions route
  flagged in review. DELETE /api/threads/{id} keeps str as the declared
  legacy-cleanup escape hatch.
- client: upload_files/delete_upload/list_uploads/get_artifact now
  validate up front, fulfilling the RFC's 'all mutating entry points'
  clause (get_thread stays unvalidated as the declared legacy read path).
- tui: the /resume literal-ref fallback validates against the canonical
  contract and reports a descriptive error instead of failing deep in
  the client.
- scripts/support_bundle.py: replace the drifted dot-allowing pattern
  with a byte-identical copy of THREAD_ID_PATTERN (kept local so the
  script still runs with a broken venv).

* test(gateway): guard the canonical thread ID contract against regressions

- test_thread_id_route_contract.py: static AST sweep asserting every
  route handler with a thread_id parameter annotates ThreadId
  (whitelist: the DELETE escape hatch), plus a runtime sweep hitting
  all 44 thread_id routes with a non-canonical ID and asserting a 422
  that names thread_id, plus a websocket upgrade-rejection case.
- test_thread_id_validation.py: client entry-point validation,
  support_bundle pattern parity, and TUI literal-ref fallback tests.
- Align two tests that encoded the old contract (dotted IDs).
 2026-08-01 19:42:44 +08:00
now-ing
f37c734406
fix(scheduler): retain launched run when post-launch bookkeeping fails (#4452) (#4504) 
* fix(scheduler): retain launched run when post-launch bookkeeping fails

`dispatch_task()` created a `queued` task-run row, then `_launch_run()`
returned a live `run_id`, and only afterward did the queued->running
bookkeeping (`update_status` + `update_after_launch`) run. When that
bookkeeping raised on a transient DB error, the `except` handler marked the
task-run `failed` with `last_run_id=None`. Because `failed` is outside the
partial unique index `uq_scheduled_task_run_active`, this released the
task's single active slot: the next dispatch cycle could no longer see the
still-live run and launched a duplicate. The launched `run_id` was also
dropped, breaking later recovery / reconciliation / cancellation.

Track `launched_run_id` / `launched_thread_id`, set only after `_launch_run`
returns. In the `except` handler:

- If launch already succeeded, keep the task-run row `running` (so it keeps
  holding the active slot and no duplicate launch can occur) and persist the
  launched `run_id` on the parent task for retention. The bookkeeping
  retries are best-effort with logging; if they fail too the row stays
  `queued`, which is still active and still holds the slot, so we still
  report the run as launched.
- If launch itself failed (no live run was created), behave as before:
  mark the task-run `failed` and release the active slot.

The overlap-skip branch is now guarded by `launched_run_id is None` so a
run that already launched can never be reclassified as a skip / failed.

Adds a stateful regression test (`test_post_launch_bookkeeping_failure_does_not_release_active_slot`)
that injects a failure on the queued->running write and asserts a second
dispatch does not launch another run (`launch_count` stays 1) while the
first `run_id` is retained on the task-run row. The test is verified to
fail on `main` and pass with this change. A complement test pins the
pre-launch-failure path (launch itself raises) to ensure the slot is still
released when no live run exists.

Fixes #4452

* style(scheduler): apply ruff format to fix lint-backend CI

Reformat the two files touched by the previous commit with
`ruff format` (line-length=240 config joins the hand-wrapped
condition/log lines). No semantic change.

Fixes the `lint-backend` CI failure on PR #4504.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(scheduler): key retention on launch_succeeded flag

The previous invariant keyed the retention branch off
`launched_run_id is not None`, but the assignment
`launched_run_id = result["run_id"]` is itself post-launch code that
can raise (KeyError/TypeError on a malformed _launch_run result).
In that case launched_run_id stays None and the dispatch falls through
to the pre-launch generic-failure path, marking the task-run row failed
and releasing the active slot -- even though a live run was just
created (same class of bug as #4452, narrower trigger).

Flip a `launch_succeeded` flag immediately after `await _launch_run(...)`
returns, before any further code that can raise, and key both the
overlap-conflict guard and the retention branch off that flag.

Add a regression test with a malformed launch result (missing run_id):
the dispatch reports outcome="launched", the row stays running, and a
second dispatch does not launch a duplicate.

Addresses willem-bd review point 1 on #4504.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(scheduler): don't surface bookkeeping transient as task last_error

In the post-launch retention path the parent task's last_error was set
to the bookkeeping exception -- an infrastructure-level transient, not
a run-level failure. Between the failed bookkeeping write and the run
completing, the task list showed an error on a task whose run was
actively running.

Clear last_error (like the success path's clear-on-launch model): the
run's real terminal outcome is written by handle_run_completion, and
the transient itself is already recorded via logger.exception.

Assert in the retention regression test that the parent task update
carries last_error=None.

Addresses willem-bd review point 2 on #4504 (taking the drop option).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: now-ing <24534365+now-ing@users.noreply.github.com>
Co-authored-by: now-ing <now-ing@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-29 23:12:41 +08:00
Daoyuan Li
ca3e510b7d
fix(scheduler): close duplicate dispatch race (#4105) 
Enforce one queued or running scheduled-task run per task with a partial unique index. The migration resolves legacy duplicates before creating the index, and losing inserts use the existing conflict or skip outcomes.
 2026-07-24 21:41:09 +08:00
Xinmin Zeng
4fc08b4f15
feat: add scheduled tasks MVP (#3898) 
* feat: add scheduled tasks MVP

* fix: harden scheduled task execution semantics

* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview

Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).

* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin

Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.

* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test

Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
  build from full metadata; guard with inspector.has_table so the revision
  no-ops when the table already exists (0004/0005 are already idempotent via
  _helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
  0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
  must-be-in-the-future validation; use a future date.

* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences

- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
  clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
  and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
  revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
  after the Scheduler notes so the sections render correctly.

* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review

* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review

- defer a once task's terminal status to the run-completion hook; the task
  stays running until the real outcome, and a startup sweep cancels once
  tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
  readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
  pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
  arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
  offset at the resolved instant (once tasks fired an hour late around
  spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
  API error helper between channels and scheduled-tasks frontends

* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics

- scrub internal-only context keys (non_interactive) from the assembled run
  config for non-internal callers: gating body.context alone left the same
  key smuggle-able through the free-form body.config copied verbatim by
  build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
  write cannot clobber a once task already finalized by a fast-failing run's
  completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
  launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
  future; previously the endpoint returned 200 with a next_run_at that could
  never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
  remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
  thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
  user-scoped guardrail providers see the owning user

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
 2026-07-04 21:51:57 +08:00