deer-flow

github/deer-flow

Fork 0

mirror of https://github.com/bytedance/deer-flow.git synced 2026-07-28 17:06:05 +00:00

Commit Graph

Author	SHA1	Message	Date
heart-scalpel	b53c1ae0e0	fix(runs): cancel degrades to lease takeover for multi-worker (#4064 ) * fix(runs): cancel degrades to lease takeover for multi-worker Work item 4 of the multi-worker ownership epic (https://github.com/bytedance/deer-flow/issues/3948). Problem: POST /runs/{run_id}/cancel landing on a non-owning worker returns 409 — the cancel button silently fails under GATEWAY_WORKERS>1 with no sticky routing. cancel() required the current worker to hold the in-memory task/abort_event, which any non-owner pod cannot satisfy. Changes: - RunManager.cancel() returns CancelOutcome enum (cancelled / taken_over / lease_valid_elsewhere / not_active_locally / not_cancellable / unknown) instead of bool, so the router can map each outcome to the right HTTP response. - New store primitive claim_for_takeover(): a single atomic conditional UPDATE that marks a run as error only when status IN (pending, running) AND (lease IS NULL OR lease < now - grace). Closes the stale-read / concurrent-heartbeat race — if the owner renews between our read and write, the UPDATE matches 0 rows and we surface lease_valid_elsewhere. - HTTP cancel + stream-join endpoints route on CancelOutcome: cancelled -> 202 (or 204 with wait=true); taken_over -> 202 immediately (no SSE streaming — the run is terminal on another worker, streaming would hang); lease_valid_elsewhere -> 409 + Retry-After header computed from lease_expires_at + grace_seconds. - RunManager.grace_seconds exposed as a public property; the router no longer reaches into _run_ownership_config. - _is_lease_expired extracted to a module-level function, shared by RunManager.cancel() and MemoryRunStore.claim_for_takeover(). - GATEWAY_WORKERS=1 + heartbeat_enabled=false is zero-regression: the non-local path short-circuits to not_active_locally, preserving the original 409 behaviour the existing tests pin. Tests: 12 new (5 store primitive + 4 cancel-takeover unit + 3 HTTP including a regression guard verifying POST /stream?action=interrupt on a dead-owner run returns 202 instead of hanging on SSE). 244 directly-related tests pass; 36/36 blocking-IO gate pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): guard update_status and self-terminate on takeover Two defenses close a split-brain window where the original owner could overwrite a peer's takeover status: - update_status (SQL + memory store) now guards on status IN ('pending','running'). When takeover already set the row to 'error', the owner's final status write matches 0 rows and is dropped. - _persist_status: when update_status returns False, check whether the row exists before attempting recovery via put(). If the row exists (takeover by another worker), skip recovery instead of blindly upserting over the takeover. - Heartbeat _renew_leases: when update_lease returns False (row no longer pending/running or owner changed), cancel the local task so wasted CPU is bounded to the next heartbeat tick (~10s) instead of the full task lifetime. Also fix three reviewer feedback items: - Re-fetch the store row when cancel() returns lease_valid_elsewhere, so Retry-After uses the owner's freshly-renewed lease instead of a stale value from request start. - Fallback 'unknown' in takeover error message when owner_worker_id is NULL (pre-ownership data). - Remove dead else-10 branch from grace_seconds property (unreachable — all callers are downstream of the heartbeat_enabled guard). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(runs): pin split-brain defences from update_status guard + heartbeat Three tests lock down the takeover authoritativeness so a late-running owner cannot overwrite a peer's claim: - update_status must reject writes when the store row is already terminal (taken over by another worker). - _persist_status must skip row-recovery via put() when the row exists but has been taken over. - Heartbeat _renew_leases must cancel the local task when update_lease returns False (row claimed by another worker). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): precise outcome + log when local cancel loses to peer takeover Two reviewer precision nits on the split-brain defence: - _persist_status: branch the skip-reason log on existing["status"]. error → WARNING "peer takeover" (anomalous); interrupted/success → INFO "local cancel/completion race" (expected when user hits stop as the run finishes). Stops noisy false-positive takeover warnings in operator logs. - cancel() local path: when _persist_status returns False, re-check the store. If a peer's claim_for_takeover flipped the row to error between our in-memory cancel and the guarded update_status, surface taken_over instead of cancelled so the client sees a status consistent with the store. Test: test_cancel_returns_taken_over_when_peer_claims_during_local_cancel pins the race outcome. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): widen update_status guard, de-duplicate lease helpers, add coverage Round 3 of reviewer feedback: - Widen update_status guard to status IN ('pending','running','interrupted'). The original guard blocked interrupted→error (the rollback finalize path), losing the "Rolled back by user" message. interrupted is now permitted while error/success stay locked — takeover protection unchanged. - claim_for_takeover False now re-reads the store row to distinguish causes: owner renewed lease → lease_valid_elsewhere; row went terminal → not_cancellable; another worker already took it over → taken_over. - Extract _raise_lease_valid_elsewhere() helper to de-duplicate the 409+Retry-After block shared across cancel_run and stream_existing_run. - Extract _lease_expired_or_null() in persistence/run/sql.py to de-duplicate the lease-expiry SQL WHERE clause shared by claim_for_takeover and list_inflight_with_expired_lease. - 11 new tests: 5 SQL-layer claim_for_takeover (expired/valid/NULL/ terminal/nonexistent), 3 _compute_retry_after unit (NULL/unparseable/ normal), 2 claim re-read precision (terminal/takeover), 1 stream endpoint 409+Retry-After. Not addressed (non-blocking, reviewer agreed): - The 2–3 store.gets in the takeover cold path: optimizing the API to accept a pre-fetched record would couple the router to the manager more tightly than justified by the perf gain. - The lease-expiry inline loop in MemoryRunStore.list_inflight_with_- expired_lease pre-computes cutoff once for all rows; switching to the shared _is_lease_expired helper would recompute datetime.now() per row with no real benefit. 260 related tests pass; 36/36 blocking-IO gate pass; ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): de-duplicate lease-expiry helper, restore defensive fallback Address final round of review feedback: - Extract is_lease_expired to deerflow.utils.time (no _ prefix, public utility). Manager and MemoryRunStore now import from the same place instead of the store reaching backward into the manager for a private function. - Restore defensive else-10 fallback in grace_seconds property (removed in an earlier round). The guard is unreachable for current callers but protects future ones from AttributeError. - Comment the transient in-memory interrupted vs store error state when a local cancel is superseded by a peer takeover. - Comment the max(1, ...) floor in _compute_retry_after — the floor is a lower bound, not a poll interval; clients should apply jitter. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: rayhpeng <rayhpeng@gmail.com>	2026-07-14 07:37:59 +08:00
john lee	9b19cca91c	fix(runtime): make RunManager.cancel() idempotent for already-interrupted runs (#3055 ) (#3058 ) A second cancel() call on an interrupted run returned False, causing the cancel and stream_existing_run router endpoints to raise 409 on double-stop. Fix: return True inside the lock when record.status == RunStatus.interrupted. This covers both the POST /cancel and POST /join endpoints without any re-fetch or extra get() call — the idempotency lives at the source. Also fixes stream_existing_run (the LangGraph SDK stop-button path), which had the identical cancel() → 409 pattern and was not covered by the original PR. Both endpoints share the fix automatically. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-20 16:37:36 +08:00

Author

SHA1

Message

Date

heart-scalpel

b53c1ae0e0

fix(runs): cancel degrades to lease takeover for multi-worker (#4064 )

* fix(runs): cancel degrades to lease takeover for multi-worker

Work item 4 of the multi-worker ownership epic
(https://github.com/bytedance/deer-flow/issues/3948).

Problem: POST /runs/{run_id}/cancel landing on a non-owning worker
returns 409 — the cancel button silently fails under GATEWAY_WORKERS>1
with no sticky routing. cancel() required the current worker to hold
the in-memory task/abort_event, which any non-owner pod cannot satisfy.

Changes:
- RunManager.cancel() returns CancelOutcome enum (cancelled /
  taken_over / lease_valid_elsewhere / not_active_locally /
  not_cancellable / unknown) instead of bool, so the router can map
  each outcome to the right HTTP response.
- New store primitive claim_for_takeover(): a single atomic
  conditional UPDATE that marks a run as error only when
  status IN (pending, running) AND (lease IS NULL OR
  lease < now - grace). Closes the stale-read / concurrent-heartbeat
  race — if the owner renews between our read and write, the UPDATE
  matches 0 rows and we surface lease_valid_elsewhere.
- HTTP cancel + stream-join endpoints route on CancelOutcome:
  cancelled -> 202 (or 204 with wait=true); taken_over -> 202
  immediately (no SSE streaming — the run is terminal on another
  worker, streaming would hang); lease_valid_elsewhere -> 409 +
  Retry-After header computed from lease_expires_at + grace_seconds.
- RunManager.grace_seconds exposed as a public property; the router
  no longer reaches into _run_ownership_config.
- _is_lease_expired extracted to a module-level function, shared by
  RunManager.cancel() and MemoryRunStore.claim_for_takeover().
- GATEWAY_WORKERS=1 + heartbeat_enabled=false is zero-regression:
  the non-local path short-circuits to not_active_locally, preserving
  the original 409 behaviour the existing tests pin.

Tests: 12 new (5 store primitive + 4 cancel-takeover unit + 3 HTTP
including a regression guard verifying POST /stream?action=interrupt
on a dead-owner run returns 202 instead of hanging on SSE).
244 directly-related tests pass; 36/36 blocking-IO gate pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): guard update_status and self-terminate on takeover

Two defenses close a split-brain window where the original owner
could overwrite a peer's takeover status:

- update_status (SQL + memory store) now guards on
  status IN ('pending','running'). When takeover already set
  the row to 'error', the owner's final status write matches
  0 rows and is dropped.

- _persist_status: when update_status returns False, check
  whether the row exists before attempting recovery via put().
  If the row exists (takeover by another worker), skip recovery
  instead of blindly upserting over the takeover.

- Heartbeat _renew_leases: when update_lease returns False
  (row no longer pending/running or owner changed), cancel the
  local task so wasted CPU is bounded to the next heartbeat
  tick (~10s) instead of the full task lifetime.

Also fix three reviewer feedback items:

- Re-fetch the store row when cancel() returns
  lease_valid_elsewhere, so Retry-After uses the owner's
  freshly-renewed lease instead of a stale value from
  request start.

- Fallback 'unknown' in takeover error message when
  owner_worker_id is NULL (pre-ownership data).

- Remove dead else-10 branch from grace_seconds property
  (unreachable — all callers are downstream of the
  heartbeat_enabled guard).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(runs): pin split-brain defences from update_status guard + heartbeat

Three tests lock down the takeover authoritativeness so a
late-running owner cannot overwrite a peer's claim:

- update_status must reject writes when the store row is already
  terminal (taken over by another worker).
- _persist_status must skip row-recovery via put() when the row
  exists but has been taken over.
- Heartbeat _renew_leases must cancel the local task when
  update_lease returns False (row claimed by another worker).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): precise outcome + log when local cancel loses to peer takeover

Two reviewer precision nits on the split-brain defence:

- _persist_status: branch the skip-reason log on existing["status"].
  error → WARNING "peer takeover" (anomalous); interrupted/success →
  INFO "local cancel/completion race" (expected when user hits stop
  as the run finishes). Stops noisy false-positive takeover warnings
  in operator logs.

- cancel() local path: when _persist_status returns False, re-check
  the store. If a peer's claim_for_takeover flipped the row to error
  between our in-memory cancel and the guarded update_status, surface
  taken_over instead of cancelled so the client sees a status
  consistent with the store.

Test: test_cancel_returns_taken_over_when_peer_claims_during_local_cancel
pins the race outcome.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): widen update_status guard, de-duplicate lease helpers, add coverage

Round 3 of reviewer feedback:

- Widen update_status guard to status IN ('pending','running','interrupted').
  The original guard blocked interrupted→error (the rollback finalize path),
  losing the "Rolled back by user" message. interrupted is now permitted
  while error/success stay locked — takeover protection unchanged.

- claim_for_takeover False now re-reads the store row to distinguish causes:
  owner renewed lease → lease_valid_elsewhere; row went terminal →
  not_cancellable; another worker already took it over → taken_over.

- Extract _raise_lease_valid_elsewhere() helper to de-duplicate the
  409+Retry-After block shared across cancel_run and stream_existing_run.

- Extract _lease_expired_or_null() in persistence/run/sql.py to
  de-duplicate the lease-expiry SQL WHERE clause shared by
  claim_for_takeover and list_inflight_with_expired_lease.

- 11 new tests: 5 SQL-layer claim_for_takeover (expired/valid/NULL/
  terminal/nonexistent), 3 _compute_retry_after unit (NULL/unparseable/
  normal), 2 claim re-read precision (terminal/takeover), 1 stream
  endpoint 409+Retry-After.

Not addressed (non-blocking, reviewer agreed):
- The 2–3 store.gets in the takeover cold path: optimizing the API to
  accept a pre-fetched record would couple the router to the manager
  more tightly than justified by the perf gain.
- The lease-expiry inline loop in MemoryRunStore.list_inflight_with_-
  expired_lease pre-computes cutoff once for all rows; switching to the
  shared _is_lease_expired helper would recompute datetime.now() per row
  with no real benefit.

260 related tests pass; 36/36 blocking-IO gate pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): de-duplicate lease-expiry helper, restore defensive fallback

Address final round of review feedback:

- Extract is_lease_expired to deerflow.utils.time (no _ prefix, public
  utility). Manager and MemoryRunStore now import from the same place
  instead of the store reaching backward into the manager for a private
  function.

- Restore defensive else-10 fallback in grace_seconds property (removed
  in an earlier round). The guard is unreachable for current callers but
  protects future ones from AttributeError.

- Comment the transient in-memory interrupted vs store error state when
  a local cancel is superseded by a peer takeover.

- Comment the max(1, ...) floor in _compute_retry_after — the floor is
  a lower bound, not a poll interval; clients should apply jitter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: rayhpeng <rayhpeng@gmail.com>

2026-07-14 07:37:59 +08:00

john lee

9b19cca91c

fix(runtime): make RunManager.cancel() idempotent for already-interrupted runs (#3055 ) (#3058 )

A second cancel() call on an interrupted run returned False, causing the
cancel and stream_existing_run router endpoints to raise 409 on double-stop.

Fix: return True inside the lock when record.status == RunStatus.interrupted.
This covers both the POST /cancel and POST /join endpoints without any
re-fetch or extra get() call — the idempotency lives at the source.

Also fixes stream_existing_run (the LangGraph SDK stop-button path), which
had the identical cancel() → 409 pattern and was not covered by the
original PR.  Both endpoints share the fix automatically.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-20 16:37:36 +08:00

2 Commits